> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/maravento/vault/llms.txt
> Use this file to discover all available pages before exploring further.
# DoFi
> Domain filtering tools for removing overlapping domains and TLD validation
# DoFi
**Domain Filtering** - A collection of Python and Bash scripts for validating, filtering, and checking domain lists.
## Overview
DoFi provides two main tools:
Python script that validates TLDs, removes overlapping domains, and filters invalid entries
Bash script that verifies domain existence using the host command
## Download Project
```bash theme={null}
sudo apt install -y python-is-python3
wget -qO gitfolder.py https://raw.githubusercontent.com/maravento/vault/master/scripts/python/gitfolder.py
chmod +x gitfolder.py
python gitfolder.py https://github.com/maravento/vault/dofi
```
## Requirements
* **Python:** 3.12.3 or later
* **Bash:** 5.2.21 or later
* **Tested on:** Ubuntu 22.04/24.04 x64
**Input Format:** Ensure your domain list has no `http://`, `https://`, or `www.` prefixes before processing.
## Domain Filter (Python)
Advanced domain filtering with TLD validation and overlap removal.
### Features
The Python script performs comprehensive domain filtering:
Downloads public suffix TLDs from multiple authoritative sources
Removes invalid or duplicate TLDs from the collection
Filters domains to ensure they end with a valid TLD
Removes overlapping domains (e.g., keeps `example.com` and removes `sub.example.com` if both exist)
Excludes duplicates from previously validated domains
Saves cleaned domains and removed entries to separate files
### Basic Usage
```bash theme={null}
# Download the script
wget -qO domfilter.py https://raw.githubusercontent.com/maravento/vault/master/dofi/domfilter.py
# Run with your domain list
python domfilter.py --input mylst.txt
```
### Output Files
By default, the script creates:
* `output.txt` - Validated and filtered domains
* `removed.txt` - Domains that were filtered out
Customize output file names:
```bash theme={null}
python domfilter.py --input mylst.txt \
--output outlst.txt \
--remove removelst.txt
```
### TLD Coverage
The filter validates against comprehensive TLD sources:
* **ccTLDs** - Country code top-level domains
* **gTLDs** - Generic top-level domains
* **sTLDs** - Sponsored top-level domains
* **eTLDs** - Effective top-level domains
* **4LDs** - Four-level domains
All TLDs are saved to `tlds.txt` during processing.
### Example
```bash theme={null}
# Input file: domains.txt
example.com
sub.example.com
invalid-domain.xyz123
test.org
# Run filter
python domfilter.py --input domains.txt
# Output (output.txt):
example.com
test.org
# Removed (removed.txt):
sub.example.com # Overlapping with example.com
invalid-domain.xyz123 # Invalid TLD
```
## Domain Check (Bash)
Verifies domain existence using DNS lookups.
### Features
The Bash script checks domain validity:
* Uses `host` command for DNS verification
* Parallel processing for speed
* Separates valid and invalid domains
* Generates difference report
### Basic Usage
```bash theme={null}
# Download the script
wget -qO domcheck.sh https://raw.githubusercontent.com/maravento/vault/master/dofi/domcheck.sh
chmod +x domcheck.sh
# Run with your domain list
./domcheck.sh my_domain_list.txt
```
### Output Files
The script generates three output files:
Existing domains verified via DNS
Non-existent domains that failed verification
Difference between input and output
### Parallel Processing
Control the number of parallel checks:
```bash theme={null}
# Default: 100 parallel processes
./domcheck.sh my_domain_list.txt
# Custom: 50 parallel processes
./domcheck.sh my_domain_list.txt 50
```
Adjust the parallel process count based on your system resources and network capacity. Higher values = faster processing but more resource usage.
### Example
```bash theme={null}
# Input file: check_domains.txt
google.com
example-does-not-exist-12345.com
github.com
# Run check
./domcheck.sh check_domains.txt
# Results:
# hit.txt:
google.com
github.com
# fault.txt:
example-does-not-exist-12345.com
# outdiff.txt:
# Shows detailed differences
```
## Workflow Examples
### Complete Domain Cleaning
Combine both tools for comprehensive cleaning:
```bash theme={null}
# Step 1: Filter and validate TLDs
python domfilter.py --input raw_domains.txt --output filtered.txt
# Step 2: Verify domain existence
./domcheck.sh filtered.txt
# Final result in hit.txt - fully validated, existing domains
```
### Large List Processing
```bash theme={null}
# Process large lists efficiently
python domfilter.py --input huge_list.txt --output stage1.txt
./domcheck.sh stage1.txt 200 # Increase parallel processes
```
## TLD Data Sources
DoFi pulls TLD data from authoritative sources:
Official IANA TLD list
Mozilla's public suffix list
Supported gTLD database
Extended TLD appendix
## Performance Tips
For large domain lists (millions of entries):
* Run on systems with adequate RAM (4GB+ recommended)
* Use SSD storage for faster I/O
* TLD cache is created once and reused
For efficient domain checking:
* Increase parallel processes on powerful systems
* Use reliable DNS servers
* Consider network bandwidth limits
* Monitor system load during processing
```bash theme={null}
# Monitor performance
./domcheck.sh large_list.txt 150 &
watch -n 1 'wc -l hit.txt fault.txt'
```
Optimize the full pipeline:
```bash theme={null}
# Filter first (fast, local)
time python domfilter.py --input raw.txt --output filtered.txt
# Then check (slower, network-dependent)
time ./domcheck.sh filtered.txt 100
```
This approach minimizes network checks by filtering invalid entries first.
## Use Cases
Clean and validate domain blocklists before deployment
Verify domain lists for SEO tools and analysis
Validate threat intelligence domain feeds
Maintain clean DNS zone files and records
## Troubleshooting
If TLD sources are unavailable:
* Check internet connectivity
* Verify firewall/proxy settings
* Script will use cached TLDs if available
If domain checks are slow or timing out:
* Reduce parallel processes
* Check DNS server responsiveness
* Consider using local DNS cache
```bash theme={null}
# Test DNS performance
time host google.com
# Use fewer parallel processes
./domcheck.sh domains.txt 25
```
For very large lists:
* Split input files into smaller chunks
* Process in batches
* Increase system swap if needed
```bash theme={null}
# Split large file
split -l 100000 huge_list.txt chunk_
# Process each chunk
for file in chunk_*; do
python domfilter.py --input $file --output filtered_$file
done
```
## Best Practices
Clean input files before processing:
* Remove `http://`, `https://`, `www.` prefixes
* Convert to lowercase
* Remove duplicates
* One domain per line
Use Python script first to validate TLDs and remove overlaps
Verify filtered domains actually exist via DNS
Check removed entries for false positives before finalizing
## License
[](https://www.gnu.org/licenses/gpl.txt)
[](https://creativecommons.org/licenses/by-sa/4.0/)
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.