DoFi

Domain filtering tools for removing overlapping domains and TLD validation

DoFi

Domain Filtering - A collection of Python and Bash scripts for validating, filtering, and checking domain lists.

Overview

DoFi provides two main tools:

Domain Filter

Python script that validates TLDs, removes overlapping domains, and filters invalid entries

Domain Check

Bash script that verifies domain existence using the host command

Download Project

sudo apt install -y python-is-python3
wget -qO gitfolder.py https://raw.githubusercontent.com/maravento/vault/master/scripts/python/gitfolder.py
chmod +x gitfolder.py
python gitfolder.py https://github.com/maravento/vault/dofi

Requirements

• Python: 3.12.3 or later
• Bash: 5.2.21 or later
• Tested on: Ubuntu 22.04/24.04 x64

Input Format: Ensure your domain list has no http://, https://, or www. prefixes before processing.

Domain Filter (Python)

Advanced domain filtering with TLD validation and overlap removal.

Features

The Python script performs comprehensive domain filtering:

1. TLD Collection

   Downloads public suffix TLDs from multiple authoritative sources

2. TLD Validation

   Removes invalid or duplicate TLDs from the collection

3. Domain Validation

   Filters domains to ensure they end with a valid TLD

4. Overlap Removal

   Removes overlapping domains (e.g., keeps example.com and removes sub.example.com if both exist)

5. Duplicate Exclusion

   Excludes duplicates from previously validated domains

6. Output Generation

   Saves cleaned domains and removed entries to separate files

Basic Usage

# Download the script
wget -qO domfilter.py https://raw.githubusercontent.com/maravento/vault/master/dofi/domfilter.py

# Run with your domain list
python domfilter.py --input mylst.txt

Output Files

Default Output

By default, the script creates:

• output.txt - Validated and filtered domains
• removed.txt - Domains that were filtered out

Custom Output

Customize output file names:

python domfilter.py --input mylst.txt \
  --output outlst.txt \
  --remove removelst.txt

TLD Coverage

The filter validates against comprehensive TLD sources:

• ccTLDs - Country code top-level domains
• gTLDs - Generic top-level domains
• sTLDs - Sponsored top-level domains
• eTLDs - Effective top-level domains
• 4LDs - Four-level domains

All TLDs are saved to tlds.txt during processing.

Example

# Input file: domains.txt
example.com
sub.example.com
invalid-domain.xyz123
test.org

# Run filter
python domfilter.py --input domains.txt

# Output (output.txt):
example.com
test.org

# Removed (removed.txt):
sub.example.com  # Overlapping with example.com
invalid-domain.xyz123  # Invalid TLD

Domain Check (Bash)

Verifies domain existence using DNS lookups.

Features

The Bash script checks domain validity:

• Uses host command for DNS verification
• Parallel processing for speed
• Separates valid and invalid domains
• Generates difference report

Basic Usage

# Download the script
wget -qO domcheck.sh https://raw.githubusercontent.com/maravento/vault/master/dofi/domcheck.sh
chmod +x domcheck.sh

# Run with your domain list
./domcheck.sh my_domain_list.txt

Output Files

The script generates three output files:

hit.txt

Existing domains verified via DNS

fault.txt

Non-existent domains that failed verification

outdiff.txt

Difference between input and output

Parallel Processing

Control the number of parallel checks:

# Default: 100 parallel processes
./domcheck.sh my_domain_list.txt

# Custom: 50 parallel processes
./domcheck.sh my_domain_list.txt 50

Adjust the parallel process count based on your system resources and network capacity. Higher values = faster processing but more resource usage.

Example

# Input file: check_domains.txt
google.com
example-does-not-exist-12345.com
github.com

# Run check
./domcheck.sh check_domains.txt

# Results:
# hit.txt:
google.com
github.com

# fault.txt:
example-does-not-exist-12345.com

# outdiff.txt:
# Shows detailed differences

Workflow Examples

Complete Domain Cleaning

Combine both tools for comprehensive cleaning:

# Step 1: Filter and validate TLDs
python domfilter.py --input raw_domains.txt --output filtered.txt

# Step 2: Verify domain existence
./domcheck.sh filtered.txt

# Final result in hit.txt - fully validated, existing domains

Large List Processing

# Process large lists efficiently
python domfilter.py --input huge_list.txt --output stage1.txt
./domcheck.sh stage1.txt 200  # Increase parallel processes

TLD Data Sources

DoFi pulls TLD data from authoritative sources:

IANA

Official IANA TLD list

Public Suffix

Mozilla’s public suffix list

WHOIS XML API

Supported gTLD database

Blackweb TLDs

Extended TLD appendix

Performance Tips

Python Script Optimization

For large domain lists (millions of entries):

• Run on systems with adequate RAM (4GB+ recommended)
• Use SSD storage for faster I/O
• TLD cache is created once and reused

Bash Script Optimization

For efficient domain checking:

• Increase parallel processes on powerful systems
• Use reliable DNS servers
• Consider network bandwidth limits
• Monitor system load during processing

# Monitor performance
./domcheck.sh large_list.txt 150 &
watch -n 1 'wc -l hit.txt fault.txt'

Combined Workflow

Optimize the full pipeline:

# Filter first (fast, local)
time python domfilter.py --input raw.txt --output filtered.txt

# Then check (slower, network-dependent)
time ./domcheck.sh filtered.txt 100

This approach minimizes network checks by filtering invalid entries first.

Use Cases

Blocklist Maintenance

Clean and validate domain blocklists before deployment

SEO Analysis

Verify domain lists for SEO tools and analysis

Security Research

Validate threat intelligence domain feeds

DNS Administration

Maintain clean DNS zone files and records

Troubleshooting

TLD Download Failures

If TLD sources are unavailable:

• Check internet connectivity
• Verify firewall/proxy settings
• Script will use cached TLDs if available

Domain Check Timeouts

If domain checks are slow or timing out:

• Reduce parallel processes
• Check DNS server responsiveness
• Consider using local DNS cache

# Test DNS performance
time host google.com

# Use fewer parallel processes
./domcheck.sh domains.txt 25

Memory Issues

For very large lists:

• Split input files into smaller chunks
• Process in batches
• Increase system swap if needed

# Split large file
split -l 100000 huge_list.txt chunk_

# Process each chunk
for file in chunk_*; do
  python domfilter.py --input $file --output filtered_$file
done

Best Practices

1. Prepare Input

   Clean input files before processing:

   • Remove http://, https://, www. prefixes
   • Convert to lowercase
   • Remove duplicates
   • One domain per line
2. Run Domain Filter

   Use Python script first to validate TLDs and remove overlaps

3. Run Domain Check

   Verify filtered domains actually exist via DNS

4. Review Results

   Check removed entries for false positives before finalizing

License