@zooom Yes, I do. Crawler code itself is opensource - https://github.com/tb0hdan/domains-crawler - just file reader and TLDs used to configure it are not. There are bugs (as always) but I’m working on getting them fixed.
I’ve used additional sources as crawler input to speed up dataset growth, all of them are listed in dataset readme.
Regarding subdomains - there are some limits in place, still I wanted to have those as well to allow for others to have doorway detection. I’m working on
autovacuum process that
will filter invalid (i.e. expired) domain names.
Regarding domainlists.io - I strongly believe that domain list should be publicly available and not sold.
@TheHolm Yes, your approach with
nmap seems to be the best so far.