Is there a way to set filters for URLs to crawl ?
While crawling it finds new urls and explores them. I’d like to avoid crawling websites by me choice, for example porn*.webiste.org or any website on domain .sex etc.
1 Like
Yes…. and I had the same desire. You can blacklist websites. An easy and fast way to get started is to download a blacklist source from another peer that has already done this.