Enable Autocrawler: [on]
Deep crawl every: 4
Rows to fetch at once: 100
Recrawl only older than # days: 1
Get hosts by query: host_s:*.ru OR host_s:*.su OR language_s:ru
Shallow crawl depth (0 to 2): 2
Deep crawl depth (1 to 5): 3
Index text: [on]
Index media: [off]
There are 73047 documents in the local index.
According to the specified query regex, Solr finds 66323 documents.
Pressed Save.
Reloaded Yacy.
No signs of work Autocrawler are visible. Automatic indexing of pages does not occur, requests to the queue are not added.
What do you need to configure that Autocrawler starts working?
Сейчас я поставил версию 1.926 и в ней Автокраулер работает. Причем так интенсивно, что при слабом ресурсе способен приостанавливать индексацию запущенных заданий.
Hi Sviatoslav! (And everyone else who comes across this from your search engines as well!)
I believe I found the culprit on why the Autocrawler does not work, small minor issue that causes a soft-crash, which aborts the whole autocrawling for any subsequent documents.
I submitted a PR to the YaCy github repo with a change that I believe will fix this issue, feel free to follow the PR if you’re still interested here: Fix autocrawler crashing by zutto · Pull Request #650 · yacy/yacy_search_server · GitHub