Failed. Reason: scraper cannot load URL: REJECTED EMPTY RESPONSE BODY 'HTTP/1.1 405 '

YaCy
1

Hi all

Im new to yacy, and testing the crawler im really surprised in how it works, im in love :P.

im getting this error even when lauching the crawler and in the crawl check, in one specific web.

failed. Reason: scraper cannot load URL: REJECTED EMPTY RESPONSE BODY 'HTTP/1.1 405 ’

can anyone tellme what im doing wrong please?

thanks in advance.

1 Like
2

in my experience, some CDN deny access, if you’re imitating GoogleBot. then REJECTED EMPTY RESPONSE BODY is returned.

3

How i can change that, in the advanced crawler?
thanks

4

robot behaviour> user agent
change from googlebot to yacy

that was my case, maybe your problem is different.

5

Maybe actually an error in the web site.
Just give generally genect feedback