Several machines I run in the Yacy project have been crashing in this past week. It appears that someone is flooding the network with repeated queries, which makes JAVA max out the CPU cycles, leading to overheating, and unresponsive servers.
I was able to stop the machines from crashing, if I blocked certain IP addresses from making connections… not with Yacy settings, but at the server level.
Surely, some of you have been seeing this also. What do you think? What is the source of the queries? I am not 100% sure, but here is what I noticed.
Command used to search the logs: grep --color=always -E 'yacy.space' ~/yacy/DATA/LOG/queries.log
What is this?
It looks like a useful tool, if it limits how many queries it sends to our network, to be reasonable, allowing machines to finish one query before starting another.
To see if it this is what is causing the crashes, I am temporarily blocking its IP address. It hides behind Cloudflare, making it difficult to find its direct IP address. But it is:
The command I am using to block it: ufw deny from 157.245.217.151
I hate to block anyone, but the repeated server crashes have caused me to investigate. There could be other machines that are responsible for the query floods. (It looked similar to a DDoS attack.) I will continue to monitor it and see if blocking this specific IP address helps the machines stay online and stop crashing.
Command used to search the logs: grep -a --color=always -E 'q="yacy.space"' ~/yacy/DATA/LOG/yacy*.log
By sampling some of the IP addresses, the name ImpossibleSearch comes up repeatedly. I wonder if a cluster of peers send out the same query at the same time. I appreciate anyone adding peers, with good motives, to the network, but maybe these need to be configured so they do not flood the network with the same query. And why search for “yacy.space” anyway? What are they wanting to find?
I’ve no idea why someone searched for yacy.space, but the queries got requested from 157.245.217.151 as local search, which results in queries to other peers as remotesearch to extend the searchresult. That’s why you see the same query from other hosts.
I guess the operator of https//yacy.space already reacted on this post
I see the Flag ‘resource=local’ now.
Having such a public portal like htts://yacy.space was discussed in the past. If I’m not wrong Orbiter set up a peer and promoted it as a demonstrator.
This one is not a peer, but a Host distributing the query and merging the result from requested peers.
I have a monitor program on one of my peers.
I’m receiving local searches and remote searches at the same time in bursts. However I dont have the yacy.space problem yet.
I have all the protection settings disabled or set to very high for testing.
My normal system load 100% CPU is equal to 24 and the the graph is plotting over 400+ for short period. The page is updated every 10 seconds. Its only a spike or someone plotting the response time.
The Ram usage (heap) goes to 95% so its ok for the moment.
Are you still running with what you stated in a previous post: RAM: 384 GB ECC?
I have several machines, and the one that is crashing the most today has only RAM: 5.8 GiB.
I suppose the server needs more RAM as it accumulates more documents. Your set-up has 10 times more documents than the server I’m troubleshooting. I think it’s the bursts that is overwhelming the machines, causing OutOfMemory.
