My first server crash in 24 hours just happened. So, I looked at the logs.
Someone performed quick, local searches on my peer. One of their search queries:
yacy.space
Yacy then complied with admin settings and temporarily blocked an IP address from rapid-fire local searches. The IP address:
This redirects to yacy.space.
For now, my firewall is blocking that IP address, in an effort to reduce server crashes.
ufw insert 1 deny from 157.245.217.151
I had a look and between codex and myself, we found out what was causing the crashes:
Close owned remote Solr HTTP clients - #809
Basically, in certain circumstances, yacy opens a thread to handle a new process, and when it is done, it doesn’t close the thread. When one of my nodes crashed, there were almost 39,000 threads left open, not on purpose. It hit the hard thread limit specified in systemd, and couldn’t create any new ones. After running a couple of nodes with the code patch in place for about 24 hours, and 5000-6000 searches on each from yacy.space:
Current counts:
… before the code change the Connection evic count was 38,393 in only a few hours, so it looks like it is now properly releasing the threads it no longer needs, adn we have this one issue beat. Hope @Orbiter sees it the same way. 
Anyone knows when a new official YaCy release get’s pushed?
Sorry, I have no idea, but in order for this question to be seen, probably best to be in its own thread. He might see it that way.
Checked:
Result:
No sign of the old runaway thread behavior.
There is only one file changed:
source/net/yacy/cora/federate/solr/instance/RemoteInstance.java
How could the rest of us implement this changed code without waiting for something to change on github?
This looks more complicated than it is. If you don’t have the build tools required, already on your system, this will take care of that as well. The only thing you should need to know is if you have it installed as a service or not, and where the install is located, then follow the steps below:
The source change is only in:
source/net/yacy/cora/federate/solr/instance/RemoteInstance.java
But editing that file in a running YaCy install will not change anything by itself. YaCy runs compiled classes from lib/yacycore.jar, so you need to build a patched yacycore.jar, stop YaCy, replace the jar, and start YaCy again.
The patch is PR #809:
On Linux Mint, the build path should look roughly like this:
sudo apt update
sudo apt install git ant default-jdk
git clone https://github.com/yacy/yacy_search_server.git yacy-pr809-build
cd yacy-pr809-build
git fetch origin pull/809/head:pr-809
git checkout pr-809
ant compile
After that, the patched jar should be:
lib/yacycore.jar
Now find your YaCy install directory. It is the directory containing startYACY.sh, stopYACY.sh, DATA/, and lib/yacycore.jar.
For a normal user-run YaCy install, for example:
export YACY_HOME="$HOME/yacy_search_server" # adjust this path
"$YACY_HOME/stopYACY.sh"
cp -a "$YACY_HOME/lib/yacycore.jar" \
"$YACY_HOME/lib/yacycore.jar.before-pr809-$(date +%Y%m%d-%H%M%S)"
install -m 0644 lib/yacycore.jar "$YACY_HOME/lib/yacycore.jar"
"$YACY_HOME/startYACY.sh"
If YaCy is installed as a system service, adjust the paths and ownership. Example:
sudo systemctl stop yacy
sudo cp -a /opt/yacy/lib/yacycore.jar \
/opt/yacy/lib/yacycore.jar.before-pr809-$(date +%Y%m%d-%H%M%S)
sudo install -o yacy -g yacy -m 0644 lib/yacycore.jar \
/opt/yacy/lib/yacycore.jar
sudo systemctl start yacy
To check whether the runaway thread issue is under control:
pid=$(pgrep -f 'net.yacy.yacy' | head -n1)
grep -E '^(Threads|VmRSS|VmSize):' "/proc/$pid/status"
for f in /proc/$pid/task/*/comm; do
cat "$f" 2>/dev/null
done | grep -c '^Connection evic$'
A restarted node will always start with a low count, so the real test is whether Connection evic stays low after searches continue. On our patched nodes it stays around 3; before the patch we saw tens of thousands.
Rollback is simple: stop YaCy, restore the backed-up yacycore.jar, and start YaCy again.
The actual code fix is small: RemoteInstance now tracks whether it owns the HTTP client created by HttpClientUtil.createClient(params), and close() calls HttpClientUtil.close(this.client) only for that owned client. It intentionally
does not close custom/shared client paths.
… If you are nervous about any of this, don’t do it. If you decide to go ahead, backups are your friend. The instructions will create a backup of the changed file, but I would backup the yacy install (maybe without the data depending on drive space limits) just in case. Also, just for fun, run the command to show your current thread usage before you apply the patch.
Also, @roamn installed this on a machine and left a comment on github that they were testing it. Let’s see if they will pop in here and let us know how it is going?
So far it has done exactly what I hoped for across all my nodes, but they are all servers running the same software stack, so it would be nice to hear from others. I mean, it is java, so it should work the same everywhere, but…?
I tried cloning the latest build using git, and it broke my project. I was glad for the backup I had just made, so I restored it. The original code is working. Next, I will try your guide, but it’s too late tonight for me to try that. The good news is that I have not seen a server crash in awhile.
Yes there was a build error which in now fixed.
#812
Thank you. A machine in my care is now patched. I had to update it first to openjdk-17-jdk, but then the code compiled without errors.
Thanks for that info, I am sure that will help several people going forward. As for the runaway thread count patch, it has been ingested into the main, so hopefully, unless it exposes a new problem, that will be the last time any of us have to deal with it. I feel stupid. 15 years ago, the last time I tried yacy, I ran into the same issue, but thought I didn’t know enough Java to tackle it. Turns out it was an extremely simple fix, and I am ashamed I didn’t try it then. Oh well, I’m back now.
I will probably work on some more critical bugs and make a new one in early september.