🚀 YaCy Search Performance Boosted Using RAM Disks (`tmpfs`) — Still Long Startup, but Worth It

YaCy
1

Hey YaCy folks,

Just sharing a quick update on how I improved search speed and interface responsiveness in my multi-agent YaCy setup using simple RAM disk mounts. Startup still takes over an hour, but runtime performance has definitely improved.

:hammer_and_wrench: System Specs:

  • Server: HP DL360 Gen8
  • Memory: 384 GB ECC RAM
  • Storage: 10 × 1TB Kunup SSDs in RAID 5
  • Index Size: ~75 million links (~1 TB)
  • YaCy Agents: agent-asx, agent-smokingwheels
  • OS: Ubuntu 24.04 (plus some services from Debian Bookworm)

:bulb: What I Did:

Mounted system temp folders in RAM using tmpfs:

fstab

tmpfs /tmp      tmpfs defaults,size=16G,mode=1777 0 0
tmpfs /var/tmp  tmpfs defaults,size=16G,mode=1777 0 0

I will try the experiment on an old i5 PC with hdd 32 GB a 14 year old pc. maybe just 2 GB for tmpfs!

YaCy was then left to use these locations naturally (no extra JVM flags or config hacks needed).

:zap: Results:

:white_check_mark: Searches and UI respond much faster
:white_check_mark: JavaScript-based interface loads smoother
:clock3: Startup still takes 60+ minutes, but search usability is noticeably improved
:floppy_disk: System temp writes no longer stress disk I/O — things feel lighter

If you’ve got spare RAM, even just mounting /tmp and /var/tmp as tmpfs can make a difference from my experimental server.

Cheers from the TurboNerdZone™,

1 Like
2

Seems crazy but promising!
Regarding the 1-hour start: what is the start-up time without tmpfs?
Do you watch the log messages during startup? What does take the most of time?

if you’re using RWIs they are sometimes merged upon startup, which usually takes a lot of time and IO. See docs.

3

Yes they are all 1.8 TB

4

that’s a lot!

i’ve already dedicated one separate instance just to RWI distribution: when there’s a several tens of gb on a primary instance, i move the RWI to distribution instance, to keep the RWIs slender on the primary one. the latter needs much more RAM as well.

that’s a workaround, i suspect kelondro for being bit inefficient and possibly could be tuned.

maybe you could ommit/reduce the RWIs while experimenting…

5

Maybe If I had a place to back it up.
The last backup is strung over 2 disks. aka RAID 0

6

or, you can move the RWI index files (DATA/INDEX/freeworld/SEGMENTS/default/text.*) outside of yacy directory tree and return them later. new ones would be created afterwards, but only as big as you crawl and you can delete them or just merge the dirs. i move them around regularly.