I captured a Java heap dump from a heavily crawling/indexing YaCy peer shortly before shutting the server down.
At the time, the peer had roughly:
- 219 GB
DATA/INDEX/freeworld - ~6.5 GB Java heap in use
- ~12–13 GB process RSS
- ~1.1 GB swap in use
- ~645 threads
- very heavy crawling/indexing activity
The resulting HPROF was 8.8 GB (8.27 GiB). I analysed it using Eclipse Memory Analyzer (MAT) 1.17.
What MAT found
The Leak Suspects analysis identified three particularly large retained-memory areas:
1. SearchEventCache.lastEvents — ~783 MB retained (~23.7%)
Most of this appears to be byte[] data retained through a LinkedHashMap.
Approximately:
- 442,499
byte[] - ~768 MB in those arrays
- 442,598
String - 35,148
LinkedHashMap$Entry - associated
DigestURL,SearchEvent,URIMetadataNode, etc.
2. IndexCell — ~893 MB retained (~27.1%)
This appears to be the RWI/index working set:
- ~1.92 million
byte[]— ~822 MB - ~640,000
ReferenceContainer - ~640,000
ConcurrentHashMap$Node
The dump also showed the associated IndexCell.FlushThread.
This may simply be normal behaviour during heavy indexing, rather than a leak.
3. HostBalancer — ~348 MB retained (~10.5%)
This was associated with the crawler host queues:
- 567
HostQueue - 1,468
RowSet - 1,468
BufferedObjectIndex - 4,800
byte[]consuming ~347 MB
A crawler thread was captured executing through HostQueue.pop() / HostBalancer.pop().
Interesting result
A class histogram taken before the heap dump showed about 6.58 million byte[] objects consuming ~2.25 GB (decimal) of shallow heap.
MAT now gives us an explanation for a large part of that memory:
SearchEventCache ~768 MB byte[]
IndexCell ~822 MB byte[]
HostBalancer ~347 MB byte[]
-------
~1.94 GB
So the large byte[] usage isn’t one single allocation source. A substantial amount comes from search-event caching, RWI indexing and crawler queues.
Question
The part I’m most curious about is SearchEventCache.lastEvents.
Is retaining roughly 783 MB / 24% of the analysed heap in SearchEventCache.lastEvents expected behaviour?
Does this cache have a size/age eviction mechanism, and would this amount of retained data be considered normal on a heavily crawling peer?
I’m not claiming this is necessarily a memory leak — the peer was under significant crawl/index load when the dump was captured. I’m mainly trying to determine whether these retained-memory sizes are expected, particularly SearchEventCache.lastEvents.
I still have the original HPROF and MAT indexes, so I can run additional MAT queries if particular retention paths would be useful.