ConcurrentLog shutdown of ConcurrentLog.Worker: injection of poison message

YaCy
1

It seems my yacy index grown so much that it can no longer start up… is this possible?

I 2021/06/04 12:05:12 HeapReader using a dump of the index of /home/els/yacy/DATA/INDEX/webportal/SEGMENTS/default/citation.index.20210604070657571.blob.
W 2021/06/04 12:05:20 ConcurrentLog net.yacy.cora.util.SpaceExceededException: 194610220 bytes needed for RowCollection grow after OutOfMemoryError Java heap space: 621142592 free at Fri Jun 04 12:05:20 CEST 2021
net.yacy.cora.util.SpaceExceededException: 194610220 bytes needed for RowCollection grow after OutOfMemoryError Java heap space: 621142592 free at Fri Jun 04 12:05:20 CEST 2021
        at net.yacy.kelondro.index.RowCollection.ensureSize(RowCollection.java:276)
        at net.yacy.kelondro.index.RowCollection.addUnique(RowCollection.java:425)
        at net.yacy.kelondro.index.RowCollection.addUnique(RowCollection.java:403)
        at net.yacy.kelondro.index.RAMIndex.addUnique(RAMIndex.java:216)
        at net.yacy.kelondro.index.RAMIndexCluster.addUnique(RAMIndexCluster.java:133)
        at net.yacy.kelondro.index.RowHandleMap.<init>(RowHandleMap.java:112)
        at net.yacy.kelondro.blob.HeapReader.initIndexReadDump(HeapReader.java:179)
        at net.yacy.kelondro.blob.HeapReader.<init>(HeapReader.java:91)
        at net.yacy.kelondro.blob.HeapModifier.<init>(HeapModifier.java:58)
        at net.yacy.kelondro.blob.ArrayStack.<init>(ArrayStack.java:209)
        at net.yacy.kelondro.rwi.ReferenceContainerArray.<init>(ReferenceContainerArray.java:68)
        at net.yacy.kelondro.rwi.IndexCell.<init>(IndexCell.java:99)
        at net.yacy.search.index.Segment.connectCitation(Segment.java:189)
        at net.yacy.search.Switchboard.<init>(Switchboard.java:594)
        at net.yacy.yacy.startup(yacy.java:209)
        at net.yacy.yacy.main(yacy.java:812)
I 2021/06/04 12:05:20 HeapReader generating index for /home/els/yacy/DATA/INDEX/webportal/SEGMENTS/default/citation.index.20210530113515632.blob, 3874 MB. Please wait.
W 2021/06/04 12:08:57 ConcurrentLog net.yacy.cora.util.SpaceExceededException: 139007280 bytes needed for RowCollection grow after OutOfMemoryError Java heap space: 424077128 free at Fri Jun 04 12:08:57 CEST 2021
net.yacy.cora.util.SpaceExceededException: 139007280 bytes needed for RowCollection grow after OutOfMemoryError Java heap space: 424077128 free at Fri Jun 04 12:08:57 CEST 2021
        at net.yacy.kelondro.index.RowCollection.ensureSize(RowCollection.java:276)
        at net.yacy.kelondro.index.RowCollection.addUnique(RowCollection.java:425)
        at net.yacy.kelondro.index.RowCollection.addUnique(RowCollection.java:403)
        at net.yacy.kelondro.index.RAMIndex.addUnique(RAMIndex.java:216)
        at net.yacy.kelondro.index.RAMIndexCluster.addUnique(RAMIndexCluster.java:133)
        at net.yacy.kelondro.index.RowHandleMap.putUnique(RowHandleMap.java:290)
        at net.yacy.kelondro.index.RowHandleMap$initDataConsumer.call(RowHandleMap.java:494)
        at net.yacy.kelondro.index.RowHandleMap$initDataConsumer.call(RowHandleMap.java:433)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
I 2021/06/04 12:10:40 ConcurrentLog shutdown of ConcurrentLog.Worker: injection of poison message
I 2021/06/04 12:10:40 HeapReader using existing gap dump instead of writing a new one: citation.index.20210604070657571.blob.pgztlU9f4t1k.gap
I 2021/06/04 12:10:40 HeapReader using existing idx dump instead of writing a new one: citation.index.20210604070657571.blob.pgztlU9f4t1k.idx
I 2021/06/04 12:10:40 ConcurrentLog terminating ConcurrentLog.Worker with 3 cached loglines.
I 2021/06/04 12:10:40 ConcurrentLog shutdown of ConcurrentLog.Worker: terminated
2

Yes, you must assign more memory in memory settings!

3

Does this mean that RAM usage depends on index size, and that YaCy will continue increasing the index until it hits that limit, after which it will crash and never recover unless I purchase more RAM or move the YaCy instance to another machine…?

4

I increased ram limit to 2GB: after few days, I faced again the irrecoverable crash.
This seems indeed the way yacy works: it grows the index until it runs out of memory, the kamikaze way. If you cannot rise the RAM, your yacy instance is gone along with your index.