My current issue is that when I run YaCy from a remote computer and try to access it via another device it crashes fairly frequently on the hosting device. I don’t know what the conditions are for it to crash, but it seems to be accessing multiple pages at once. I can see from the pages taking a long time to load and the hosting devices’ CPU usage skyrockets to 70%+. This has been a problem since I’m currently trying to develop a web extension to improve YaCy’s functionality.
Is there a current fix to this issue? I’m running release: 1.924/10042 on win10.
I would ask for the get request information to restart YaCy in the affected computer, but it becomes completely unresponsive and must be killed via task manager.
The hardware shouldn’t be the issue. This had been running on worse hardware. The crashes are happening on this system:
i7 4790K
28gb DDR3
NVIDIA GTX 1070
But not on this system:
i7-4712MQ
8gb DDR3
Integrated graphics
Both are running in senior mode with crawling and the same version of 1.924/10042. I also noticed that the mouse pointer turns in to a black box when trying to type a search query if that’s any indication.
that seems to be a memory issue. one thing is RAM available on the machine, another is how much RAM is assigned to yacy. default value is pretty low. YaCy is RAM hungry, especially Kelondro crap.
See Performance Tuning. You can monitor RAM usage in First Steps > RAM/Disk usage. For me, usual value of MAX ram is something like 8 to 12 GB, with tens of millions pages in local index.
There was also a memory management improvement after 1.924 version, current github version is much better and stable. Unfortunately, @orbiter still didn’t make a release of 1.930, so a lot of people still download and use the old and buggy one, right away.
We have problems with the windows installer, so this is currently not maintained. You migrate by replacing either everything but the DATA folder with the new files or vice versa - move the DATA folder to the new release.
I’ve migrated the new update you gave but it fails to run on windows. Running the start batch file only opens and closes the command prompt and no log file is updated in DATA\LOG. Any ideas?
I’m running the jdk version 22, so yes. But it was broken somehow? I was getting errors getting the version number so I did a reinstall. I’ll try again
Here is the output that I was getting. I can run YaCy when deleting my data so I might just run fresh
[ YaCy v1.940 by Michael Christen / www.yacy.net ]
STARTUP: Trying to load logging configuration from file C:\Users\YaCy\DATA\LOG\yacy.logging
I 2024/06/21 17:01:07 UPNP * found device: RT-AC3200
I 2024/06/21 17:01:08 Word * hashCache.size = 20000
I 2024/06/21 17:01:08 Domain Init * local host address: 192.168.2.25 (local)
I 2024/06/21 17:01:12 Domain Init * local host address: 0::1 (local)
I 2024/06/21 17:01:12 Domain Init * local host address: 0:0:0:0:0:0:0:1 (local)
I 2024/06/21 17:01:12 Domain Init * local host address: 127.0.0.1 (local)
I 2024/06/21 17:01:12 Domain Init * local host address: fe80:0:0:0:f2bc:c28f:7d62:8e9a (local)
I 2024/06/21 17:01:12 Domain Init * local host address: fe80::f2bc:c28f:7d62:8e9a (local)
I 2024/06/21 17:01:12 SWITCHBOARD * Initializing Segment 'C:\Users\YaCy\DATA\INDEX\freeworld\SEGMENTS.
W 2024/06/21 17:01:12 UPNP * could not map mapped port 8090 to port 8090 on device RT-AC3200, external IP is 192.168.1.214
W 2024/06/21 17:01:12 UPNP * could not map mapped port 8090 to port 8090 on device RT-AC3200, external IP is 192.168.1.214
I 2024/06/21 17:01:12 Domain Init * local host address: fe80:0:0:0:27a5:1f20:5529:17fb (local)
I 2024/06/21 17:01:12 Domain Init * local host address: fe80::27a5:1f20:5529:17fb (local)
I 2024/06/21 17:01:13 HeapReader * saturation of text.index.20240619194626774.blob.oJUqvX7Pg3f-.idx: keylength = 7, vallength = 4, size = 1380017, Maximum saving for index-compression = 9 MB, exact saving for value-compression = 5 MB
I 2024/06/21 17:01:13 HeapReader * using a dump of the index of C:\Users\YaCy\DATA\INDEX\freeworld\SEGMENTS\default\text.index.20240619194626774.blob.
I 2024/06/21 17:01:13 HeapReader * saturation of text.index.20240619232056928.blob.voS9x-tQSFcq.idx: keylength = 7, vallength = 4, size = 383848, Maximum saving for index-compression = 2 MB, exact saving for value-compression = 1 MB
I 2024/06/21 17:01:13 HeapReader * using a dump of the index of C:\Users\YaCy\DATA\INDEX\freeworld\SEGMENTS\default\text.index.20240619232056928.blob.
I 2024/06/21 17:01:13 HeapReader * saturation of text.index.20240619232952047.blob.Jkp5m671huUM.idx: keylength = 5, vallength = 3, size = 1749, Maximum saving for index-compression = 0 MB, exact saving for value-compression = 0 MB
I 2024/06/21 17:01:13 HeapReader * using a dump of the index of C:\Users\YaCy\DATA\INDEX\freeworld\SEGMENTS\default\text.index.20240619232952047.blob.
I 2024/06/21 17:01:13 org.apache.solr.core.SolrXmlConfig Loading solr.xml from C:\Users\YaCy\defaults\solr\solr.xml
I 2024/06/21 17:01:13 Domain Init * local host address: fe80:0:0:0:f7d8:38ce:5dfe:fdce (local)
I 2024/06/21 17:01:13 Domain Init * local host address: fe80::f7d8:38ce:5dfe:fdce (local)
I 2024/06/21 17:01:13 Domain Init * local host address: 192.168.1.213 (local)
I 2024/06/21 17:01:14 org.eclipse.jetty.util.log Logging initialized @7486ms to org.eclipse.jetty.util.log.Slf4jLog
I 2024/06/21 17:01:14 org.apache.solr.core.CorePropertiesLocator Found 2 core definitions underneath C:\Users\YaCy\DATA\INDEX\freeworld\SEGMENTS\solr_9_0
I 2024/06/21 17:01:14 org.apache.solr.core.CorePropertiesLocator Cores are: [collection1, webgraph]
I 2024/06/21 17:01:14 org.apache.solr.core.TransientSolrCoreCacheDefault Allocating transient core cache for max 2147483647 cores with initial capacity of 1024
I 2024/06/21 17:01:14 org.apache.solr.update.SolrIndexConfig IndexWriter infoStream solr logging is enabled
I 2024/06/21 17:01:14 org.apache.solr.update.SolrIndexConfig IndexWriter infoStream solr logging is enabled
I 2024/06/21 17:01:14 Domain Init * local host address: fe80:0:0:0:be57:5dac:4721:2d1b (local)
I 2024/06/21 17:01:14 Domain Init * local host address: fe80::be57:5dac:4721:2d1b (local)
I 2024/06/21 17:01:14 Domain Init * finished
I 2024/06/21 17:01:15 org.apache.solr.update.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
I 2024/06/21 17:01:15 org.apache.solr.update.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
I 2024/06/21 17:01:15 org.apache.solr.update.CommitTracker Hard AutoCommit: if uncommitted for 180000ms;
I 2024/06/21 17:01:15 org.apache.solr.update.CommitTracker Soft AutoCommit: if uncommitted for 5000ms;
I 2024/06/21 17:01:15 org.apache.solr.update.CommitTracker Hard AutoCommit: if uncommitted for 180000ms;
I 2024/06/21 17:01:15 org.apache.solr.update.CommitTracker Soft AutoCommit: if uncommitted for 5000ms;
I 2024/06/21 17:01:15 org.apache.solr.metrics.SolrMetricManager Closing metric reporters for registry=solr.core.webgraph tag=SolrCore@3d35bf87
I 2024/06/21 17:01:15 org.apache.solr.metrics.SolrMetricManager Closing metric reporters for registry=solr.core.collection1 tag=SolrCore@164bd263
I 2024/06/21 17:01:15 org.apache.solr.update.DirectUpdateHandler2 Committing on IndexWriter.close() … SKIPPED (unnecessary).
I 2024/06/21 17:01:15 org.apache.solr.update.DirectUpdateHandler2 Committing on IndexWriter.close() … SKIPPED (unnecessary).
I 2024/06/21 17:01:16 SolrEmbeddedInstance * detected default solr core: collection1
E 2024/06/21 17:01:16 STARTUP * YaCy cannot start: SolrCore ‘collection1’ is not available due to init failure: Error opening new searcher
org.apache.solr.core.SolrCoreInitializationException: SolrCore ‘collection1’ is not available due to init failure: Error opening new searcher
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:2106)
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:2079)
at net.yacy.cora.federate.solr.instance.EmbeddedInstance.(EmbeddedInstance.java:106)
at net.yacy.search.index.Fulltext.connectLocalSolr(Fulltext.java:163)
at net.yacy.search.Switchboard.(Switchboard.java:585)
at net.yacy.yacy.startup(yacy.java:215)
at net.yacy.yacy.main(yacy.java:812)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.(SolrCore.java:1204)
at org.apache.solr.core.SolrCore.(SolrCore.java:1048)
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1560)
at org.apache.solr.core.CoreContainer.lambda$load$10(CoreContainer.java:950)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:259)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2426)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2559)
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1263)
at org.apache.solr.core.SolrCore.(SolrCore.java:1148)
… 9 more
Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path=“C:\Users\YaCy\DATA\INDEX\freeworld\SEGMENTS\solr_9_0\collection1\data\index\segments_cpi”))): This index was initially created with Lucene 7.x while the current version is 9.0.0 and Lucene only supports reading the current and previous major versions. This version of Lucene only supports indexes created with release 8.0 and later by default.
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:355)
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:299)
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:288)
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:1079)
at org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:145)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:110)
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:258)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:125)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2378)
… 12 more
E 2024/06/21 17:01:16 STARTUP * YaCy cannot start: SolrCore ‘collection1’ is not available due to init failure: Error opening new searcher
org.apache.solr.core.SolrCoreInitializationException: SolrCore ‘collection1’ is not available due to init failure: Error opening new searcher
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:2106)
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:2079)
at net.yacy.cora.federate.solr.instance.EmbeddedInstance.(EmbeddedInstance.java:106)
at net.yacy.search.index.Fulltext.connectLocalSolr(Fulltext.java:163)
at net.yacy.search.Switchboard.(Switchboard.java:585)
at net.yacy.yacy.startup(yacy.java:215)
at net.yacy.yacy.main(yacy.java:812)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.(SolrCore.java:1204)
at org.apache.solr.core.SolrCore.(SolrCore.java:1048)
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1560)
at org.apache.solr.core.CoreContainer.lambda$load$10(CoreContainer.java:950)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:259)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2426)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2559)
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1263)
at org.apache.solr.core.SolrCore.(SolrCore.java:1148)
… 9 more
Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path=“C:\Users\YaCy\DATA\INDEX\freeworld\SEGMENTS\solr_9_0\collection1\data\index\segments_cpi”))): This index was initially created with Lucene 7.x while the current version is 9.0.0 and Lucene only supports reading the current and previous major versions. This version of Lucene only supports indexes created with release 8.0 and later by default.
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:355)
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:299)
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:288)
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:1079)
at org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:145)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:110)
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:258)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:125)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2378)
… 12 more
embeded solr database engine is version 9 in yacy 1.940, while version 8 in yacy 1.930. according thee log error, version you used before was 7. so I’d try first upgrade to 1.930, and (after some time to convert the database) to 1.940. older releases could be downloaded here. did it help?