YacyGrid working, but now what?

I finally got around to try to switch to YacyGrid instead of Yacy. I set it up on a windows machine first because of the great development tools. Would of moved it to docker later once I see it work. I realized before I started that there wasn’t a single example of a front end for it. Or how to do anything beyond a simple crawl. I went ahead anyway figuring it would all make sense when I got to this point. This actually took many days because the gradle build java and python files are out of date with the latest. So after hours of prodding it and looking for some hints. I see that not even susper made a proper frontend for it. The project seems it hasn’t been touched in years. nor has yacy for that matter. It not just a matter of working out the rest calls, it uses Warc files. and I don’t see any fine tuning settings for the crawler. It uses RabbitMQ which I would have to dive into this.
Is YacyGrid a dead project? I mean it seems at this moment a truly scalable search engine would be really important right now. If not, how about some examples of some client front-ends? and how to manage and maintain the index?

First of all, thank you very :green_heart: much for trying out YaCy Grid and the feed-back!
Well, YaCy Grid is a new approach to make a crawler using cherry-picking of ‘legacy YaCy’ components. This was done for a commercial customer as consulting after I promoted the concept in public.

The customer still exists and I make changes on the code base only if the customer accepts changes specific concepts. That does not mean that library updates are not done if required, on the contrary, I do this as support whenever it looks like it should be done.

The future of that project exists certainly with enhancements to the queuing concept (something else than rabbitmq) and the storage (probably s3 instead of ftp). Furthermore a front-end concept was made, but this is only very rudimentary.

So no, this is not dead at all, it is just handled in a different way.

2 Likes