Hi all! Long story short, this spring I found out about existence of YaCy, tried it out and got fascinated about it’s idea. Reference implementation has it’s flaws and since I had some free time I decided to spend it on building prototypes where I reimplemented parts of YaCy functionality as microservices while addressing problems I cared about.
I’d like to show you the demo of one of those prototypes here - https://search.my-yacy-node.org/. It allows you to search over freeworld DHT network and it’s main advantages are quality and relevance of results, especially on multi-word queries, and speed of search. Also when you click on a search result it’s page would be automatically crawled and indexed by local full text index so next time the same query would show results both from DHT and full text index. I’d be really glad if you try it and leave your feedback about it.
Would be happy to tell you more in the comments if you find it interesting.
Hi @nikitakarpei, welcome to the YaCy community! Great to see someone exploring new approaches to searching the network.
I tried a search on your demo, and it seems to be missing over 100 URLs compared with the same query on my own YaCy peer. Does the prototype limit the number of results or query fewer peers?
Thanks for sharing your work—I’d be interested to hear more about how it handles DHT searches and multi-word queries.
Hi! Yes, this prototype limits amount of shown results at 20, sorted by the relevance score. It could be increased but their relevance after second page would most likely be low. The reason of that is that this prototype usually finds a few thousands documents from peers and to assess their relevance properly we should fetch live versions of that pages. Fetching thousands of pages would probably take a lot of time, more than average user would probably like to wait for a search to complete. So I first sort results from peers based on relevance signals they can provide (not much though, mostly URL address and titles), select top-50 results, fetch their live pages and then resort them based on new relevance signals once again. So it’s a balance between speed and quality which is hard to achieve but I’d keep experimenting to find what else is achievable.
Multi-word queries work differently from how YaCy performs them. For example, let’s say our query is “one two three” and we have three peers - A, B, C. YaCy, if I understand everything correctly, would ask all three peers the same question “what documents with words one AND two AND three you have in your RWI index?”. This prototype instead tries to guess, based on the peers’ position on DHT ring, which peers most likely to hold documents with word “one”, then with word “two” and then with word “three”. For example, let’s say A is most likely to hold docs with word “one”, B with word “two” and C with word “three”. It then asks each peer what documents they have for one given word they’re most likely to hold docs for. Each peer provides list of documents and prototype then keeps only ones which all three peers returned.
I have to say this idea is not something I invented, actually there was an attempt to build similar algorithm in original YaCy and I couldn’t have built it if protocol wasn’t designed for this case. But for some reason it seems like it wasn’t fully implemented. This algorithm theoretically should be able to find more results because RWI index of each peer is design to only answer the question “what documents with words I’m responsible for based on my peer hash I hold?” It’s not guaranteed to hold information about words it’s not responsible for and if DHT distribution is enabled they would be sent to other peers anyway.
