Hello everyone,
I have been working on YagoSeek, a self-hosted, YaCy-compatible peer-to-peer search node written in pure Go.
Repository: https://github.com/D4rk4/yago/
Demo: https://yagoseek.dev/
Despite the deliberately provocative title, this is not a line-by-line port of the Java YaCy codebase. It is a new implementation of a decentralized search node that speaks the YaCy network protocols and can participate in the existing YaCy swarm.
YagoSeek can:
-
maintain and search its own local web index;
-
crawl websites using separate crawler workers;
-
exchange RWI postings and URL metadata with YaCy peers;
-
perform remote searches across the YaCy network;
-
bootstrap from YaCy seed lists and participate in peer discovery;
-
expose YaCy-compatible search endpoints;
-
provide a Tavily-compatible API;
-
serve a customizable public search portal;
-
run its administration interface without requiring JavaScript.
The node and crawler are separate Go binaries: yago-node and yagocrawler. A basic installation can be started using Docker Compose, and a new node automatically generates and persists its peer identity on its first run.
The search engine combines a sharded Bleve index with federated YaCy swarm searches. Results from different sources are merged using reciprocal-rank fusion and diversified using MMR. It also supports familiar YaCy query operators such as site:, filetype:, language:, quoted phrases, exclusions and date-oriented searches.
The crawler supports more than ordinary HTML pages. It can extract content from PDF, DOCX, XLSX, PPTX, OpenDocument files, RTF, EPUB, Markdown, CSV and several legacy office formats. It normally uses a lightweight HTTP fetcher but can fall back to headless Firefox for JavaScript-rendered or otherwise difficult pages.
Operationally, the project includes:
-
Docker and Docker Compose deployment;
-
Debian packages and hardened systemd units;
-
Prometheus metrics and health endpoints;
-
scoped API keys and Argon2id administrator authentication;
-
storage quotas;
-
backup and restore tooling;
-
SSRF protection for crawler traffic;
-
automated unit, integration and containerized end-to-end tests;
-
release artifacts for amd64 and arm64.
The current release work has focused heavily on search and crawler reliability: preserving completed local results when a peer or web-search branch fails, bounded multilingual ranking, reducing crawler and storage contention, improving PDF extraction, and making the administration interface report the real state of the node.
There are some intentional differences from the original YaCy implementation. The administration interface is new rather than a clone of the Java pages, authentication uses sessions and scoped API keys, and several Java-specific or obsolete interfaces are deliberately not implemented. The compatibility matrix currently records 30 implemented YaCy surfaces, five partially implemented surfaces and five explicit non-goals.
This is still alpha software. It runs on real nodes and is covered by tests, but it needs broader real-world interoperability testing, especially against different YaCy versions, larger indexes and long-running swarm participation.
I started it primarily as a self-hosted project for my own infrastructure. Like many decentralized and homelab projects, it is intended to be usable on an ordinary home server or a small VPS rather than requiring a large hosted platform.
I would particularly appreciate feedback from the YaCy community on:
-
interoperability with existing Java YaCy peers;
-
RWI and URL-metadata exchange behaviour;
-
remote search compatibility;
-
crawling and indexing differences;
-
missing protocol surfaces that are genuinely useful;
-
deployment on small or unusual systems.
Bug reports, packet captures demonstrating incompatibilities and test cases are all welcome.
The project is licensed under AGPL-3.0. It also owes a great deal to YaCy and its two decades of work on decentralized search.