I’ve been stress-testing a YaCy peer while crawling at around 4000 pages/minute and noticed something interesting regarding the outgoing HTTP connection pool.
The default setting on my installation was:
General connection pool: 200
During sustained crawling I increased it first to 2000, and later to 4000 for testing.
Current observations:
Crawl rate: ~4000 pages/minute
Connection Pool
----------------
Maximum: 4000
Active: 68
Idle: 316
Pending: 0
Other observations:
- Open file descriptors: ~260
- File descriptor limit: 524288
- No socket exhaustion
- No significant TIME_WAIT accumulation
- Pending connections remain at 0
What surprised me is that the pool can maintain well over 300 idle reusable connections while the operating system only has a relatively small number of active TCP sockets. That suggests the connection pool is not simply holding open hundreds of sockets.
With the previous limit of 200, I wondered whether YaCy might have been forced to recycle connections more aggressively during heavy crawling. Since increasing the limit, the pool has operated comfortably without any pending requests.
Questions
- Has anyone else experimented with increasing the outgoing connection pool?
- Is the default value of 200 still appropriate for today’s servers and crawl rates?
- Would a higher default (for example 1000 or 2000) improve performance on modern systems without causing unnecessary resource usage?
- Are there any downsides to a larger pool that I may have overlooked?
I’m interested in hearing from anyone running large crawlers or high-throughput peers.