Does yacy use compression when indexing webpages? Because I’m not sure if it does. It’s sometimes slow and bandwidth usage is significantly more then the storage space used. If not supported, could brotli and gzip be added? Brotli is very efficient and most websites support it, and gzip could be for compatibility for the rare sites that don’t.
1 Like
Hi, you mean bandwidth http compression while crawling?
personally, i got no idea, @orbiter would know that.
if not, it’s a more than reasonable idea! maybe optional, given by a system setting, while compression can take some CPU time and some people might like to tune bandwidth/cpu load ratio.
are you into java and able to implement or not?
if not, you can start issue for that at github and hope someone would do that.