After fine tuning crawler I need to update my figures.
https://community.searchlab.eu/t/auto-blacklisting-crawl-delay-offenders/3361
I’m running a solid 2000 ppm anyway im going to run out of space very soon.
Disk burn rate
-
Per day: 111 ÷ 2 = 55.5 GB/day
-
Per week: 55.5 × 7 = 388.5 GB/week (~0.39 TB/week)
-
Per 30-day month: 55.5 × 30 = 1,665 GB/month (~1.67 TB/month)
-
Per year: 55.5 × 365 = 20,257.5 GB/year (~20.3 TB/year)