Work Text:
Chapter 4: Polite by design (the SRE part)
Chapter Text
The Internet Archive is a nonprofit running shared infrastructure, so the archiver runs on an explicit politeness budget, the same way you would budget error rates for a production service:
- It always leaves two of your Save Page Now capture slots free, so a greedy queue never locks you out of archiving something by hand.
- It caps itself at 5 attempts per page per day, half of what the service allows.
- When throttled, it backs off exponentially, up to a full day.
- It asks the server to skip pages archived within the last 30 days instead of burning a capture on a duplicate.
None of these numbers are guesses; they are written into the config and verified against it in the README's diagrams. The tool is built to never lose a URL, even when that makes it slow.