Large systems can reach a point where collecting everything is no longer the main problem. Deciding what to leave out can matter as much.
The first Google index in 1998 already had 26 million pages, and by 2000 the Google index reached the one billion mark.
The Curation Threshold
Google's blog explained that many pages have multiple URLs with exactly the same content or URLs that are auto-generated copies of each other. It went on: "Even after removing those exact duplicates, we saw a trillion unique URLs". The post added, "We don't index every one of those trillion pages".
The difference between finding an address and choosing to index it is worth noticing. In general terms, when a collection grows very large, processing everything in it may cost more than it returns, and a count of what is stored says little about what is useful.
One reading is that filtering, deciding what to discard before it enters a system, becomes a central design task at large scale.
Collecting more than is needed can slow a system down and raise its costs. A designer may ask where in a system relevance should be judged, and by what rules. Clear rules for what to include can help, though they bring their own risk of leaving out something that later turns out to matter. Different systems strike that balance in different ways, and the right balance can change as a collection grows.
Perhaps scale is measured less by how much a system holds than by how well it chooses what to ignore.
Sources
- We knew the web was big... - Official Google Blog, 2008-07-25
