Communications of the ACM

You are searching for something.

You prepare yourself for an arduous journey. Before the first door you come upon a pedestal. The box that lies on the pedestal gives airs of gildedness despite being as plain as the walls that surround it. It isn't adorned with a title, but its name echoes in your mind, intuitively: the Answer Box. A plaque reads:

I have crawled through each and every door. Not just the doors in this hallway, but the doors in every hallway in existence, the doors within doors, as well as some doors that I dare not show you, doors that would make you flee in terror. I have seen everything. I am impartial. I have your best interests at heart. I understand what it is you want to know and it is knowable. I have the answer that you seek.

Your finger caresses the latch.

Cataloging the Web was doomed from the start. In the summer of 1993, Matthew Gray created the World Wide Web Wanderer (WWWW), arguably the first internet bot and web crawler. During its first official attempt to index the web, the Wanderer returned from its expedition with 130 URLs. But even in the baby years of the internet, this list was incomplete.