Technology

This reminds me of that one time a guy figured out how to make "gzip bombs" that bricked automated vuln scanners.

Read without sign-up

I suspect that there are many websites that already dynamically generate an unbounded number of pages based on the links one clicks, and that Web spiders will have needed to deal with those for as long as there have been people spidering the Web, which is going to be no later than the first Web search engines.

I'd guess that if nothing else, they cap how far they spider a site. Probably a lot more sophisticated, use heuristics to figure out which sites are more worth spending indexing resources on, as it's not just whether to spider but also the frequency with which to do so. Some parts of a site are more "valuable" than others -- for a search engine, a more desirable target for users clicking on results -- and some will update more frequently and are more-useful to re-spider at higher frequency. Google will return current news articles, yet still indexes a large portion of the content out there. They won't be doing that by simply sending GoogleBot at everything that they've indexed at a fixed frequency.

This genus named genius game is sending pain to these previous devious data devourors

The modern equivalent of making a page that loads in two frames, left and right, which each load in two frames, top and bottom, which each load in two frames, left and right ...

As I recall, this was five lines of HTML.

it might he useful to generate text on the random urls then test different repetitions to see of you can leave a mark on the training data... So after X repetitions or injected information, release the bot back into the wild with whatever message or false info you want it saddled with.