[REBOL] Having fun with msnbot (was: REBOL.org Outage)
From: SunandaDH::aol::com at: 2-Jul-2004 6:22
Hallvard:
> The msnbot works hard on indexing the whole of www.oops-as.no. We've had
the
> msnbot around every 10 seconds for about two months now. This is no problem
> for the server, but could turn out to be one for the bot. The oops server
has
> only got about 10 different documents. The stuff that the bot fetches, is
> parsed through the distorter (www.oops-as.no/roy/dis). And so it seems msn
is
> aiming at downloading the whole internet through the distorter.
MSNbos isn't the only misbehaved entity out there that might come and suck
your website and its bandwidth dry.
REBOL.org has just been attacked (it seems the appropriate word) by HTTRACK
-- an off-line viewer for websites. It decided to download nearly half-a-meg a
minute for two and a half hours.
I guess the good news is that it didn't crash REBOL.org -- so the zombie bug
is not *entirely* related to volume of activity. Like the Altme recycle bug,
it's exact etiology remains a mystery.
> Maybe I ought to forbid the msnbot throught the robots.txt file. But then
> again...
At least you can now preview MSNbot's search pages:
http://techpreview.search.msn.com/
It looks like a throwback to the days before Google, when SEs indexed
strictly according to the words on the page. Can only get better, I hope.
Sunanda.
