Internet Marketing Toolbox

Jack Humphrey has had enough! Enough of the bandwidth hogging Yahoo crawler, Slurp. So much so that he’s re-written his robots.txt file to ban Slurp from his “Friday Traffic Report” servers completely!

And if you haven’t already noticed, you really should have a good long look at your log files just to see what effect Slurp might be having on your sites.

I responded to Jack’s refreshing post “Yahoo Slurp Has Been Banned From FTR!” on June 9th, and this is what I said there…

Jack, I encourage EVERYONE to look at their log files. These figures are from my awstats figures for one of my sites in May… and I’m appalled by it!

Yahoo Slurp - 29274+2618 HITS - 395.71 MB bandwidth
MSNBot - 13963+1722 HITS - 207.43 MB
Googlebot- 12539+9 HITS - 188.87 MB
(the + figure is # of hits on robots.txt file)

However, my visitors from each search engine during May tell another story:
Yahoo! - 2,407 visits
MSN Search - 1,726 visits
Google - 57,576 visits

So far in June for EIGHT days I have:
Yahoo Slurp, 12269+1109 hits for 165.14 MB = 431 visits
MSNBot, 4077+399 hits for 62.02 MB = 394 visits
Googlebot, 2986+9 hits for 44.42 MB = 14,603 visits

Another one of my sites Slurp hit 70953+4405 times, consuming 1.23 GB of bandwidth to send me 307 visitors in May.

Is it really worth putting such a HUGE extra load on your server to allow Slurp in? And since I have 20 or so sites on that VPS, it’s working a lot harder than it should!

If you want to keep Slurp, then try their crawl delay recommendation.

But I’m with you Jack! Slurp’s goooone!
Stephen

Some other posters argued that bandwidth is cheap, so why worry? Well, while it might be cheap in the good ole US of A, here in Oz we STILL only get miniscule allowances with our local hosting (and YES, before you say it, I DO host ALL my Aussie sites overseas - I could never afford to pay for the traffic they generate if hosted locally).

But it’s more than bandwidth… it’s the total unnecessary load on the server generated by all these Slurp hits that don’t result in anything much at all… I’d much rather NOT have my server do so much extra work if at all possible.

It’s just like sp@m… unwanted!

Popularity: 78% [?]