Dav Posted May 19, 2007 Report Posted May 19, 2007 9 guests, 8 members, 0 anonymous membersDav, Google.com, Falcoknight, the_accountant, Rowen, Top_Gun, Yahoo.com, SeVeR, Suicide_Run, No_Remorse, MSN.com Hopefully this will mean more people downloading the game
»Maverick Posted May 20, 2007 Report Posted May 20, 2007 hm indeed Well as long they don't hog the bandwidth I don't think they are much of a problem
Dav Posted May 20, 2007 Author Report Posted May 20, 2007 i was just getting at the high search engine activity means (hopefully) more people will find us. I'm sure they take up next to no bandwidth doing their thing.
rootbear75 Posted May 20, 2007 Report Posted May 20, 2007 i think half of those guests are also robots from other search engines.
PoLiX Posted May 20, 2007 Report Posted May 20, 2007 Nah. If you saw the Robots list I have entered... yeah, haha. It is a cleaned up version of the "ULTIMATE ROBOTS LIST" but still, it has most of em. Googlebot hits us the most. In 10mins we had 86 hits from it yesterday, 4 from yahoo, and 1 from msn. I had to disable the logs, cause last time there was almost 500000 from Googlebot alone.
Gravitron Posted May 20, 2007 Report Posted May 20, 2007 That's silly.All they do is archive the page 20 times a minute, leech bandwidth and eat system resources.At least the google one seems to pay attention to robot.text and follow guidelines so it doesn't rape your page with updating its cache (although it WILL rape the server unless told not to, so fix your bot.txt properly).Aside google, the only other bot you're going to want to allow access is web.archive.org, any other bot should be IP banned from AXS, website-wide.
»Maverick Posted May 21, 2007 Report Posted May 21, 2007 Aside google, the only other bot you're going to want to allow access is web.archive.org, any other bot should be IP banned from AXS, website-wide.Erm, but wouldn't SSForum then be found by any other search website apart from google?
Gravitron Posted May 21, 2007 Report Posted May 21, 2007 I'm speaking from resources standpoint, you don't want more than these two bots to crawl/cache/refresh/b-w leech your site.All the other engines are deploying dataminers to crawl on google, so basically once you're listed on google by their bot, you're listed on everyone.
Recommended Posts