spiders ip list
- Thread starter nickk
- Start date
Grizzy
Senior Member
- Nov 11, 2008
- 920
- 1,005
I used to use
but it looks like it hasn't been updated in a while... I've found it's hard to find and keep good updated lists unless your constantly lurking on webmaster forums.
I think the problem with spider ip lists, is that the big SEs like G are constantly changing their spider's IP ranges. So it's often better to identify spiders by user agent (this is even what big G recommends).
For that I have been using
and
Code:
hxxp^^w3*iplists*comI think the problem with spider ip lists, is that the big SEs like G are constantly changing their spider's IP ranges. So it's often better to identify spiders by user agent (this is even what big G recommends).
For that I have been using
Code:
hxxp^^spiderlist*codeforgers*com/list.php
Code:
hxxp^^w3*user-agents*org