Либо по HTTP_USER_AGENT либо по IP.
ip:
Код:
[Turtle]
195.161.113.10|scanner-001.turtle.ru
195.161.113.12|scanner-003.turtle.ru
195.161.113.13|scanner-004.turtle.ru
195.161.113.15|scanner-006.turtle.ru
195.161.113.16|scanner-007.turtle.ru
195.161.113.17|scanner-008.turtle.ru
195.161.113.18|scanner-009.turtle.ru
195.161.113.20|scanner-011.turtle.ru
195.161.113.40|scanner-012.turtle.ru
195.161.113.41|scanner-013.turtle.ru
195.161.113.42|scanner-014.turtle.ru
195.161.113.43|scanner-015.turtle.ru
[Yandex]
213.180.194.135|zhivete.yandex.ru
213.180.194.136|zelo.yandex.ru
213.180.194.137|zemlya.yandex.ru
213.180.194.138|izhe.yandex.ru
213.180.194.139|i.yandex.ru
213.180.194.141|ludi.yandex.ru
213.180.194.143|nash.yandex.ru
213.180.194.144|on.yandex.ru
213.180.194.145|omega.yandex.ru
213.180.194.146|pokoj.yandex.ru
213.180.194.148|slovo.yandex.ru
213.180.194.151|fert.yandex.ru
213.180.194.158|yer.yandex.ru
213.180.194.159|yery.yandex.ru
213.180.194.161|yu.yandex.ru
213.180.194.165|little-yus.yandex.ru
213.180.194.163|ya.yandex.ru
213.180.194.167|ksi.yandex.ru
213.180.194.168|psi.yandex.ru
213.180.194.169|fita.yandex.ru
213.180.194.171|ayb.yandex.ru
213.180.194.172|ben.yandex.ru
213.180.194.173|gim.yandex.ru
[Inktomi]
66.196.65.12|si3001.inktomisearch.com
66.196.65.13|si3002.inktomisearch.com
66.196.65.14|si3003.inktomisearch.com
66.196.65.16|si4000.inktomisearch.com
66.196.65.17|si4001.inktomisearch.com
66.196.65.18|si4002.inktomisearch.com
66.196.65.23|si3006.inktomisearch.com
66.196.65.24|si3007.inktomisearch.com
66.196.65.25|si3008.inktomisearch.com
66.196.65.28|si3011.inktomisearch.com
66.196.72.19|j3109.inktomisearch.com
66.196.72.22|j3112.inktomisearch.com
66.196.72.25|j3115.inktomisearch.com
66.196.72.39|j3129.inktomisearch.com
66.196.72.40|j3130.inktomisearch.com
66.196.72.47|j3137.inktomisearch.com
66.196.72.54|j3144.inktomisearch.com
66.196.72.60|j3150.inktomisearch.com
66.196.72.67|j3157.inktomisearch.com
66.196.72.73|j3163.inktomisearch.com
66.196.72.86|j3176.inktomisearch.com
66.196.72.94|j3184.inktomisearch.com
66.196.72.103|j3193.inktomisearch.com
66.196.72.108|j3198.inktomisearch.com
66.196.73.11|j3401.inktomisearch.com
66.196.73.15|j3405.inktomisearch.com
66.196.73.16|j3406.inktomisearch.com
66.196.73.18|j3408.inktomisearch.com
66.196.73.21|j3411.inktomisearch.com
66.196.99.20|si520.inktomi.com
[Google]
216.239.46.27|crawl1.googlebot.com
216.239.46.41|crawl2.googlebot.com
216.239.46.80|crawl3.googlebot.com
216.239.46.106|crawl4.googlebot.com
216.239.46.133|crawl5.googlebot.com
216.239.46.151|crawl6.googlebot.com
216.239.46.171|crawl7.googlebot.com
64.68.82.5|crawler10.googlebot.com
64.68.82.47|crawler11.googlebot.com
64.68.82.66|crawler12.googlebot.com
[Fast]
66.77.73.140|cr001r01.sac2.fastsearch.net
66.77.73.141|cr002r01.sac2.fastsearch.net
66.77.73.254|fpcr003.sac2.fastsearch.net
66.77.73.204|cr001r01- test.sac2.fastsearch.net
[Altavista]
64.152.75.199|buildrack69.sv.av.com
64.152.75.205|buildrack75.sv.av.com
[Looksmart]
209.249.67.146|209.249.67.146.looksmart.net
[???]
212.69.115.194|relay.bix.ru
[Gigablast]
217.75.103.130|130.eu.gigablast.com
HTTP_USER_AGENT :
Код:
[Rambler]
StackRambler/1.4
[Yandex]
Yandex/1.03.000 (compatible; Win16; M)
Yandex/ 1.03.003 (compatible; Win16; D)
Yandex/1.01.001 (compatible; Win16; I)
Addurl/2.0
[Turtle]
TurtleScanner/1.2 (compatible; MSIE 5.01; Win32)
[???relay.bix.ru]
URLTracer_Looksmart_Utility
[Gigablast]
Gigabot/1.0
[Altavista]
Scooter/3.3
[Google]
Googlebot/2.1 (+http:// www.googlebot.com/ bot.html)
[Looksmart]
Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http:// www.WISEnutbot.com)
[Inktomi]
Mozilla/5.0 (Slurp/ cat; slurp@inktomi.com; http:// www.inktomi.com/ slurp.html)
Slurp/si (slurp@inktomi.com; http:// www.inktomi.com/ slurp.html)
[Fast]
FAST-WebCrawler/3.6 (atw- crawler at fast dot no; http:// fast.no/support/crawler.asp
FAST-WebCrawler/ 3.6/FirstPage (crawler@fast.no; http://fast.no/ support.php?c=faqs/ crawler)
FAST-WebCrawler/3.2 test
Списки не полные конечно.
А лучше не париться и прописывать кук сразу на "содержательной странице"