Subject: How to get rid of it?
Author: IpGuru
In response to: It's due to the Omni-crawler
Posted on: 06/22/2006 04:05:11 AM
Here is how to get rid of it:
The Omni-Explorer agent is: OmniExplorer_Bot. To prevent it from crawling your site, simply put the following in your robots.txt file:
User-Agent: OmniExplorer_Bot
Disallow: /
That's it.
>
> On 06/22/2006 03:59:01 AM
IpGuru wrote:
You are not alone. Tons of website got targeted by the Omni-crawler.
Actually, starting from the IP Lookup
http://www.allaboutip.com/ip_routing.html
and input 65.19.150.233, you will get the following info:
IP routing lookup results for 65.19.150.233
route: 65.19.128.0/18
descr: Hurricane Electric
55 South Market St
San Jose, CA
origin: AS6939
notify: noc-dist@he.net
mnt-by: HE-NOC
changed: mtindle@he.net 20040309
source: RADB
This is the company that developed the crawler. Definitely, this crawler is not as smart as Google crawler. Instead, it was working too hard -- resulting nothing but troubles. Here is a more complete list of IPs it may use:
1x 65.19.150.218
1x 65.19.150.219
3x 65.19.150.224
1x 65.19.169.250
1x 65.19.150.226
1334x 65.19.150.245
1x 65.19.150.227
2278x 65.19.150.207
2x 65.19.150.221
8x 65.19.150.213
2x 65.19.150.223
1815x 65.19.150.214
1x 65.19.150.215
561x 65.19.169.233
3x 65.19.150.252
8x 64.71.131.113
1x 65.19.150.220
1x 65.19.169.245
References:
