I've been looking for a good whois scraper/harvester.

For a start, I need something that can check 10k specific domains w/o getting blocked every 8-15 records. Or at least can change its ip address after every 5-10 records pulled.

hidemyass.c0m/vpn/ can change my ip every xx minutes. That's as close as i've come to finding a way to do this. But still, xx minutes isn't good enough. I still get blocked. I need to change more frequently than by xx minutes. I need to change by the # of records or cycle through multiple whois systems(?).

I may be going about this all wrong.

But if not, do you have any idea of how to go about doing whois lookups w/o getting banned/blocked every handful of search results? Or know of an all-in-one bh tool that can do the harvesting for me en masse?

Oh. Eventually I will need to lookup 250k+ specific domains. This initial 10k test group will determine if is worth the effort or not to continue with the idea.

Thanks

Advertise

I can program you a small application that will do this. PM me if your interested.

thanks. Pm sent.

OMR Whois Scraper

The software works now, i just need a few details from you.
Like how you want the list format to be.
it changes to a new proxy built in every search so there is no chance of being locked out.

If the one posted above works thats fine, just let me know what you want me to do - if you still want the one i coded let me know, and any additional features you want added.

Hey doctorwar , Just realized i had grabbed your whois scraper from the other thread a bit ago (blackhatworld.c0m/blackhat-seo/member-downloads/124531-get-onlinemarketresults-whois-email-scraper-v1.html#post1168895). It gets blocked too, at least for me

New version would be killer if it could

1) work with list of proxy addresses instead of a single proxy address and

2) could rotate ip address every 4 or so requests, since 4 requests per minute per ip seems to be the max allowed for most registrars.

Rotating the ip every third or fourth request would bypass the 4 per minute limitation. Could then grab whois records as fast as your connection speed and pc could handle. Would work provided the ip changes every 3-4 requests or cycles in a way that the same ip is never used more than 3-4 times in any given minute.

That would be a killer whois scraper!!!

Hey doctorwar, just realized i had grabbed your whois scraper from the other thread a bit ago (blackhatworld.c0m/blackhat-seo/member-downloads/124531-get-onlinemarketresults-whois-email-scraper-v1.html#post1168895). It gets blocked too, at least for me

New version would be killer if it could

1) work with list of proxy addresses instead of a single proxy address and

2) could rotate ip address every 4 or so requests, since 4 requests per minute per ip seems to be the max allowed for most registrars.

Rotating the ip every third or fourth request would bypass the 4 per minute limitation. Could then grab whois records as fast as your connection speed and pc could handle. Would work provided the ip changes every 3-4 requests or cycles in a way that the same ip is never used more than 3-4 times in any given minute.

That would be a killer whois scraper!!!

Earlier I wrote:

"hidemyass.c0m/vpn/ can change my ip every xx minutes. That's as close as i've come to finding a way to do this. But still, xx minutes isn't good enough. I still get blocked."

I meant to type "I'd still get blocked" instead of "I".

When I get a few free minutes I can update this whois scraper to use proxy lists with delays/rotation

i can put u in the right direction if u r still looking for this tupe of bot

__________________
Understand the value of TIME and keep working hard till you drop.

In the mean time, what about using proxyfirewall to rotate proxies?

Seems something more custom, more specific to whois searches is needed [like your software and howdoyou's software].

On my end i'm finding registrars limiting searches to 4 per minute before temporarily banning or delaying my ip. So maybe cycling connections not by the minute but by the number of whois requests will get past autobanning.

Also, I can't find a setting to make proxyfirewall or hidemy@$$ cycle per # of requests. Too bad. Because I want access to that 'free' whois data!!! lol

I thought of maybe requesting the data direct from registrars so i could setup a new search engine but really with the goal of 'hacking' my own database remotely so i could claim someone else did it -hehe - but I bet i'd unknowingly leave a trail and still get caught. Oh well. Gotta do things legit...by scraping.

Maybe i'm overlooking something(?).

Interesting

d@mn! Not finding a scraper that can avoid getting banned every fourth request is costing me millions in possible opportunity losses. Too bad I'm too dang broke to do anything about it - lol

Maybe i will setup my own search engine and create a 'reason' for internic to share all that private data with me. Ok. Bad idea.

If anyone has or knows of a tool that can extract whois contact data into csv format and won't time out/get blocked after 3 or 4 requests, please hook a brotha up

thx

How about one of the xml/json providers:

http://www.whoisxmlapi.com/?gclid=CM...FRhfagod4nRUqw

I'm looking for something similar as well. Anyone have any new ideas about getting unlimited who is info? I would appreciate it.

Since this is an old thread I thought I mention it here again.

Thank you

Have your tried TOR (the onion router).
I use it at work to scrape piratebay and works great.

TOR is very risky and hideous software. I don't recommend using it - there are way better alternatives for anonymity.

ScrapeBox has a whois scraper in it.

__________________
There is nothing so stable as change. - Bob Dylan