Serp scraping

R

ricky0161

Newbie
Joined
Aug 23, 2014
Messages
2
Reaction score
1
Hello

We have a very basic SERP tracking software to monitor our own serps nothing special but does what we need it too and it runs on our servers so we don’t have to do any thing once running

now I keep getting blocked from google to many requests even when we use recaptcha solving solution

now I’m guessing it’s because every thing is coming from the same ip (school boy error )

now my question is

should I be using proxies or should I just use the google and bing api works out

we have about 10k of search terms we need to check a week over google and google mobile and bing

if we should be using proxy’s how many should we have so we wouldn’t get flagged as spam or is there another way around this

thanks in advanced hope this makes sense

rick
 
Advertise on BHW
Hi Ricky
Google is so strict when it comes to scraping. Your proxies will get banned so fast by Google when you're scraping(even if they are dedicated ones) especially when Footprints are used along with keywords(not in your case btw).

backconnect proxies are what's needed for yor purpose. There's one gateway and each request is made by one of the proxies which are in the pool. But it's gonna cost at least $100/Month
Bare in mind that Free proxies are not useful when you're scraping and it's totally waste of time. They get burned too fast because many users are scraping them as well
Best of luck
 
I'm hoping to do the same, but on a smaller scale.
In my case, I'll be searching Google using the default browser- Chrome in my case. Perhaps in the future this'll be a headless browser, should it work the same, but for now this is what I will do, and what you could consider doing yourself:
-Searching Google using the default browser ON your server (over scraping)
-Getting the results from that page with ADBlockers OFF

There are over 600k seconds in a week, so you can hit your goal of 100k searches a month doing one search every- what- 10 seconds, with sleeps overnight?
 
If you can do one search every 60 seconds, you will probably not be blocked by Google, that's 10 K per week

I have checked indexation from a lot of URLs with no proxy using PHPSEO and 60seconds delay.
 
You better use ahrefs for that. Tracking with your own ip most of the times get you not actual results
 
It will also depend on if you are using operator searches also. These will get you banned very quickly.
 
You'll need a ton of proxies and set a delay or use a backconnect service that doesn't have all their IP pool banned.
 
Why do people spend time reinventing the wheel when rank trackers and their APIs are so readily available?
 
macdonjo3 said:
Why do people spend time reinventing the wheel when rank trackers and their APIs are so readily available?
Click to expand...
Good point. Maybe he thinks he's cutting down on costs? Those APIs are pretty reasonable though.

But yes OP, if you want to use your own tool you should definitely be using proxies.
 
you need 2 proxies for this work specifically,
10k a week means 1 request per minute, but since you gonna fetch both google and google mobile, you'll need 2 different ips and you're good to go
 
KraftyKyle said:
Good point. Maybe he thinks he's cutting down on costs? Those APIs are pretty reasonable though.

But yes OP, if you want to use your own tool you should definitely be using proxies.
Click to expand...
Proxies will be expensive. APIs are so readily available. SERP trackers are basically a race to the bottom when it comes to pricing. If it's too expensive, you can always find someone much cheaper. Some rank trackers even own a few different brands to cater to different budgets with the same value. It's a really weird space.
 
this is best service i aslo screp all social media data
 
How difficult would it be to create a serp scraper that can provide the same level of service as an Ahrefs or SEMRush? By difficult I mean how expensive would it be to create and then maintain? And can anyone point me in the direction of someone who would know how to build it?
 
What data exactly you want to scrape?

I used to do that to check how meta titles and descriptions were being picked up by Google, but then i decided to pay for thruuu which is way more reliable and less time consuming.

Otherwise there's SEOlium for keyword tracking. I have Ahrefs which is good too, but more expensive if you don't need a full SEO suite and SERPs don't always get updated so frequently as you may need.

Hope this helps, but I'm curious about this too, so feel free to pm me as well if you like
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top