lazlopanaflex
Junior Member
- Oct 15, 2020
- 176
- 52
Scrapebox not working on Google search
- Thread starter lazlopanaflex
- Start date
GoDesignerNetworks
Registered Member
- Oct 1, 2020
- 52
- 16
Sounds like a big challenge for Scrapebox.
lazlopanaflex
Junior Member
- Oct 15, 2020
- 176
- 52
Without basic google search scraping, I think the software becomes almost worthless
I hope they solve this soon.
- Sep 17, 2008
- 6,037
- 3,974
To get it working it will need to be a headless browser. Gone are the days of hitting Google with hundreds of threads on a basic computer. A part of one of my tools is affected, luckily it is a much smaller part in mine than it is in SB.
- Aug 9, 2017
- 5,579
- 6,868
The Scrapebox Google harvester will still work if you add a cookie header to the engine and include some pre-farmed cookies, but this will only work for very slow scraping, and even then will have a limited lifespan as you have to stop Scrapebox completely to change cookies.
- Sep 17, 2008
- 6,037
- 3,974
What cookies need entering? Just any from a JS enabled search results page? From what you're saying it shouldn't be too hard to figure out but it's always good to gather relevant info before jumping into it myself.
I'd really rather stay away from instantiating a browser for every thread, if that is even possible / likely to work for the long term.
I have thought on this topic many times over the years, what happens if Google uses a complex JS algorithm to prove identity. With how protective they are with their SERPs I do wonder why they took so long to do this. They must think that now it is finally worth ditching backwards compatibility for really old devices and the few that roll without JS on.
For me this is like when Yahoo did away with their API, that was such a sad time for developers like myself.
- Sep 17, 2008
- 6,037
- 3,974
It's a challenge for many more people than just SB
- Aug 9, 2017
- 5,579
- 6,868
It appears to be the NID and SOCS cookies. Open up your VPS, open a browser and run a few searches on Google, then copy the resulting cookies for the Google.com domain. With my limited testing, including them in basic GET requests serves up parseable html, but there will be limits reached sooner or later. Farmed cookies might become a scraping expense if you don't wanna go full Javascript.
The Yahoo API was the backbone of my first adsense spam network - great days!
- Sep 17, 2008
- 6,037
- 3,974
I've been playing around with HTTP Debugger, I can see the headers required and am just looking at the best way to grab valid cookies, or even if that is necessary. I would imagine this is going to have some pretty significant issues for high thread counts.
I dunno if you're into coding yourself but i'm thinking the easiest way would be to fire up a browser component instance, perform a search to get the cookies and then switch back to sockets with the spoofed headers but I don't like using full blown browsers when not necessary.
I'm lucky that most of my solutions using SERPs, these days, aren't their main focus. Dedicated SERP scraper apps are going to be worse going forward imo.
Yahoo API was the shit. I made The Free Blog Commenter back in the day using that. I dunno if it was before your time but it was similar to Blog Comment Demon. All available options since then have been inferior and costly in comparison.
northlondon
Newbie
- Nov 13, 2012
- 36
- 10
I'm gutted I've been using SB for nearly ten years, this week I'm launching a new managed email marketing service but I've had to put the launch on hold now because I've got no tool to scrape the email lists for the clients campaigns
ElGrecco
Newbie
- Jan 19, 2024
- 14
- 5
I've just opened ScrapeBox, updated it to the latest version and this statement appeared on my screen...
It's quite an issue, though I think SB Devs will solve it up as fast as they can.
During my web scraping with SB I saw problems with keywords from google around 5 months ago, so it could be a problem that has been around for a while.
I'm still positive there's something being done
It's quite an issue, though I think SB Devs will solve it up as fast as they can.
During my web scraping with SB I saw problems with keywords from google around 5 months ago, so it could be a problem that has been around for a while.
I'm still positive there's something being done
Attachments
- Aug 9, 2017
- 5,579
- 6,868
A version of Scrapebox capable of javascript execution/emulation would be quite the game changer, especially after all these years!I've just opened ScrapeBox, updated it to the latest version and this statement appeared on my screen...
It's quite an issue, though I think SB Devs will solve it up as fast as they can.
During my web scraping with SB I saw problems with keywords from google around 5 months ago, so it could be a problem that has been around for a while.
I'm still positive there's something being done
yes this works, the question is for how long. I did something similar: used a headless browser to do a single google search, saved the cookies and then reused the cookies with simple requests(non-js) and it worked.
Another thing you can do is simply change the user agent to one of these: https://github.com/benbusby/whoogle-search/wiki/User-Agents , and the search will work without JS, but you won't have all data, like the number of results and SERP features. This is working for now, but I don't expect it to work well forever, since if everyone uses these agents it will become clear they are bots.
- Sep 17, 2008
- 6,037
- 3,974
SHhhhhhhhhh!
I fixed it in my tools on Sunday night. There is a workaround that gets all results still. Like you said though, it probably won't last forever.
Jesus, as this is SB main features, you'd think they'd have been on top of a change like this.I've just opened ScrapeBox, updated it to the latest version and this statement appeared on my screen...
It's quite an issue, though I think SB Devs will solve it up as fast as they can.
During my web scraping with SB I saw problems with keywords from google around 5 months ago, so it could be a problem that has been around for a while.
I'm still positive there's something being done
northlondon
Newbie
- Nov 13, 2012
- 36
- 10
I tried a couple of the user agents but its not working on my SB.
I need a scraper as a temporary replacement while they are updating SB. Today I'm going to test this script that I found on Apify, I don't know how good it is https://apify.com/apify/google-search-scraper
- Aug 9, 2017
- 5,579
- 6,868
$3.50 per 1k requests is on the pricier end of the market. ValueSERP is at $2.50 per 1k requests (I can attest to this currently working), or Serper.dev is probably the lowest at $1 per 1k requests for the smallest package (though I'm not sure if they're accepting new clients at the moment)
We've already been through this several times in the thread already.
northlondon
Newbie
- Nov 13, 2012
- 36
- 10
I did'nt know about ValueSERP and Serper.dev nice one mate I'll check them out
Would you mind providing some guidance on how to add valueserp and serper.dev to scrapebox? I opened some trial accounts to test it but...
What is the proper naming convention with respect to IP, port, username/pass or API when adding to scrapebox?
I appreciate the assistance.
- Aug 9, 2017
- 5,579
- 6,868
Sorry, I don't use those in Scrapebox, I access them directly via the API. It might be worth asking the support of each of the services to see if they have a solution for Scrapebox available. It will probably involve adding a new harvester engine definition. I know ScraperAPI have an integration (but I've not tested it with the current iteration of the SERPs): https://www.scraperapi.com/integration-tutorials/scrapebox-integration/
You can also look at @mihai1497 's service here, and see if they have a solution: https://www.blackhatworld.com/seo/serp-data-api-for-developers-serpextractor-com.1669161/