[Journey] Making Scrapebox Google scraping great again (with some code magic)

[satyr85]

Hey folks

I’m making this thread mostly for myself. I was about to finish and release what I have multiple times, but something more important alwaaaays came up and stole my time and focus.
Scrapebox feels pretty abandoned these days - the latest updates and the beta harvester aren’t much to look at from a programmer perspective. The browser-based approach (in current form) doesn’t solve scraping problems, and honestly it doesn’t feel like a step forward.

Sure, scraping isn’t the hot niche it used to be… but let’s be real, people still want to quickly pull links from Google or check indexation,

The plan:

• Finish integration of the old harvester (the one that works without a browser) with my backend, so scraping requests is redirected there, all the scraping magic happens server-side, and users get a clean, pre-parsed response. No bans, no captchas, nearly no failed queries.
• Stress test Google scraping under the new rules. Since Google killed the 100 results per page option, it now takes 10x more requests to grab 100 results than it did last month. Thanks, Google
• Fire up a BST in the marketplace.
• ???
• Profit

Backup plan (if Scrapebox drives me insane):

• Build and release (maybe for free) my own Google harvester/index checker.

Timeline:

• With motivation: 2–4 weeks.
• Without motivation: lol, never.

So yeah, hit me with some motivation, folks Memes work too

 

[satyr85]

Like this post if you want to be tagged in final thread update

 

[bonzo90]

good luck with this project. is there a reason you are using scrapebox instead of something more modern seems there are a million sites that will scrape for you now with ai scrapers out.

 

[willyfast]

I am sure you will achieve your goals. Find something that is important to you and attach your feeling with your current goal

 

[satyr85]

good luck with this project. is there a reason you are using scrapebox instead of something more modern seems there are a million sites that will scrape for you now with ai scrapers out.

Sure, there are many sites, but these cost a ton as you have to pay for every request.

I am sure you will achieve your goals. Find something that is important to you and attach your feeling with your current goal

Congrats on your first post mate and thanks for kind words

 

[ActualOutreach]

Hey folks

I’m making this thread mostly for myself. I was about to finish and release what I have multiple times, but something more important alwaaaays came up and stole my time and focus.
Scrapebox feels pretty abandoned these days - the latest updates and the beta harvester aren’t much to look at from a programmer perspective. The browser-based approach (in current form) doesn’t solve scraping problems, and honestly it doesn’t feel like a step forward.

Sure, scraping isn’t the hot niche it used to be… but let’s be real, people still want to quickly pull links from Google or check indexation,

The plan:

• Finish integration of the old harvester (the one that works without a browser) with my backend, so scraping requests is redirected there, all the scraping magic happens server-side, and users get a clean, pre-parsed response. No bans, no captchas, nearly no failed queries.
• Stress test Google scraping under the new rules. Since Google killed the 100 results per page option, it now takes 10x more requests to grab 100 results than it did last month. Thanks, Google
• Fire up a BST in the marketplace.
• ???
• Profit

Backup plan (if Scrapebox drives me insane):

• Build and release (maybe for free) my own Google harvester/index checker.

Timeline:

• With motivation: 2–4 weeks.
• Without motivation: lol, never.

So yeah, hit me with some motivation, folks Memes work too

It's getting very difficult to scrape Google with scrapebox, did you find a solution for it?
I tried to change to yahoo search engine and it's the only thing that works for me atm

 

[Kuzno]

may be you should check A-Parser?

 

[shijin]

Stress test results will be gold, I’m curious if Google soft bans faster now with all the extra requests.

 

[satyr85]

It's getting very difficult to scrape Google with scrapebox, did you find a solution for it?
I tried to change to yahoo search engine and it's the only thing that works for me atm

Yeah, I have solution. Hopefully it will be offered as service for Scrapebox (where you dont need to buy proxies for scraping, only access to service). Ofc if all goes well and I put this together instead of chasing projects.

may be you should check A-Parser?

Hell no, I've build myself scraper better than a-parser. Its about bringing access to my solution (or similar solution) for everyone who bought Scrapebox.

Stress test results will be gold, I’m curious if Google soft bans faster now with all the extra requests.

Google did serious changes when it comes to proxy bans/blocks in last 2-3 years. Its all because of AI.
Scraping serps alone didnt cost google that much computing power. Now you have AI response in many serps - this cost google a ton. So they do their best to make scraping harder.

2-3 years ago I was able to reach 25k SERPs per minute easily (assuming 100 results per SERP it was 2.5 milion urls max scraped per minute).
Now I need to push myself to reach 10k SERPs a minute (now its 10 results per SERP so 100k urls max scraped per minute total).

By stress test I meant stress testing my own infrastructure (servers, proxies etc) to see whats should I improve.


Here is video from today's small test. CPU used is ryzen 7 4700u. As you can see bandwidth usage is little (considering scraping speed) - thats because of pre-parsed results feature

Will need to run more of these tests, and for longer time. Todays test was successful

 

[ciphercipher1]

hey are you looking for partners? what is your tech stack? i would love to hear more

 

[satyr85]

hey are you looking for partners? what is your tech stack? i would love to hear more

Depends what you can bring to the table
Backend for this one is python, postresql and redis.

 

[picapiedra18]

Your end reward:

 

[myblogginglab]

I can be your first customer if you bring this as Service

Love to See you are doing new things with abandoned software

 

[Ranking Expert]

Any update? @satyr85

 

[StormClaw]

I was breaking my head for the last 4-hours with ScrapeBox thinking why it's not scraping from Google, never knew about the Google Scraping issue as I wasn't using it for the last 1 year, thankfully found this thread to clear my mind.

@satyr85 Did you drop the effort? :|

 

[satyr85]

Any update? @satyr85

Sorry for late update, missed this one.

I was breaking my head for the last 4-hours with ScrapeBox thinking why it's not scraping from Google, never knew about the Google Scraping issue as I wasn't using it for the last 1 year, thankfully found this thread to clear my mind.

@satyr85 Did you drop the effort? :|

Nope, I didnt drop my efforts. I've had to move my focus on something else.
I expect my proxy solution for google scraping with SB will be ready this year. Probably around christmass.

 

[StormClaw]

Sorry for late update, missed this one.

Nope, I didnt drop my efforts. I've had to move my focus on something else.
I expect my proxy solution for google scraping with SB will be ready this year. Probably around christmass.

Thanks for the update. No rush, take your time. If a fix comes by Christmas, that would be the best Winter Gift for all SB users.

 

[Zher0]

Sorry for late update, missed this one.

Nope, I didnt drop my efforts. I've had to move my focus on something else.
I expect my proxy solution for google scraping with SB will be ready this year. Probably around christmass.

Any updates on this?