moonshine7000
Elite Member
- Mar 4, 2013
- 4,386
- 4,014
But the majority of online redirect checkers fail except one
Scraping Google about to get even harder?
- Thread starter Steptoe
- Start date
TimDawsone
Regular Member
- Jul 21, 2017
- 216
- 202
Everyone is looking at this from the Ahrefs and Semrush side, which is fair, (data for seo is cheaper btw ) but there is a more direct hit for this section of the forum that nobody has mentioned yet. Link building tools that scrape their own targets.
GSA SER, Scrapebox, all of them lean on search engine footprints to find places to post. Every one of these changes lands on that workflow too. num=100 going away already meant ten times the requests for the same target list. Now if every result URL has to be resolved through a redirect on top, that is another request per result, more proxy burn, more captchas, more bans. The cost of building your own target list has gone up something like 10x in about a year while the quality of what you scrape has not improved at all.
I stopped scraping years ago for exactly this reason, before it got this bad. Time and proxy costs were already more than a list subscription. Now the gap is silly. i run ser verified lists as a base across all my tiers, gsa link surge for my tier 1 and 2 projects, and have used ser links and power lists at various points. Each one has its ups and downs and none of them is perfect, but $30 to $60 a month against what scraping now costs in proxies alone is not a hard sum. The list providers still have to eat these scraping costs of course, so i would expect prices to creep up, but they spread it across hundreds of subscribers and you don't.
The wider point for tools like Ahrefs is right though. Anything built on cheap SERP data is going to get more expensive or less fresh, probably both.
GSA SER, Scrapebox, all of them lean on search engine footprints to find places to post. Every one of these changes lands on that workflow too. num=100 going away already meant ten times the requests for the same target list. Now if every result URL has to be resolved through a redirect on top, that is another request per result, more proxy burn, more captchas, more bans. The cost of building your own target list has gone up something like 10x in about a year while the quality of what you scrape has not improved at all.
I stopped scraping years ago for exactly this reason, before it got this bad. Time and proxy costs were already more than a list subscription. Now the gap is silly. i run ser verified lists as a base across all my tiers, gsa link surge for my tier 1 and 2 projects, and have used ser links and power lists at various points. Each one has its ups and downs and none of them is perfect, but $30 to $60 a month against what scraping now costs in proxies alone is not a hard sum. The list providers still have to eat these scraping costs of course, so i would expect prices to creep up, but they spread it across hundreds of subscribers and you don't.
The wider point for tools like Ahrefs is right though. Anything built on cheap SERP data is going to get more expensive or less fresh, probably both.