Scraping Google : how many requests /minute ?
- Thread starter Arold79
- Start date
GrizzlyBeardMan
Power Member
- Sep 16, 2017
- 626
- 311
Matheus Moreira
Power Member
- Jul 14, 2017
- 753
- 405
Im also interested in this... i think something between 30 to 90 seconds should be safe, assuming you didnt send too many queries before the first one in the sequence.
Something important to note is how many people are connected in the same wifi network as yours... in this case it would be safer to increase the delay even more.
Something important to note is how many people are connected in the same wifi network as yours... in this case it would be safer to increase the delay even more.
Tabun
Registered Member
- Jan 17, 2020
- 54
- 42
Yes but this doesn’t return the same links as regular Google. For instance it misses news, videos, and the featured answers.
@GrizzlyBeardMan thank you for your experience 
I will try to run Scrapebox one night with simple keywords and 90sec between each Google request.
@nikos90 you mean this API ? : https://developers.google.com/custom-search/docs/overview ?
I will try to run Scrapebox one night with simple keywords and 90sec between each Google request.@nikos90 you mean this API ? : https://developers.google.com/custom-search/docs/overview ?
nikos90
Junior Member
- Mar 9, 2011
- 167
- 47
Correct, I don't know if there is any setting to include that, but as far as I can see on the documentation it doesn't return such links
- Jan 25, 2009
- 6,378
- 3,774
The only answer I can give is try and see.Hello,
I would like to know how many request per minute can I do on Google without have a captcha please ?
Request will be simple keywords (no command like inurl: or site
I have a residential german IP.
Thank you for your help
Google does not disclose this information and my experience has been that they ban based on all sorts of stuff. For example if you have had lots of bans with them before on that ip, they ban faster then if you have not had lots of bans. Also they seem to ban residential ips slower and for less time then commercial ips.
Further they may ban based on activity of the other people on your subnet. The list goes on....
So basically I try an absurdly high delay, if it works, I keep dialing it back and lowering the delay until it stops working, then you have your answer. Wish I had a cut and dry answer, but with google things just are not cut and dry.
nikos90
Junior Member
- Mar 9, 2011
- 167
- 47
Yes, this is what I am talking about.@GrizzlyBeardMan thank you for your experience
@nikos90 you mean this API ? : https://developers.google.com/custom-search/docs/overview ?
canweoffshoreit
Junior Member
- Oct 26, 2014
- 134
- 43
Just use Scraper api
Bots-Scrapers
Elite Member
- Jan 2, 2014
- 4,041
- 1,078
1 to 10 seconds and rotating IP per request will be fine.
blogtaufiq
Regular Member
- Mar 22, 2011
- 237
- 32
i use ubot to scrape google with delay just 10s. if i found captcha then i will reset my modem and get new ip.
FLASH POINT
Junior Member
- Jun 9, 2020
- 128
- 17
Correct but this doesn't return the same links an google.
Sebastiann
Junior Member
- Apr 27, 2020
- 182
- 243
100% this... when you sign up for G Cloud you get a bunch of free credit most of the time, and there is a fairly generous free threshold on the custom-search.
Even after you go over it, the relative lack of pain is totally worth the couple of bucks you'll spend.
RebeccaThomas
Junior Member
- Jun 18, 2019
- 147
- 142
If you rotate your IP address you could in theory go as fast as you like. Even running 10 proxies would speed things up quite a bit.
As for the speed at which you can go on a static IP, Google is waaaay smart thesedays. There's no set time, infact if Google decides your IP is spam just watch how quickly it decides to throw a captcha!
IMHO, set up rotated IP addresses and run each one on a low burn, it's much easier in the long run - proxies don't cost all that much.
As for the speed at which you can go on a static IP, Google is waaaay smart thesedays. There's no set time, infact if Google decides your IP is spam just watch how quickly it decides to throw a captcha!
IMHO, set up rotated IP addresses and run each one on a low burn, it's much easier in the long run - proxies don't cost all that much.
VirtualSystems
Banned - selling outside of the marketplace.
- Oct 5, 2018
- 224
- 46
Not sure if it still works, but you could send queries to different google's top-level domains.
*.google.ac, *.google.ad, *.google.ae, *.google.af, *.google.ag, *.google.ai, *.google.al, *.google.am, *.google.as, *.google.at, *.google.az, *.google.ba, *.google.be, *.google.bf, *.google.bg, *.google.bi, *.google.bj, *.google.bs, *.google.bt, *.google.by, *.google.ca, *.google.cat, *.google.cc, *.google.cd, *.google.cf, *.google.cg, *.google.ch, *.google.ci, *.google.cl, *.google.cm, *.google.cn, *.google.co.ao, *.google.co.bw, *.google.co.ck, *.google.co.cr, *.google.co.hu, *.google.co.id, *.google.co.il, *.google.co.im, *.google.co.in, *.google.co.je, *.google.co.jp, *.google.co.ke, *.google.co.kr, *.google.co.ls, *.google.co.ma, *.google.co.mz, *.google.co.nz, *.google.co.th, *.google.co.tz, *.google.co.ug, *.google.co.uk, *.google.co.uz, *.google.co.ve, *.google.co.vi, *.google.co.za, *.google.co.zm, *.google.co.zw, *.google.com, *.google.com.af, *.google.com.ag, *.google.com.ai, *.google.com.ar, *.google.com.au, *.google.com.bd, *.google.com.bh, *.google.com.bn, *.google.com.bo, *.google.com.br, *.google.com.by, *.google.com.bz, *.google.com.cn, *.google.com.co, *.google.com.cu, *.google.com.cy, *.google.com.do, *.google.com.ec, *.google.com.eg, *.google.com.et, *.google.com.fj, *.google.com.ge, *.google.com.gh, *.google.com.gi, *.google.com.gr, *.google.com.gt, *.google.com.hk, *.google.com.iq, *.google.com.jm, *.google.com.jo, *.google.com.kh, *.google.com.kw, *.google.com.lb, *.google.com.ly, *.google.com.mm, *.google.com.mt, *.google.com.mx, *.google.com.my, *.google.com.na, *.google.com.nf, *.google.com.ng, *.google.com.ni, *.google.com.np, *.google.com.nr, *.google.com.om, *.google.com.pa, *.google.com.pe, *.google.com.pg, *.google.com.ph, *.google.com.pk, *.google.com.pl, *.google.com.pr, *.google.com.py, *.google.com.qa, *.google.com.ru, *.google.com.sa, *.google.com.sb, *.google.com.sg, *.google.com.sl, *.google.com.sv, *.google.com.tj, *.google.com.tn, *.google.com.tr, *.google.com.tw, *.google.com.ua, *.google.com.uy, *.google.com.vc, *.google.com.ve, *.google.com.vn, *.google.cv, *.google.cz, *.google.de, *.google.dj, *.google.dk, *.google.dm, *.google.dz, *.google.ee, *.google.es, *.google.eus, *.google.fi, *.google.fm, *.google.fr, *.google.frl, *.google.ga, *.google.gal, *.google.ge, *.google.gg, *.google.gl, *.google.gm, *.google.gp, *.google.gr, *.google.gy, *.google.hk, *.google.hn, *.google.hr, *.google.ht, *.google.hu, *.google.ie, *.google.im, *.google.in, *.google.info, *.google.iq, *.google.ir, *.google.is, *.google.it, *.google.it.ao, *.google.je, *.google.jo, *.google.jobs, *.google.jp, *.google.kg, *.google.ki, *.google.kz, *.google.la, *.google.li, *.google.lk, *.google.lt, *.google.lu, *.google.lv, *.google.md, *.google.me, *.google.mg, *.google.mk, *.google.ml, *.google.mn, *.google.ms, *.google.mu, *.google.mv, *.google.mw, *.google.ne, *.google.ne.jp, *.google.net, *.google.ng, *.google.nl, *.google.no, *.google.nr, *.google.nu, *.google.off.ai, *.google.pk, *.google.pl, *.google.pn, *.google.ps, *.google.pt, *.google.ro, *.google.rs, *.google.ru, *.google.rw, *.google.sc, *.google.se, *.google.sh, *.google.si, *.google.sk, *.google.sm, *.google.sn, *.google.so, *.google.sr, *.google.st, *.google.td, *.google.tel, *.google.tg, *.google.tk, *.google.tl, *.google.tm, *.google.tn, *.google.to, *.google.tt, *.google.ua, *.google.us, *.google.uz, *.google.vg, *.google.vu, *.google.ws, *.googleadapis.com, *.googleadsserving.cn, *.googleapis.cn, *.googlecnapps.cn, *.googlecommerce.com, *.googleusercontent.cn, *.googlevideo.com, *.gstatic.cn, *.gstatic.com, *.gstaticcnapps.cn, *.gvt1.com, *.gvt2.com, *.jp.doubleclick.net, *.metric.gstatic.com, *.uk.doubleclick.net, *.urchin.com, *.url.google.com, *.wear.gkecnapps.cn, *.youtube-nocookie.com, *.youtube.com, *.youtubeeducation.com, *.youtubekids.com, *.yt.be, *.ytimg.com, ad.mo.doubleclick.net, android.clients.google.com, android.com, developer.android.google.cn, developers.android.google.cn, doubleclick.com, doubleclick.net, g.co, ggpht.cn, gkecnapps.cn, goo.gl, google-analytics.com, google.ac, google.ad, google.ae, google.af, google.ag, google.ai, google.al, google.am, google.as, google.at, google.az, google.ba, google.be, google.bf, google.bg, google.bi, google.bj, google.bs, google.bt, google.by, google.ca, google.cat, google.cc, google.cd, google.cf, google.cg, google.ch, google.ci, google.cl, google.cm, google.cn, google.co.ao, google.co.bw, google.co.ck, google.co.cr, google.co.hu, google.co.id, google.co.il, google.co.im, google.co.in, google.co.je, google.co.jp, google.co.ke, google.co.kr, google.co.ls, google.co.ma, google.co.mz, google.co.nz, google.co.th, google.co.tz, google.co.ug, google.co.uk, google.co.uz, google.co.ve, google.co.vi, google.co.za, google.co.zm, google.co.zw, google.com.af, google.com.ag, google.com.ai, google.com.ar, google.com.au, google.com.bd, google.com.bh, google.com.bn, google.com.bo, google.com.br, google.com.by, google.com.bz, google.com.cn, google.com.co, google.com.cu, google.com.cy, google.com.do, google.com.ec, google.com.eg, google.com.et, google.com.fj, google.com.ge, google.com.gh, google.com.gi, google.com.gr, google.com.gt, google.com.hk, google.com.iq, google.com.jm, google.com.jo, google.com.kh, google.com.kw, google.com.lb, google.com.ly, google.com.mm, google.com.mt, google.com.mx, google.com.my, google.com.na, google.com.nf, google.com.ng, google.com.ni, google.com.np, google.com.nr, google.com.om, google.com.pa, google.com.pe, google.com.pg, google.com.ph, google.com.pk, google.com.pl, google.com.pr, google.com.py, google.com.qa, google.com.ru, google.com.sa, google.com.sb, google.com.sg, google.com.sl, google.com.sv, google.com.tj, google.com.tn, google.com.tr, google.com.tw, google.com.ua, google.com.uy, google.com.vc, google.com.ve, google.com.vn, google.cv, google.cz, google.de, google.dj, google.dk, google.dm, google.dz, google.ee, google.es, google.eus, google.fi, google.fm, google.fr, google.frl, google.ga, google.gal, google.ge, google.gg, google.gl, google.gm, google.gp, google.gr, google.gy, google.hk, google.hn, google.hr, google.ht, google.hu, google.ie, google.im, google.in, google.info, google.iq, google.ir, google.is, google.it, google.it.ao, google.je, google.jo, google.jobs, google.jp, google.kg, google.ki, google.kz, google.la, google.li, google.lk, google.lt, google.lu, google.lv, google.md, google.me, google.mg, google.mk, google.ml, google.mn, google.ms, google.mu, google.mv, google.mw, google.ne, google.ne.jp, google.net, google.ng, google.nl, google.no, google.nr, google.nu, google.off.ai, google.pk, google.pl, google.pn, google.ps, google.pt, google.ro, google.rs, google.ru, google.rw, google.sc, google.se, google.sh, google.si, google.sk, google.sm, google.sn, google.so, google.sr, google.st, google.td, google.tel, google.tg, google.tk, google.tl, google.tm, google.tn, google.to, google.tt, google.ua, google.us, google.uz, google.vg, google.vu, google.ws
Matheus Moreira
Power Member
- Jul 14, 2017
- 753
- 405
They will return different results
VirtualSystems
Banned - selling outside of the marketplace.
- Oct 5, 2018
- 224
- 46
True, but sometimes it's even better.