Python : how to scrap Website without proxies , please advice?

[ibmjango]

hey friends,
I am new in python, just trying to get my feet wet with automation stuff, i am using selenium + python to make my scrapping scripts.
the problem i am facing is, my script is working fine until it reach a certain limit, like till 20-25 scrapping request, i get data but after that site start sending me error that either solve captcha or simply stop responding or refuse request.

i cant use proxies as it will increase scrapping cost, as the data i am trying to scrap is big, hence i am looking for any solution where i can scrap bulk data without using proxies, can you help and suggest any way to do it?

any input is appreciated
Thanks.

 

[BlogPro]

Unfortunately, there is no other workaround. Proxies are built for this exact same purpose you're requesting.

You can achieve some sort of mitigation by staggering your requests. Don't scrape 25 listings at once. Go for 3 listings per minute and run parallel connections via threads.

You setup a worker module that runs parallel to your main thread and creates a new thread to execute your task. This can either append to your existing file or make a main file

You should learn the usage of threading in python

https://www.datacamp.com/community/tutorials/threading-in-python
from threading import Thread

Unfortunately, to make the most out of this, your best bet is to use proxies.

Even if the source dataset is large, you can get proxies for very cheap. I recommend trying https://www.scraperapi.com/, they offer 5000 requests for free on their proxy. And beyond that, they offer 1000 requests free every month.

If you have specific need, DM me and I'll tell you how you can cut your costs.

For free workarounds, you can also scrape something like https://spys.me/proxy.txt

However, the reliability of free proxies is often questionable. You'll have to write a script to make a connection to your source, check the response to see if the proxies are valid and then insert them into a separate CSV. This CSV is then used by your main script for scraping.

With a little creativity, you can make this a continuous setup.

 

[Steptoe]

I think if you're really dead set against using proxies, the only way to do this would be some severe rate-limiting. Experiment with different lengths of pauses between requests and see what you can get away with. But this will massively limit your productivity.

After pulling your hair out with that, give the free tiers of webshare.io and scraper.com a go.

 

[Smackhead]

I've seen some posts previously about setting up mobile 4g proxies which seemed fairly inexpensive, could be worth a look.

 

[NulledCode]

You should really have a look at Apify to run your scraper as this will allow you to build, schedule, and execute your scraper. It’s an amazing platform and gets you to build efficient scrapers that scale. They provide proxies (shared) or you can use your own from scraperapi.com or somewhere else.

 

[ibmjango]

TOR browser is suitable for this task ? if yes, please share how to open tor without getting detected, as i gone through some tor browser tutorial on yourtube and some resources on stackoverflow but when i implement them the problem i am facing is,
when i open tor that is not able to open scrapping website url, for example when i try to open article directory or wiki pedia pages tor gives error... can you suggest any workaround to work with tor, please feel free to share resources here or through pm, or i am reachable at my skpe id : ibmjango as well.

 

[NulledCode]

TOR is not going to help you. You have a web scraping script that works, now you just need to scale horizontally so you can run many instances of your script. Either get yourself proxies, and/or start running your script on many VPS machines that have their own IPs.

Find yourself a platform that you can run your script on, or one will that has API you can use to fetch the web page content for you, then your script can parse the results.

1. Scrapestack
2. ScraperAPI
3. ScrapingBee
4. Apify

 

[ibmjango]

i tried one more solution, as on every restart , my internet modem gives new IP, will it work? did anyone tested this approach , even though i am getting new ip on every request but i worry for one more thing, will i recieve any problem or notice from ISP provider ? (Personally in my opinion its stopping my speed of data scrapping, can we do anything like this in any other way on vps server ?)

 

[Xdfffninja]

i tried one more solution, as on every restart , my internet modem gives new IP, will it work? did anyone tested this approach , even though i am getting new ip on every request but i worry for one more thing, will i recieve any problem or notice from ISP provider ? (Personally in my opinion its stopping my speed of data scrapping, can we do anything like this in any other way on vps server ?)

Dude, just purchase some proxies and forget about it. These processes you're taking about a just too long and complicated.

And no, I don't think you would have problems with your ISP because of scraping.

 

[NulledCode]

i tried one more solution, as on every restart , my internet modem gives new IP, will it work? did anyone tested this approach , even though i am getting new ip on every request but i worry for one more thing, will i recieve any problem or notice from ISP provider ? (Personally in my opinion its stopping my speed of data scrapping, can we do anything like this in any other way on vps server ?)

Horrible idea. ISPs will very unlikely give you unlimited IPs each time you reset your device. When it comes to web scrapping and bots... don't shit where you sleep.

 

[ibmjango]

Horrible idea. ISPs will very unlikely give you unlimited IPs each time you reset your device. When it comes to web scrapping and bots... don't shit where you sleep.

agree seems really terrible idea let see if i can implement any of above suggestion or anything new way suggested by fellow bhw members

 

[Sandman_]

Stop being cheap there is no way around getting proxies for what your trying to do

 

[JavascriptPufferfish]

agree seems really terrible idea let see if i can implement any of above suggestion or anything new way suggested by fellow bhw members

i dont think there are free magical solutions. there is a reason proxies are a huge industry, you just can't do some stuff without them.
i guess hypothetically you could load your software in a laptop, go to a coffee shop, and use their wifi to do it from there? but then you have the cost of the coffee. are there no places with public wifi that are frequented by many people close to you? the more people use the internet in an area, the more leeway you get with requests, or so I heard.
still, just buying a few proxies would be the best in this situation. concentrate on learning, not on solving problems you could solve with 10-20 bucks a month.

 

[Gefrekssiy]

Now proxy servers are often indispensable without them

 

[Alexion]

What website are you scraping? Maybe there's some workaround. Have you tried putting delays between requests? Like half a second?

What kind of captcha are you getting? Does DeathByCaptcha offer captcha resolves for that one? If yes, it might be cheaper compared to proxies. It's like $3 for 5,000 captchas. reCaptcha is more expensive.

Maybe the website has an API?
Maybe most of the pages are on web.archive or google cache, you can get the pages from there.

Without knowing the website you are scraping, it's hard to come up with a solution.
Usually, this is why people use Selenium in the first place. If Selenium is not enough, the ONLY next step is to use proxies.

 

[Jack Daniel$]

i tried one more solution, as on every restart , my internet modem gives new IP, will it work? did anyone tested this approach , even though i am getting new ip on every request but i worry for one more thing, will i recieve any problem or notice from ISP provider ? (Personally in my opinion its stopping my speed of data scrapping, can we do anything like this in any other way on vps server ?)

Ip's are different, but they are from the same provider subnet.

You can search for some free public proxies if you don't want to spend on them.

 

[leaoroxo]

Yeah, you better get yourself some cheap proxies and/or solve those captchas.

It's way faster and cheaper to get a captcha solving service working those for you. I like DBC.

You might eventually need some proxies to avoid getting your IP compromised.

 

[ibmjango]

What website are you scraping? Maybe there's some workaround. Have you tried putting delays between requests? Like half a second?

What kind of captcha are you getting? Does DeathByCaptcha offer captcha resolves for that one? If yes, it might be cheaper compared to proxies. It's like $3 for 5,000 captchas. reCaptcha is more expensive.

Maybe the website has an API?
Maybe most of the pages are on web.archive or google cache, you can get the pages from there.

Without knowing the website you are scraping, it's hard to come up with a solution.
Usually, this is why people use Selenium in the first place. If Selenium is not enough, the ONLY next step is to use proxies.

working on ezinearticles and wikipedia, wikipedia have api, i will use it but ezine dont have api or may be i am not aware if they have any

 

[ibmjango]

Yeah, you better get yourself some cheap proxies and/or solve those captchas.

It's way faster and cheaper to get a captcha solving service working those for you. I like DBC.

You might eventually need some proxies to avoid getting your IP compromised.

i didnt think for solving captcha but yeah i have already credit in captcha service let see if this solution can sort this out. i will try this captcha solving approach, lets see how it goes ?

 

[Alexion]

working on ezinearticles and wikipedia, wikipedia have api, i will use it but ezine dont have api or may be i am not aware if they have any

I built a wikipedia scraper to look for expired domains for a client a few weeks ago. It even runs multithreaded, and I never faced captchas or rate-limiting. Can't say for ezine, but I'm pretty sure those don't rate limit you anyway. Maybe you are doing something wrong.