Tips for advanced web scraping?

B

beginnerxyz

Registered Member
Joined
Jan 17, 2020
Messages
65
Reaction score
18
I have been building bots to scrape and automate stuff on the web but I can't find tips or resources to guide me to the next level. like people who build farms of bots, how to deal with servers in terms of performance, efficiency.. there's so much to scraping very large websites like facebook etc but all I can find online is some mickey mouse scraping 2009 websites.

please recommend some advanced resources or a community that i can join. thanks
 
Advertise on BHW
Well there are some features that "big bois" in this field implement in their web scrapers these features being - https://oxylabs.io/blog/rotate-ip-address and https://oxylabs.io/blog/what-are-http-headers rotation, basically what they do is fool the website that it is not a bot accessing the website but many users from different locations using different devices. You can also find a lot of help for coding a web scraper your self on https://stackoverflow.com/questions/2081586/web-scraping-with-python (or coding projects in general).
 
Thank you all so much. Another question, is there a relatively cheap priced proxy service? For medium size projects sometimes you just need to change to whatever IP and it doesnt have to be "virgin" or anything.
 
For projects without any requirement in quality ip-adresses I use the tor network with a local implementation of rotating proxy. You can find tutorials and ready To use Docker Images.
Otherwise I would use one of the many free proxy lists (here at blackhatw, or in the web).
Then use a own Webseite without any filtering to Check if it is possible To reach the Site with the selected Proxy. Use the proxy for a random (<50) number of times and switch To a new one
 
Marcel Freiberg said:
For projects without any requirement in quality ip-adresses I use the tor network with a local implementation of rotating proxy. You can find tutorials and ready To use Docker Images.
Otherwise I would use one of the many free proxy lists (here at blackhatw, or in the web).
Then use a own Webseite without any filtering to Check if it is possible To reach the Site with the selected Proxy. Use the proxy for a random (<50) number of times and switch To a new one
Click to expand...
Thanks for the info, I tried using the tor docker image for rotating proxies here : https://github.com/mattes/rotating-proxy
But it kept crashing the 5$ droplet from DigitalOcean, not sure whats the reason but every couple of minutes It would make the system run out of memory (it has 1GB of ram) and terminal commands would return "segmentation fault" etc. So I gave up on it.
 
Do you guys know if there's a service to buy email addresses for registering accounts on sites in order to fake being multiple users. I need that for apps like freshline, yeezysupply etc
 
beginnerxyz said:
Thanks for the info, I tried using the tor docker image for rotating proxies here : https://github.com/mattes/rotating-proxy
But it kept crashing the 5$ droplet from DigitalOcean, not sure whats the reason but every couple of minutes It would make the system run out of memory (it has 1GB of ram) and terminal commands would return "segmentation fault" etc. So I gave up on it.
Click to expand...
Do you have your proxies/server? If you coded some bots in the past it is not hard to implement a proxy manager with rotating proxies yourself.
 
TikolaNesla said:
Do you have your proxies/server? If you coded some bots in the past it is not hard to implement a proxy manager with rotating proxies yourself.
Click to expand...
I was hoping I could use Tor's free IPs
 
Concerning your RAM issue you could either use a higher Tier of vps with more RAM (+money/ -time) or build your own image based on alpine with a low memory consumption (+time/ no money). But After Dreingabe your questions I would suggest to ask someone here (marketplace / wtb) To get the result instead ...
 
beginnerxyz said:
Thanks for the info, I tried using the tor docker image for rotating proxies here : https://github.com/mattes/rotating-proxy
But it kept crashing the 5$ droplet from DigitalOcean, not sure whats the reason but every couple of minutes It would make the system run out of memory (it has 1GB of ram) and terminal commands would return "segmentation fault" etc. So I gave up on it.
Click to expand...
You can try the stormproxy service for rotating proxies (They have a thread on here if I'm not mistaking)
 
beginnerxyz said:
I was hoping I could use Tor's free IPs
Click to expand...

Haven't done that myself. Free IPs are not really worth the hassle IMO, but again. Send me a PM. I have some Python code for you that scrapes free proxies and manages them. Lot to be found on Github as well by the way.
 
TikolaNesla said:
Haven't done that myself. Free IPs are not really worth the hassle IMO, but again. Send me a PM. I have some Python code for you that scrapes free proxies and manages them. Lot to be found on Github as well by the way.
Click to expand...

Can't find a way to send you PM. I think cause my account is new. Maybe you send to start the convo. Appreciate your help.
 
Anyone here good with scrapers i have a project i am trying to get done an will pay for it. PM me thanks
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top