Looking for help with web scraping.

I am looking for help in web scraping a news site in the most efficient way possible. Speed is a factor.


My goal is to scrape a news site and have news articles delivered to me within 1-5 seconds of being posted. I previously made a news scraper for this site while I was learning to code, but it was buggy and broke down often. I am looking to acquire a better one that will be low maintenance, coded clearly and efficiently, and last.



These are the requirements of the web scraper:



-Must be coded in Python as that is the only language I am capable of understanding. Sorry about that.

-Efficiently login with multiple accounts and multiple proxies, and manage the login sessions. I have emailed the the web admins, they do not allow web scraping, but they do attempt to block it in any way either. They let vanilla Chrome Driver on Selenium through. There will not be much of any security to bypass, and as far as I know their site is actively scraped by trading algorithms. I can log into the website myself with Selenium, I'm asking for someone with more talent to be able to do the login with requests, as that is less load on my computer.



I'm offering $50 for a scraper (coded with requests) that can stay logged into the site. I do not expect it to take too long to code for someone who knows what they are doing. Send me contact info for more details if you are interested. I cannot PM as of yet. I will only take jobs from people with good reviews who aren't on the shit list. I will pay before you send any code over.

Edit: *I can pay through PayPal only. I do not have a CryptoCurrency Wallet.

Have you tried running the Chrome in --headless mode?

That'll save some CPU/Memory resources on your computer as the browser doesn't have to render the elements but you can still traverse through the DOM.


$50 is too small for the features you asked for but I suggest you to get a windows VPS which costs just $8-$10/month in our Marketplace and actively run your Selenium program on the VPS to scrape the data.