Best way to scrape for plain old HTML?

What is the best way to scrape for plain old HTML? I'm not looking for a headless browser to scrape pages. I'm looking for a really simple HTTPS crawler to efficiently scrape every single page on this website with about 250k pages and download them to a drive. I'd imagine whatever CDN they're using is going to block automated requests at some point. So ideally something with rotating proxies? I don't know. I haven't scraped something of this size before. I need to get all the plain HTML (not modified by JS) so I can run analysis locally. Thanks!

Do you want the actual html page content or just every URL on the site?

you can copy a website by HTTrack .

graham25s said:

Do you want the actual html page content or just every URL on the site?

Click to expand...

HTML page content would be ideal, but just every URL would also be a good start.

herps said:

HTML page content would be ideal, but just every URL would also be a good start.

Click to expand...

You can try to find as many pages as possible through their sitemap. And using the site:blah.com google dork.

You can write a simple script to use something like wget or curl to download pages with rotating proxies. Furthermore, you can find tons of working, free proxies to rotate through without really having to buy proxies. Obviously, if you can't scrape enough, you'd need to buy reliable proxies. I used this technique on multiple occasions with great success.