List of Web Scraping Tools to Extract Online Data

[rootjazz]

raw sockets because its the fastest and low on resources for scraping which in most cases works just fine. of course using http requests is also good.

The increase of speed and reduction in resources from sockets to HTTP requests is going to be negliable IMO on a modern machine. Plus there is always the risk with a custom solution that you have missed than using wrapper. But each to their own

as for selenium, it is way too bloated if you plan on running something multithreaded and fast such as a webspider that scrapes data from hundreds of thousands or even millions of domains.

Clearly, that is not really what you should be using selenium. Right tool for the job.

if you have to execute javascript, there are sandboxed solutions where you can execute the js and get the data you need and return it in a http request.

Been a while since I have looked at them, but none were reliable. Spin up an instance, run a script to setup the environment, run selenium as required and close down. Much easier than fighting with a non browser to process the JS

 

[itz_styx]

The increase of speed and reduction in resources from sockets to HTTP requests is going to be negliable IMO on a modern machine. Plus there is always the risk with a custom solution that you have missed than using wrapper. But each to their own

for smaller tasks yes, but for larger ones you will see a difference, just give it a try for example doing scraping in C with raw sockets is way faster as using say C# with httprequests and also much lighter on resources.

Clearly, that is not really what you should be using selenium. Right tool for the job.

the question was about scraping, so in most cases thats not about just scraping 1 site, but more like thousands.

Been a while since I have looked at them, but none were reliable. Spin up an instance, run a script to setup the environment, run selenium as required and close down. Much easier than fighting with a non browser to process the JS

it can be done reliable unless its really ultra complex js. starting instances and instances of a full browser is not only slow, but overkill. however most people go with the easy solution instead of spending time to work out a proper custom solution that is fast and low on resources. its the same with web development these days. people load a whole framework just for 1 function and add anotherone to use just 1-2 new functions, creating bloat and possible security risks.

 

[GeeLark AI]

That isn't how you thread selenium.You need a system to spin up a cloud VPS, run your code, then close down.

What are the Components of Selenium?

 

[rootjazz]

for smaller tasks yes, but for larger ones you will see a difference, just give it a try for example doing scraping in C with raw sockets is way faster as using say C# with httprequests and also much lighter on resources.

A raw socket implementation in c is of course faster than c# using the http library for a webrequest. But if C can run 10,000,00 per second and c# can do 8,000,000 it's irrelevant. and if the C program uses 1 mb or 10gb less memory, really who cares. It's so small to not be noticable. Add in developer time, less rope to hang yourself with, simple syntactic sugar for async patterns and I cannot see any benefit to writing http libraries with sockets. What are HTTP libraries? Just a wrapper on socket programming written by programmers much more skilled than us (well me at least).

the question was about scraping, so in most cases thats not about just scraping 1 site, but more like thousands.

Really depends. I would say it's about equally. Certainly in projects I've taken, people want specific data from specific sites. Sometimes that has meant HTTP requests, other times a browser. Obviously if you are scraping thousands of sites, sockets / requests are the way to do

it can be done reliable unless its really ultra complex js

Isn't all JS a complicated mess - it is what JS does best lol

. starting instances and instances of a full browser is not only slow, but overkill. however most people go with the easy solution instead of spending time to work out a proper custom solution that is fast and low on resources

If you need to thread browsers, it is the only way to go IMO. No it's not easy, but as we are also debating writing your own socket library we are not talking easy solutions, but the best

. its the same with web development these days. people load a whole framework just for 1 function and add anotherone to use just 1-2 new functions, creating bloat

No argument there. But some tasks do require a complex solution

 

[rootjazz]

What are the Components of Selenium?

Sorry, I don't understand your question.

Selenium is a library you can use to control an actual fully fledged web browser via the webdriver protocol. Basically it will open your everyday chrome browser and let you control it from code.

As per the debate in this thread,I think it is the right tool for some jobs. Others don't

 

[itz_styx]

A raw socket implementation in c is of course faster than c# using the http library for a webrequest. But if C can run 10,000,00 per second and c# can do 8,000,000 it's irrelevant. and if the C program uses 1 mb or 10gb less memory, really who cares. It's so small to not be noticable. Add in developer time, less rope to hang yourself with, simple syntactic sugar for async patterns and I cannot see any benefit to writing http libraries with sockets. What are HTTP libraries? Just a wrapper on socket programming written by programmers much more skilled than us (well me at least).

well personally i care to optimize my own software to use a low amount of CPU and RAM where possible, but thats just me
also not trying to brag or anything, but i'm comming from a security background where i worked a lot with low level networking and kernel code, written libs before also for sockets and many other stuff in the last 25 years, so to me its not all that complex to work with raw sockets. often wrote libs myself also because i want to understand each part of the system i'm working on. its just easier to understand/debug your own code than having to dig into someone elses code. sure its more work and takes more time, but you know exactly where everything is hehe

Really depends. I would say it's about equally. Certainly in projects I've taken, people want specific data from specific sites. Sometimes that has meant HTTP requests, other times a browser. Obviously if you are scraping thousands of sites, sockets / requests are the way to do

sure i've also used headless browsers for customer projects on request, but personally i prefer using sockets/http requests as 99% of the time thats the best solution.

Isn't all JS a complicated mess - it is what JS does best lol

lol yes well thats a whole new discussion about how much JS sucks. its a horrible language. someone should kill it with fire!

If you need to thread browsers, it is the only way to go IMO. No it's not easy, but as we are also debating writing your own socket library we are not talking easy solutions, but the best

well using headless browsers is pretty easy, everything is already done for you, you just have to plug it together kinda like LEGO

No argument there. But some tasks do require a complex solution

indeed

 

[bartosimpsonio]

True. Actually I use it more for UX testing than scraping. For scraping, there are better ways.

What's the simplest way to scrape with javascript enabled?

 

[Gogol]

What's the simplest way to scrape with javascript enabled?

Python for me.
If you don't need 100s of threads, but need something very quickly; I doubt you can do it easier with anything other than python selenium.

For example, check this reply of mine (it could be done wayy better though. ):
https://www.blackhatworld.com/seo/need-help-from-python-experts.1259364/#post-13547484

 

[tuliosousa25]

Seems great! Thank you for sharing

 

[bartosimpsonio]

Python for me.
If you don't need 100s of threads, but need something very quickly; I doubt you can do it easier with anything other than python selenium.

For example, check this reply of mine (it could be done wayy better though. ):
https://www.blackhatworld.com/seo/need-help-from-python-experts.1259364/#post-13547484

Python is awesome.

BTW Perl getting slowly forgotten. I remember when Yahoo directory was a CGI script.

 

[Gogol]

Python is awesome.

BTW Perl getting slowly forgotten. I remember when Yahoo directory was a CGI script.

Our company uses Perl for it's backend.
What you can do with Python + Selenium, you can do with Perl + selenium too. It's a matter of choice ultimately.

 

[rootjazz]

you can do with Perl + selenium too. It's a matter of choice ultimately.

Just because you "can", doesn't mean you "should"

 

[Gogol]

Just because you "can", doesn't mean you "should"

Perl is floating around for decades though

A vast majority of our UX testing scripts are written in Perl + Selenium. Now-a-days, I am slowly migrating them to use Python instead.

 

[skbytes]

Can I ask if any of these or any other tool scrapes a whole website of it's emails? I think scrapebox does it to a certain degree but is there any others? thanks for sharing.

 

[Carrieackson]

Great share dude, most helpful

 

[rootjazz]

Perl is floating around for decades though

A vast majority of our UX testing scripts are written in Perl + Selenium.

ouch lol. jk

But it could be worse. You could be using javascript!!!

 

[sev1024]

True. Actually I use it more for UX testing than scraping. For scraping, there are better ways.

Wow theres a ton of nuggets and I havent even reached page 2 yet!

 

[eabuilder]

It would be good if we had like a Wiki page for all these tools, their descriptions and how they compare to each other.. Just my 2 cents - it would be difficult to maintain / organize anyway.

 

[iseolator]

Nothing better than screaming-frog

 

[JPower]

good stuff