List of Web Scraping Tools to Extract Online Data

itz_styx said:
raw sockets because its the fastest and low on resources for scraping which in most cases works just fine. of course using http requests is also good.
Click to expand...
The increase of speed and reduction in resources from sockets to HTTP requests is going to be negliable IMO on a modern machine. Plus there is always the risk with a custom solution that you have missed than using wrapper. But each to their own

as for selenium, it is way too bloated if you plan on running something multithreaded and fast such as a webspider that scrapes data from hundreds of thousands or even millions of domains.
Click to expand...
Clearly, that is not really what you should be using selenium. Right tool for the job.

if you have to execute javascript, there are sandboxed solutions where you can execute the js and get the data you need and return it in a http request.
Click to expand...
Been a while since I have looked at them, but none were reliable. Spin up an instance, run a script to setup the environment, run selenium as required and close down. Much easier than fighting with a non browser to process the JS
 
Advertise on BHW
rootjazz said:
The increase of speed and reduction in resources from sockets to HTTP requests is going to be negliable IMO on a modern machine. Plus there is always the risk with a custom solution that you have missed than using wrapper. But each to their own
Click to expand...
for smaller tasks yes, but for larger ones you will see a difference, just give it a try :) for example doing scraping in C with raw sockets is way faster as using say C# with httprequests and also much lighter on resources.

rootjazz said:
Clearly, that is not really what you should be using selenium. Right tool for the job.
Click to expand...
the question was about scraping, so in most cases thats not about just scraping 1 site, but more like thousands.

rootjazz said:
Been a while since I have looked at them, but none were reliable. Spin up an instance, run a script to setup the environment, run selenium as required and close down. Much easier than fighting with a non browser to process the JS
Click to expand...
it can be done reliable unless its really ultra complex js. starting instances and instances of a full browser is not only slow, but overkill. however most people go with the easy solution instead of spending time to work out a proper custom solution that is fast and low on resources. its the same with web development these days. people load a whole framework just for 1 function and add anotherone to use just 1-2 new functions, creating bloat and possible security risks.
 
rootjazz said:
That isn't how you thread selenium.You need a system to spin up a cloud VPS, run your code, then close down.
Click to expand...
What are the Components of Selenium?
 
itz_styx said:
for smaller tasks yes, but for larger ones you will see a difference, just give it a try :) for example doing scraping in C with raw sockets is way faster as using say C# with httprequests and also much lighter on resources.
Click to expand...
A raw socket implementation in c is of course faster than c# using the http library for a webrequest. But if C can run 10,000,00 per second and c# can do 8,000,000 it's irrelevant. and if the C program uses 1 mb or 10gb less memory, really who cares. It's so small to not be noticable. Add in developer time, less rope to hang yourself with, simple syntactic sugar for async patterns and I cannot see any benefit to writing http libraries with sockets. What are HTTP libraries? Just a wrapper on socket programming written by programmers much more skilled than us (well me at least).

the question was about scraping, so in most cases thats not about just scraping 1 site, but more like thousands.
Click to expand...
Really depends. I would say it's about equally. Certainly in projects I've taken, people want specific data from specific sites. Sometimes that has meant HTTP requests, other times a browser. Obviously if you are scraping thousands of sites, sockets / requests are the way to do

it can be done reliable unless its really ultra complex js
Click to expand...
Isn't all JS a complicated mess - it is what JS does best lol :)

. starting instances and instances of a full browser is not only slow, but overkill. however most people go with the easy solution instead of spending time to work out a proper custom solution that is fast and low on resources
Click to expand...
If you need to thread browsers, it is the only way to go IMO. No it's not easy, but as we are also debating writing your own socket library we are not talking easy solutions, but the best :)

. its the same with web development these days. people load a whole framework just for 1 function and add anotherone to use just 1-2 new functions, creating bloat
Click to expand...
No argument there. But some tasks do require a complex solution
 
KEEN-G said:
What are the Components of Selenium?
Click to expand...
Sorry, I don't understand your question.

Selenium is a library you can use to control an actual fully fledged web browser via the webdriver protocol. Basically it will open your everyday chrome browser and let you control it from code.

As per the debate in this thread,I think it is the right tool for some jobs. Others don't :)
 
rootjazz said:
A raw socket implementation in c is of course faster than c# using the http library for a webrequest. But if C can run 10,000,00 per second and c# can do 8,000,000 it's irrelevant. and if the C program uses 1 mb or 10gb less memory, really who cares. It's so small to not be noticable. Add in developer time, less rope to hang yourself with, simple syntactic sugar for async patterns and I cannot see any benefit to writing http libraries with sockets. What are HTTP libraries? Just a wrapper on socket programming written by programmers much more skilled than us (well me at least).
Click to expand...
well personally i care to optimize my own software to use a low amount of CPU and RAM where possible, but thats just me :)
also not trying to brag or anything, but i'm comming from a security background where i worked a lot with low level networking and kernel code, written libs before also for sockets and many other stuff in the last 25 years, so to me its not all that complex to work with raw sockets. often wrote libs myself also because i want to understand each part of the system i'm working on. its just easier to understand/debug your own code than having to dig into someone elses code. sure its more work and takes more time, but you know exactly where everything is hehe

rootjazz said:
Really depends. I would say it's about equally. Certainly in projects I've taken, people want specific data from specific sites. Sometimes that has meant HTTP requests, other times a browser. Obviously if you are scraping thousands of sites, sockets / requests are the way to do
Click to expand...
sure i've also used headless browsers for customer projects on request, but personally i prefer using sockets/http requests as 99% of the time thats the best solution.

rootjazz said:
Isn't all JS a complicated mess - it is what JS does best lol :)
Click to expand...
lol yes well thats a whole new discussion about how much JS sucks. its a horrible language. someone should kill it with fire!

rootjazz said:
If you need to thread browsers, it is the only way to go IMO. No it's not easy, but as we are also debating writing your own socket library we are not talking easy solutions, but the best :)
Click to expand...
well using headless browsers is pretty easy, everything is already done for you, you just have to plug it together kinda like LEGO :D

rootjazz said:
No argument there. But some tasks do require a complex solution
Click to expand...
indeed :)
 
bartosimpsonio said:
Python is awesome.

BTW Perl getting slowly forgotten. I remember when Yahoo directory was a CGI script.
Click to expand...
Our company uses Perl for it's backend. :D
What you can do with Python + Selenium, you can do with Perl + selenium too. It's a matter of choice ultimately. :)
 
rootjazz said:
Just because you "can", doesn't mean you "should" :D
Click to expand...
Perl is floating around for decades though :D

A vast majority of our UX testing scripts are written in Perl + Selenium. Now-a-days, I am slowly migrating them to use Python instead.
 
Can I ask if any of these or any other tool scrapes a whole website of it's emails? I think scrapebox does it to a certain degree but is there any others? thanks for sharing.
 
Gogol said:
Perl is floating around for decades though :D

A vast majority of our UX testing scripts are written in Perl + Selenium.
Click to expand...
ouch lol. jk

But it could be worse. You could be using javascript!!!
 
It would be good if we had like a Wiki page for all these tools, their descriptions and how they compare to each other.. Just my 2 cents - it would be difficult to maintain / organize anyway.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top