WebScraper: need some advice

X

xuinia

Newbie
Joined
Mar 9, 2016
Messages
4
Reaction score
0
I need to build little web scraping tool. Problem is that I haven't done any c++ programing for many years, so I have no ideas where even to start now. Few years back I made such scraper with php, but its pain in the a** to use, so I want to recode in c++ and compile standalone application. So a have few questions.
1. Before I used c++ builder to code. Should I stick to that again? Or maybe there something better (preferably free)?
2. I will need some sort of library to get website contents. Any advice on that?
3. I will need to go through html to find needed data. In php I used simple_html_dom. Anything like that in c++?
4. Any other advice very wellcome :)

P.S. Scraped data will go to mysql database.
 
Last edited:
Advertise on BHW
I would use Python for a webscraping tool, personally I could do it much easier with python because of all the sweet sweet modules and add ons.
 
Bahmer said:
I would use Python for a webscraping tool, personally I could do it much easier with python because of all the sweet sweet modules and add ons.
Click to expand...
+1 For Python + Scrapy, it will make things way easier for you.
 
Des_cartes said:
+1 For Python + Scrapy, it will make things way easier for you.
Click to expand...

Even beautiful soup or any of those would be better than writing it in C++. God that just gives me the heaby jeebies thinking about it haha.
 
Bahmer said:
I would use Python for a webscraping tool, personally I could do it much easier with python because of all the sweet sweet modules and add ons.
Click to expand...
The only reason why I think of c++ is cause I did some "advanced begginers :) " programing few years back. So I thaught it would be easyer.
Don't know anything about python. It's just scripting (similar to php) or you can compile actuall app?
 
c++ while be harder if you haven't built any type of scrapers before and have only done advanced beginners stuff.

You mention compile standalone app, what's your reason for this sell as product,distribute to staff?

thanks
9to5
 
Few people will use that stuff on different mashines, so standalone app would be best solution I gues.
9to5destroyer said:
c++ while be harder if you haven't built any type of scrapers before and have only done advanced beginners stuff.

You mention compile standalone app, what's your reason for this sell as product,distribute to staff?

thanks
9to5
Click to expand...
 
I would say c# then for standalone app, but that's my preferred language so slightly biased.
 
If you have JavaScript experience, then have a look at Node JS. No better way to scrape HTML than using something built to traverse the DOM.
 
If you use C# you can use Microsoft Visual Studio Express which is a really good free IDE, if you only have a basic knowledge of coding you may find it easier in the long run to learn some C# as it is a fairly quick language to pick up, more so than c++.

With c# you could then use the HtmlAgilityPack, again free, to parse through the document as if it were an xml document which is really easy.

Whenever I have gotten stuck with something in C# there is often as tutorial (admittedly there are tuts for pretty much all modern languages) and it doesn't sound like you are looking to do anything too crazy.

For interfacing with a MySQL db you can use MySQL Connector/Net, again free, which makes it just as easy as it would be in something like php.

One of my biggest bugbears is creating a functional yet not overly hideous UI for my software and having something like Visual Studio Express makes at least basic UIs quick and easy to create so you can get down to the nitty gritty of coding much quicker.
 
Well C++ is the hard way compared to C#/Python but it can be very powerful.

Today C++ with Qt is very simple to work with, even they have very nice IDE for programming.

Check their documentation and you will see...
 
Deff. use Python. It was built by google for the purposes of quick processing with low CPU requirements. Python was basically built for math (algorithms) and data collection (scraping). With Python you can then run multi-threaded workers ;)
For the guy that said Node.js/Javascript...bro...you can't. While technically you can, it requires to much CPU in order to process everything that it would only be cost effective to run it in a single thread.
 
I would suggest you to start with C# cause it has a number of base classes/functions already written to write code with more ease & performance. Also, there are a lot of 3rd party framework/api has already shared for free like HTMLAgility, CSQuery, AngleSharp etc for processing HTML pages. Visual Web Express is a light weight & FREE IDE to write code fast!
 
There are a few very good open source solution for scrapping - like mentioned here scrapy (python) It is highly configurable and has very good performance. You can set it up and forget.
 
I used C# to make my proxy scraper. I recommend you use C# or Vb.net if you're a beginner.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top