1. This website uses cookies to improve service and provide a tailored user experience. By using this site, you agree to this use. See our Cookie Policy.
    Dismiss Notice

Best Software for Web Scraping (Discussion)

Discussion in 'Black Hat SEO Tools' started by Donbuffy, Aug 9, 2019.

  1. Donbuffy

    Donbuffy Power Member

    Joined:
    Jul 23, 2012
    Messages:
    639
    Likes Received:
    117
    Gender:
    Male
    Occupation:
    Self Employed
    Location:
    Wakanda
    Hello Guys, trying to go into proper data mining for my new project and i've been surfing the net for a long while, i've read a lot of reviews and i've seen a couple of sales thread on here but i'll need to use this thread as a basis of discussion cos it seems am not the only one asking of this information, it should help other noobs out there too, if you've been using a web scraper automation software please comment on this thread, it would enable one make a choice on the best.

    What's the best leads generation software you've used?
    If you don't use a software, what script/plugin do you use to extract tons of leads?
     
  2. letuvis

    letuvis Regular Member

    Joined:
    Nov 13, 2017
    Messages:
    202
    Likes Received:
    49
    Gender:
    Male
    Occupation:
    Digital Nomad
    Location:
    Lithuania
    I was using Parse Hub as my software to scrape data. It's easy to use and has a free plan. Would advise to use Parse Hub it's the best imo.
     
    • Thanks Thanks x 1
  3. Peter Samaki

    Peter Samaki BANNED BANNED

    Joined:
    Jul 19, 2019
    Messages:
    72
    Likes Received:
    31
    Gender:
    Male
    Limited features are free.. But expensive for normal meaningful scraping.
     
  4. KJREDDY247@

    [email protected] Senior Member

    Joined:
    Aug 31, 2018
    Messages:
    867
    Likes Received:
    345
    Gender:
    Male
    1. web scraper
    2. parse hub
    3. scrapy

    I use these because I don't have proper knowledge in coding
    if you know them well;
    you can go with
    python with selenium, beautiful soap etc
     
  5. letuvis

    letuvis Regular Member

    Joined:
    Nov 13, 2017
    Messages:
    202
    Likes Received:
    49
    Gender:
    Male
    Occupation:
    Digital Nomad
    Location:
    Lithuania
    Of course you get limited features when you use free plan for the app. This is normal from my point of view. I used free plan of Parse Hub and everything was scraped for me like a charm. It gave me everything I need, so I don't see any big issue regarding this.. :)
     
  6. redarrow

    redarrow Elite Member

    Joined:
    Apr 1, 2013
    Messages:
    10,695
    Likes Received:
    3,123
    piaghim bot it the best for any leads .....
    (( cost not free))

    you can also use scrapebox , does many things .(( cost not free))

    you can also use gsa tools they do many scrapping tools. (( cost not free ))

    all the above are paid tools but have free trails .

    good luck.
     
  7. Donbuffy

    Donbuffy Power Member

    Joined:
    Jul 23, 2012
    Messages:
    639
    Likes Received:
    117
    Gender:
    Male
    Occupation:
    Self Employed
    Location:
    Wakanda
    Heard Scrapebox is the best, i'll have a look at paigham bot
     
  8. nirose

    nirose Senior Member

    Joined:
    Oct 24, 2008
    Messages:
    1,175
    Likes Received:
    497
    Home Page:
    beautiful soup* ;)

    On topic, scrapping is the easiest thing you can do in programming. and there are a ton of tutorials everywhere. From my experience in JavaScript, PHP, and Python scraping couldn't be any easier. you just have to learn some CSS selectors or XPath and you can grab a lot of data on auto. You can go to lengths with header modification and authentication but most of the sites don't require this.

    The only problem starting with your own scraping bot with going to be threading, as single threads are going to be slow. but it should be a learning process. If you can pay, scrapebox is a good investment. it's really fast
     
  9. KJREDDY247@

    [email protected] Senior Member

    Joined:
    Aug 31, 2018
    Messages:
    867
    Likes Received:
    345
    Gender:
    Male
     
  10. Gogol

    Gogol Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2010
    Messages:
    5,377
    Likes Received:
    5,090
    Gender:
    Male
    Occupation:
    Programmer
    Location:
    Pale Blue Dot
    Home Page:
    There's no "best" software. It's about what tool you want to use to scrape. I personally prefer python requests (if there's no need for js support) or selenium (if the bot has to be js enabled).
    Perl can also do the same job, as it has full selenium support. Even PHP can do the same job, but it will be tougher to do it in php; if js support is needed. I am not sure if php has selenium support.

    As for scrapebox, I am not actually sure. I do have sb licence, but I seldom use it. I mostly code my own bot whenever needed.
     
  11. Donbuffy

    Donbuffy Power Member

    Joined:
    Jul 23, 2012
    Messages:
    639
    Likes Received:
    117
    Gender:
    Male
    Occupation:
    Self Employed
    Location:
    Wakanda
    So from all am understanding here, there's no best bot it all balls down to what is to be scraped and how the user utilizes the tool... so it's best one learn python to build a custom tool that enables one to scrape, right? So computer language for scraping is Python, JavaScript and PHP?

    I know many Digital Marketers who has zero programming knowledge but use tools to scrape leads, i wonder what tool they use. :smirk:
     
  12. Gogol

    Gogol Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2010
    Messages:
    5,377
    Likes Received:
    5,090
    Gender:
    Male
    Occupation:
    Programmer
    Location:
    Pale Blue Dot
    Home Page:
    The problem with a scraper is that it is pretty much platform dependent. A scraper which works on platform A, might not work on platform B. It really does depend on what actually you are trying to scrape. In case you are interested in Scrapebox, here's a list of available plugins that are free with SB (so that you can decide whether to try SB or not):

    upload_2019-8-10_16-47-59.png



    Believe me, making a scraper with python isn't as hard as you think it is. With this little skill, you will not be tied to a specific tool or platform.
     
    • Thanks Thanks x 1
  13. javadth

    javadth Power Member

    Joined:
    Aug 27, 2012
    Messages:
    633
    Likes Received:
    104
    Gender:
    Male
    Occupation:
    telegram : @Instajt
    Location:
    skype:javadxth_1
    Home Page:
    scrapebox i am using it
     
    • Thanks Thanks x 2
  14. 710fla

    710fla Jr. VIP Jr. VIP

    Joined:
    Aug 25, 2015
    Messages:
    2,191
    Likes Received:
    1,160
    Gender:
    Male
    Occupation:
    Marketing
    Location:
    FL
    Home Page:
    Scrapebox hands down. Lots of extra features for free too
     
    • Thanks Thanks x 2
  15. ebubekirtrkr

    ebubekirtrkr Newbie

    Joined:
    Apr 20, 2015
    Messages:
    2
    Likes Received:
    0
    Python Selenium
     
  16. BHSEONewb

    BHSEONewb Regular Member

    Joined:
    Jul 5, 2018
    Messages:
    241
    Likes Received:
    107
    Gender:
    Male
    Is there a tutorial you recommend for learning this? It's been on my list to learn python and web scraping for quite some time now, but finding the right course (and time) has been the main problem.
     
  17. Gogol

    Gogol Jr. VIP Jr. VIP

    Joined:
    Sep 10, 2010
    Messages:
    5,377
    Likes Received:
    5,090
    Gender:
    Male
    Occupation:
    Programmer
    Location:
    Pale Blue Dot
    Home Page:
    Good question. Yepp I have a few channels on youtube that I really enjoy learning from (you can never learn enuf lol).
    As for example:
    Code:
    https://www.youtube.com/channel/UCWv7vMbMWH4-V0ZXdmDpPBA
    https://www.youtube.com/user/edurekaIN   (this is an Indian channel, and you might not like the pronunciation. Nevertheless, the amount of knowledge shared on this channel is freaking awesome. Just get used to the pronunciation lol).
    Also, 
    https://www.youtube.com/user/schafer5
    
    I have a few more in my mind, but can't find them right now. I will update this thread later.
     
    • Thanks Thanks x 1
  18. BHSEONewb

    BHSEONewb Regular Member

    Joined:
    Jul 5, 2018
    Messages:
    241
    Likes Received:
    107
    Gender:
    Male
    THANK YOU!!!!
     
    • Thanks Thanks x 1
  19. Donbuffy

    Donbuffy Power Member

    Joined:
    Jul 23, 2012
    Messages:
    639
    Likes Received:
    117
    Gender:
    Male
    Occupation:
    Self Employed
    Location:
    Wakanda
    Oh wow! Thanks
     
    • Thanks Thanks x 1
  20. Advertising Manager

    Advertising Manager Newbie

    Joined:
    Jul 23, 2019
    Messages:
    40
    Likes Received:
    12
    Gender:
    Male
    Good to know for future scrapping