I'm starting a Perl Bot/Scraper programming tutorial.

VoidITSolutions

VoidITSolutions

BANNED
Joined
Apr 5, 2013
Messages
162
Reaction score
44
Like the title says, I'm creating a tutorial that'll teach you how to scrape websites and create bots.
Part 1 is done which covers just the basics.
Part 2 is also finished which covers retrieving web pages and submitting forms.

If I see enough traffic and social shares from the content, I'll create a full-blown series and it'll be out there for all of you BlackHatters for free.

Being able to create your own bots is a powerful ability.
I wanted to share it with the community that's taught me so much.

I'll add Part 3 tomorrow.

I've also linked to my Perl for Beginners series in the post if you're not familiar with Perl programming.

Please, if you comment SITE suggestions do so in here.
If you have direct CODE suggestions or questions, comment on the site.

Let me know what you think!

http://www.fryitservice.com/p/creating-web-bots-with-perl-part-1-perl-basics/
 
Advertise on BHW
Good stuff!

How do you rate Perl as a language for this type of thing?
 
seeplusplus said:
Good stuff!

How do you rate Perl as a language for this type of thing?
Click to expand...

I love using perl. I use it to automate everything.
Perl was made for manipulating data. I find i very easy to code pattern matching algorithms with which I think makes it perfect for bot/scraper coding.
 
Nice tutorial.. Can you make be a bot that will scrape all of yelp and put it into a csv so i can upload it to a new site i am working on?
 
Bostoncab said:
Nice tutorial.. Can you make be a bot that will scrape all of yelp and put it into a csv so i can upload it to a new site i am working on?
Click to expand...

Ha. Not for free :D. I'd say PM me if you wanna chat about it.

My goal here though is to teach others how to build their own bots.
I don't usually take up freelance gigs.
Maybe I'll get JrVip and set up a BST later this month, we'll see.

Either way, I'll be continuing the tutorial tomorrow.
Hope it helps someone!
 
Kries said:
Hi, does this work for twitter-relevant projects? Like scraping tweets etc.
Click to expand...

It should for work for almost all sites.

You may need to fiddle with the UserAgent string and/or use a proxy however.
I'll be including proxy management in my 3rd or 4th entry. I haven't decided which yet.

The more IPs you are using, the higher your bot/scraper success rate..

It is important to have IP addresses from the same geographical location for the highest bot success rate.
One big red-flag that causes captchas and/or banned accounts is creating an account in one country and using it in another.
 
Last edited:
As a Rubyist I am going to watch some nice Perl. Just make sure you add syntax highlighting to your code examples.
 
As a Rubyist I am going to watch some nice Perl. Just make sure you add syntax highlighting to your code examples.
 
Thanks, very helpful example using LWP::UserAgent. I agree with CodeNinja on the syntax highlighting. Do you have any experience with Mojolicious? It's a power web framework that can be used for some powerful crawler/bots.
 
I agree with both of you about the syntax highlighting. I overlooked it but I just added it to all of my posts.
It makes the posts much more festive :D

As far as Mojolicious mast3rmind, no I haven't.
I'm very stubborn when comes to third party tools.
The only one I'll touch is ScrapeBox.
Everything else I build from scratch. It takes longer initially but when a site/program changes it's code, it makes it easy for me to update mine instead of waiting on the developer. I'm also a big fan of the command line and scripting things. Most 3rd party tools are GUI. I suppose I can't speak for Mojo though.
 
Part 3 is done - http://www.fryitservice.com/p/creating-web-bots-with-perl-part-3-basic-processing/

Now ya'll can code your own google scrapers!
 
Have you considered expaned your tutorial into teaching how to web scrape using WWW::Mechanize?
Mechanize gives a more browserlike experience, without the need to define your own cookie jars.
 
CodeNinja said:
Have you considered expaned your tutorial into teaching how to web scrape using WWW::Mechanize?
Mechanize gives a more browserlike experience, without the need to define your own cookie jars.
Click to expand...

Someone brought it to my attention on Facebook.
WWW::Mechanize is not my style and it has no real benefits.

It may be easier for some to use but I like to program on a lower-level (I started programming first with Intel Assembler if that tells you anything).

Maybe in the future I'll make another set of tutorials geared around it.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top