1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[PERL] Script for Proxy Scraping many sources

Discussion in 'PHP & Perl' started by rodvan, Aug 24, 2016.

  1. rodvan

    rodvan Jr. VIP Jr. VIP

    Joined:
    Jul 27, 2010
    Messages:
    1,293
    Likes Received:
    492
    Occupation:
    developer, marketing, automation, machine learning
    Location:
    Wizard of Bots
    Home Page:
    The code is simple and uses basic Perl modules to crawl the given pages from a file, and prints out a list of proxies:
    Code:
    #!/usr/bin/perl
    # Usage: proxyScraper.** [proxy_file]
    use strict;
    use warnings;
    use WWW::Mechanize;
    use Try::Tiny;
    my $source_file=shift;
    open (INPUT_FILE, "<$source_file")  || die "Can't open $source_file: $!\n";
    my @sources = ;
    my $crawler = WWW::Mechanize->new();
    foreach (@sources) {
        try {
            $crawler->get($_);
            # hunt for IP:PORT combination
            my @ips= $crawler->text() =~ /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5})/g;
            foreach (@ips){
                print "$_\n";
            }
        } catch {
            warn "[!] Error, who cares\n";
        }
    }
    And you load your proxy sources like this:
     
    • Thanks Thanks x 2
  2. IZpk-A

    IZpk-A Newbie

    Joined:
    Aug 28, 2016
    Messages:
    28
    Likes Received:
    1
    Gender:
    Male
    How to save the proxy into txt file?
     
  3. royserpa

    royserpa Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 28, 2011
    Messages:
    4,972
    Likes Received:
    3,727
    Gender:
    Male
    Occupation:
    Negative Options aka Rebills!
    Location:
    Exploiting Loopholes!
    Home Page:
    Will give it a shot
     
  4. rodvan

    rodvan Jr. VIP Jr. VIP

    Joined:
    Jul 27, 2010
    Messages:
    1,293
    Likes Received:
    492
    Occupation:
    developer, marketing, automation, machine learning
    Location:
    Wizard of Bots
    Home Page:
    You open write, then write the proxy into the file, after you finish the loop, close the file.

    Code:
    text_file = open("Output.txt", "w")
    text_file.write(proxyvariable)
    text_file.close()