Need a fast Proxy Scraper Script

U

update0

Newbie
Joined
Mar 25, 2018
Messages
6
Reaction score
0
hey.

i am searching for a tiny proxy scraper script that scrapes the proxies from a .txt list of sources and outputs me the results like this script here https://www.blackhatworld.com/seo/perl-script-for-proxy-scraping-many-sources.871981/

does anybody have one or can modify the script above to get a output in a txt file ?

thank you !!!
 
Advertise on BHW
This Python script will scrape proxies from https://www.ip-adress.com/proxy-list and will output them into a text document called "proxy_list.txt". This only works for one site, and I would need to change around the parameters for each site to work on multiple sites. However, you could scrape proxies from about 100 different sites with about a day's worth of coding work making scripts very similar to this one.

You have to have Python and the Requests and Pandas library to run this properly. This took me about 10 minutes to do.

Code: 
import requests
import pandas as pd

url = 'https://www.ip-adress.com/proxy-list'
r = requests.get(url)
html = r.text
soup = BeautifulSoup(html, 'lxml')
table = soup.table
df = pd.read_html(str(table))
df = df[0]
proxy_list = df['Proxy']
proxy_list = proxy_list.tolist()

with open('proxy_list.txt', 'w') as file:
    for proxy in proxy_list:
        file.write(proxy+"\n")
 
AdvertisingGuy said:
This Python script will scrape proxies from https://www.ip-adress.com/proxy-list and will output them into a text document called "proxy_list.txt". This only works for one site, and I would need to change around the parameters for each site to work on multiple sites. However, you could scrape proxies from about 100 different sites with about a day's worth of coding work making scripts very similar to this one.

You have to have Python and the Requests and Pandas library to run this properly. This took me about 10 minutes to do.

Code: 
import requests
import pandas as pd
from bs4 import BeautifulSoup

url = 'https://www.ip-adress.com/proxy-list'
r = requests.get(url)
html = r.text
soup = BeautifulSoup(html, 'lxml')
table = soup.table
df = pd.read_html(str(table))
df = df[0]
proxy_list = df['Proxy']
proxy_list = proxy_list.tolist()

with open('proxy_list.txt', 'w') as file:
    for proxy in proxy_list:
        file.write(proxy+"\n")
Click to expand...

fixed your code a bit:) you forgot to import BeautifulSoup
 
Swagmonster said:
fixed your code a bit:) you forgot to import BeautifulSoup
Click to expand...

Haha. I promise it worked in my IPython console. :D Just got a little lost in translation when cutting / pasting.

Anyway, if OP cares to, he could have some functions + create new scraping rules for each of the sites. But honestly the real trick is testing the proxies and getting the speed so the only proxies remaining are fast / live ones. You could create a program to do that as well.
 
proxygo said:
a proxy script that outputs BeautifulSoup
nice concept. can i have tomato soup
Click to expand...

You can have BeautifulStoneSoup :D

For those wondering, BeautifulSoup is a python-based XML parser. You put HTML in BeautifulSoup, parse the code and then find the data you need. Really fast, and useful.

OP, if you ever wanted to get into Python, scraping is a good place to start. This video should help:

 
i was pulling your leg bro, but it made me go in the
kitchen and make a mug of

img]
 
You can try this tool: https://github.com/constverum/ProxyBroker

It can scrape proxies from multiple sources and also check them concurrently.
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top