expireddomains.net scraping script [Python]

yellowcat

Regular Member
Joined
Aug 27, 2015
Messages
379
Reaction score
261
Made a quick script today to scrape expireddomains.net (Hacked it together in 10 mins don't judge)
Simply just go to expireddomains enter in your search query and click search.
Replace "URL_HERE" with your url
Here's a sample url
https://www.expireddomains.net/domain-name-search/?o=bl&r=d&ftlds[]=2&q=cats

Code:
#all cats are yellow
import requests,time
from bs4 import BeautifulSoup

url = "URL_HERE"
r = requests.get(url)

listy = []
main_url = "https://www.expireddomains.net"
while True:
    try:
        r = requests.get(url)
        html = BeautifulSoup(r.text,"lxml")
        url = main_url + html.find("div",class_="right").find("a")["href"]

        print url
        links = html.find_all(class_="field_domain")

        for x in links:
            listy.append(x.find("a")["title"])
            print x.find("a")["title"]

        r = requests.get(url)
        print "Total Urls Found ", len(listy)
    except:
        print html
        if html.text.__contains__("You hit the rate limiter. Slow down!"):
            File = open("sites.txt", "w")
            for x in listy:
                File.write(x + "\n")
            File.flush()
            print "Total Urls Found ", len(listy)
            print "Sleeping..."
            time.sleep(5)


        else:
            print "Breaking"
            break

File = open("sites.txt","w")
for x in listy:
    File.write(x + "\n")
File.close()
 

StoneDB

Registered Member
Joined
Nov 20, 2016
Messages
55
Reaction score
29
Thanks for your sharing but im noob with python.

I have this error when im launching script:

NameError: name 'html' is not defined.

Maybe you can help me, i lauch it on windows os, all python module needed are installed.
 

ozcam

Registered Member
Joined
Dec 7, 2015
Messages
79
Reaction score
10
Thanks.. works well.. although there is an error in your code..
every print command needs brackets ( ) example - print URL should be print (URL)
at least thats what I had to do to get it to work.. but am only a newbie!
 

Gogol

Jr. VIP
Jr. VIP
Joined
Sep 10, 2010
Messages
9,687
Reaction score
16,632
Website
twitter.com
Thanks.. works well.. although there is an error in your code..
every print command needs brackets ( ) example - print URL should be print (URL)
at least thats what I had to do to get it to work.. but am only a newbie!
That's not a mistake. It is a python 2 script.
 

ozcam

Registered Member
Joined
Dec 7, 2015
Messages
79
Reaction score
10
I see..
Said I was a newbie
I must be using python 3..
Thanks anyway for the script
 
Top