expireddomains.net scraping script [Python]

yellowcat

Regular Member
Joined
Aug 27, 2015
Messages
404
Reaction score
297
Made a quick script today to scrape expireddomains.net (Hacked it together in 10 mins don't judge)
Simply just go to expireddomains enter in your search query and click search.
Replace "URL_HERE" with your url
Here's a sample url
https://www.expireddomains.net/domain-name-search/?o=bl&r=d&ftlds[]=2&q=cats

Code:
#all cats are yellow
import requests,time
from bs4 import BeautifulSoup

url = "URL_HERE"
r = requests.get(url)

listy = []
main_url = "https://www.expireddomains.net"
while True:
    try:
        r = requests.get(url)
        html = BeautifulSoup(r.text,"lxml")
        url = main_url + html.find("div",class_="right").find("a")["href"]

        print url
        links = html.find_all(class_="field_domain")

        for x in links:
            listy.append(x.find("a")["title"])
            print x.find("a")["title"]

        r = requests.get(url)
        print "Total Urls Found ", len(listy)
    except:
        print html
        if html.text.__contains__("You hit the rate limiter. Slow down!"):
            File = open("sites.txt", "w")
            for x in listy:
                File.write(x + "\n")
            File.flush()
            print "Total Urls Found ", len(listy)
            print "Sleeping..."
            time.sleep(5)


        else:
            print "Breaking"
            break

File = open("sites.txt","w")
for x in listy:
    File.write(x + "\n")
File.close()
 
Thanks for your sharing but im noob with python.

I have this error when im launching script:

NameError: name 'html' is not defined.

Maybe you can help me, i lauch it on windows os, all python module needed are installed.
 
Thanks.. works well.. although there is an error in your code..
every print command needs brackets ( ) example - print URL should be print (URL)
at least thats what I had to do to get it to work.. but am only a newbie!
 
Thanks.. works well.. although there is an error in your code..
every print command needs brackets ( ) example - print URL should be print (URL)
at least thats what I had to do to get it to work.. but am only a newbie!
That's not a mistake. It is a python 2 script.
 
I see..
Said I was a newbie
I must be using python 3..
Thanks anyway for the script
 
Back
Top