expireddomains.net scraping script [Python]

yellowcat

Regular Member
Aug 27, 2015
399
285
Made a quick script today to scrape expireddomains.net (Hacked it together in 10 mins don't judge)
Simply just go to expireddomains enter in your search query and click search.
Replace "URL_HERE" with your url
Here's a sample url
https://www.expireddomains.net/domain-name-search/?o=bl&r=d&ftlds[]=2&q=cats

Code:
#all cats are yellow
import requests,time
from bs4 import BeautifulSoup

url = "URL_HERE"
r = requests.get(url)

listy = []
main_url = "https://www.expireddomains.net"
while True:
    try:
        r = requests.get(url)
        html = BeautifulSoup(r.text,"lxml")
        url = main_url + html.find("div",class_="right").find("a")["href"]

        print url
        links = html.find_all(class_="field_domain")

        for x in links:
            listy.append(x.find("a")["title"])
            print x.find("a")["title"]

        r = requests.get(url)
        print "Total Urls Found ", len(listy)
    except:
        print html
        if html.text.__contains__("You hit the rate limiter. Slow down!"):
            File = open("sites.txt", "w")
            for x in listy:
                File.write(x + "\n")
            File.flush()
            print "Total Urls Found ", len(listy)
            print "Sleeping..."
            time.sleep(5)


        else:
            print "Breaking"
            break

File = open("sites.txt","w")
for x in listy:
    File.write(x + "\n")
File.close()
 
Thanks for your sharing but im noob with python.

I have this error when im launching script:

NameError: name 'html' is not defined.

Maybe you can help me, i lauch it on windows os, all python module needed are installed.
 
Thanks.. works well.. although there is an error in your code..
every print command needs brackets ( ) example - print URL should be print (URL)
at least thats what I had to do to get it to work.. but am only a newbie!
 
Thanks.. works well.. although there is an error in your code..
every print command needs brackets ( ) example - print URL should be print (URL)
at least thats what I had to do to get it to work.. but am only a newbie!
That's not a mistake. It is a python 2 script.
 
I see..
Said I was a newbie
I must be using python 3..
Thanks anyway for the script
 
Back
Top
AdBlock Detected

We get it, advertisements are annoying!

Sure, ad-blocking software does a great job at blocking ads, but it also blocks useful features and essential functions on BlackHatWorld and other forums. These functions are unrelated to ads, such as internal links and images. For the best site experience please disable your AdBlocker.

I've Disabled AdBlock