1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to Scrape Lists for WHOIS data

Discussion in 'Black Hat SEO' started by zee007, Feb 20, 2014.

  1. zee007

    zee007 Senior Member

    Joined:
    Jun 25, 2012
    Messages:
    806
    Likes Received:
    95
    Location:
    Texas
    I want to scrape lists for WHOIS data, but when I use the allintitle function in SB, I get a lot of local directories pop up. Is there an easier way to scrape a list of local businesses without having local directories popup?
     
  2. TrevorB

    TrevorB Jr. VIP Jr. VIP Premium Member

    Joined:
    Dec 21, 2011
    Messages:
    1,185
    Likes Received:
    361
    Location:
    Canada
    Just go to the WHOIS data pages and make custom footprints. I would think that would work.
     
  3. zee007

    zee007 Senior Member

    Joined:
    Jun 25, 2012
    Messages:
    806
    Likes Received:
    95
    Location:
    Texas
    I lost you. What are you recommending?

    The issue I'm having is that I'm getting a lot of local directories pop-up (Manta, Super Pages etc) when I scrape for local businesses. How can I avoid this from happening.
     
  4. macdonjo3

    macdonjo3 Jr. VIP Jr. VIP Premium Member

    Joined:
    Nov 8, 2009
    Messages:
    5,560
    Likes Received:
    4,317
    Location:
    Toronto
    Home Page:
    Then go to a Whois website and scrape. Don't scrape Manta... lol?

    Google search isn't required for this.
     
  5. zee007

    zee007 Senior Member

    Joined:
    Jun 25, 2012
    Messages:
    806
    Likes Received:
    95
    Location:
    Texas
    I want to first scrape a list of business owners BEFORE I put them into the WHOIS database. Make sense?

    Step 1:

    Scrape local business websites from search results (without scraping local directories, manta, yellow pages etc)

    Step 2

    Take the data from Step 1 and parse it through WHOIS.

    Make sense now?
     
  6. HelloInsomnia

    HelloInsomnia Jr. Executive VIP Jr. VIP Premium Member

    Joined:
    Mar 1, 2009
    Messages:
    1,814
    Likes Received:
    2,910
    I'm pretty sure what he is saying is that he is entering a keyword like los angeles and getting yellowpages results and so on instead of plumber-lost-angeles.org or whatever.

    @OP you should create a blacklist of these larger sites to filter against. Add their domains into a list in the format of:

    yelp.com
    angieslist.com

    don't add the full url though - just like that ^^

    And then after each scrape do a remove - > remove urls containing and then choose that text file and it will get rid of the entries in there.

    You can always use the - operator and add on things like -review or -ratings to the ends of your keywords; it will help but it won't be perfect.
     
    • Thanks Thanks x 1