1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How To Check If 20,000+ LInks are Indexed

Discussion in 'Black Hat SEO' started by aboutusnow, Dec 11, 2011.

  1. aboutusnow

    aboutusnow Newbie

    Joined:
    Dec 20, 2009
    Messages:
    2
    Likes Received:
    0
    I have 20,000 links to check to see if they are indexed, and this number is growing by about 10,000 links a week. So my goal is to determine if these backlinks are indexed or not.

    Here is where I am so far:

    1. Scrapebox - I have scrapebox, but the problem I run into is that my IPs get blocked after about 500 index calls. I scraped free proxies using SB proxy scraper, and ended up with 10 proxies. Those are the ones that died after 500 URLs. I used to have 10 squidproxy private proxies, but I wonder if those 10 would die just as quickly?

    2. Indexbear - I tried this free app and ran a test of 170 links. It showed 165 were indexed, and I knew that wasn't true. Scrapebox usually shows I have 10%-20% indexed. So I checked the links manually, and sure enough, they were not indexed, so I didn't see how Indexbear was going to be much use.

    3. So I know I have seen onine apps that you pay a monthly fee for and it will track your keyword ranking and backlinks to see if they are indexed. I hate to go this route because it will be another $100 or so a month. But if I do have to go this way, does anyone have any good suggestions?

    Or maybe other suggestions overall? I'm interested in getting SB to work since I own it, but it seems I am going to have to get a bunch of proxies or find a way to spread the index checking a little every few minutes.
     
  2. SuperLinks

    SuperLinks Elite Member

    Joined:
    Jul 14, 2008
    Messages:
    2,903
    Likes Received:
    847
    Location:
    New York
    Scrapebox matched with public proxies is really the only answer....

    For public proxies I'd recommend using ProxyGo's service, he scrapes
    and finds proxies to use and provides lists about 2x a day, except for
    weekends.

    That way you can have thousands of potential proxies, rather than a
    handful that you'll find with Scrapebox's proxy tester.

    Otherwise you may need something custom coded, which likely wouldn't
    cost much and could likely be automated entirely.

    The real question is why do you care that much about whether or not they
    are indexed? Use indexing services and automate the process, those links
    that will get indexed WILL get indexed.
     
    • Thanks Thanks x 1
  3. aboutusnow

    aboutusnow Newbie

    Joined:
    Dec 20, 2009
    Messages:
    2
    Likes Received:
    0
    Well, I gathered that getting links indexed was pretty important. Most of these links are links to my main links (tier2 or tier 3 links, if you will), and I am running all of them through backlinkindexer. In fact, I was curious as to how well BI was working which is one reason I wanted to run them through ScrapeBox.