1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Url harvesting problem

Discussion in 'Black Hat SEO' started by antsaoo, Mar 25, 2009.

  1. antsaoo

    antsaoo Supreme Member

    Joined:
    Oct 1, 2008
    Messages:
    1,291
    Likes Received:
    637
    Soo

    Id like to harvest 12k url from same key phrase or more likely from same website but google only gives up to 1000 =/ best way around this ?
     
  2. kouxx

    kouxx Junior Member

    Joined:
    Aug 2, 2008
    Messages:
    160
    Likes Received:
    38
    [your querry] + [keyword list]

    edit:
    hit the reputation button :p
     
    Last edited: Mar 26, 2009
  3. antsaoo

    antsaoo Supreme Member

    Joined:
    Oct 1, 2008
    Messages:
    1,291
    Likes Received:
    637
    i think site doesn't matter ;) Well i tried with Diff KW got first like 6k results, when removed duplicates had 1.9k left lol :p oh well , thats a start.
     
  4. kouxx

    kouxx Junior Member

    Joined:
    Aug 2, 2008
    Messages:
    160
    Likes Received:
    38
    drop me a pm... if you want

    i have a tool for you..
     
  5. justone

    justone Elite Member

    Joined:
    Oct 12, 2008
    Messages:
    1,516
    Likes Received:
    1,037
    Occupation:
    -
    Location:
    Europe
    also it is important to know that the number of hits google shows is usually WAY wrong.
    If google says it knows a million websites about a topic it is very possible that you can not get out more than a few % of them and google does not really know much more.

    I don't know how they estimate the total number of sites but that algorithm is very inaccurate at least.
     
  6. deef

    deef Junior Member

    Joined:
    Dec 16, 2008
    Messages:
    101
    Likes Received:
    17
    I have xrumer and hrefer.. could run for you a hrefer search.
     
  7. genie1

    genie1 Junior Member

    Joined:
    May 30, 2008
    Messages:
    151
    Likes Received:
    50
    Home Page:
    I did some reading on this a few months ago because I ran into the same problem. It ''seems'' and I might be wrong, that its part of google's design not to go past 1000 results. They claim something like ''if you can't find it in the first 1000 results, your search isn't focused enough'' lol
     
  8. justone

    justone Elite Member

    Joined:
    Oct 12, 2008
    Messages:
    1,516
    Likes Received:
    1,037
    Occupation:
    -
    Location:
    Europe
    and don't forget: if you hammer google too much you get soft banned which requires a captcha to continue.
    If you continue to hammer google they temp ban your ip.
    If you continue to use ips in the same subnet the same process happens.
    if you keep hammering they temp ban the whole subnet.
    After some hours to days the ban is lifted, no idea what happens if you keep scraping google, I read about permanent bans in various mailing lists.