1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Need a "URL Spider" that can do this

Discussion in 'Black Hat SEO Tools' started by dabugen, Mar 14, 2011.

  1. dabugen

    dabugen Newbie

    Joined:
    Dec 20, 2010
    Messages:
    8
    Likes Received:
    2
    ... I want to give it a list of URL (all main-sites, so for example, microsoft.com, aol.com, etc.). Then I want it to go through ALL pages on that site and in the end show me the complete URLs that contain a specific keyword within the page-text (for example: "download"). Important: it should NOT use Google to search through the sites, but manually search on each site by following all sub-pages on the main-site. I know that there are programs who do exactly this but only for email addresses. I need it to spit out the full URLĀ“s that contain my specific keyword within the site-text.

    Any idea what program can do this?
     
  2. dabugen

    dabugen Newbie

    Joined:
    Dec 20, 2010
    Messages:
    8
    Likes Received:
    2
    Anyone has any idea about such a spider? Please help. Thank you.