1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[REQ] Duplicate URL finder in search result?

Discussion in 'Black Hat SEO Tools' started by Alpenkorps, Apr 10, 2013.

  1. Alpenkorps

    Alpenkorps BANNED BANNED

    Joined:
    Dec 31, 2011
    Messages:
    57
    Likes Received:
    14
    Hi Guys,

    I am working on a data mining project where I search on google for something, visit the results manually and collect what I need. I need one site only once, I need to avoid listing them twice.

    After a few hundred pages, I am constantly finding dup links. I can't remember all the pages I've already visited, I am only human! So most of the time, I revisit a link, then do another search on my excel sheet, and confirm its a dup .. total waste of time. I am at some point when 75% of the links I am visiting is dupe, I have already visited them !!

    I need someway, probably a google chrome plugin, that will mark the pages in search result that already visited once, or any other method that might save me some time.

    You help is greatly appreciated, Thank You.
     
  2. Maiami

    Maiami Registered Member

    Joined:
    Dec 17, 2009
    Messages:
    51
    Likes Received:
    11
    Occupation:
    novilinks.com
    Home Page:
    just grab it, and then check it for Duplicate URL
     
  3. Alpenkorps

    Alpenkorps BANNED BANNED

    Joined:
    Dec 31, 2011
    Messages:
    57
    Likes Received:
    14
    Grab it with what? Do you recommend any tool?
     
  4. ija61

    ija61 Senior Member

    Joined:
    Mar 2, 2011
    Messages:
    965
    Likes Received:
    635
    Gender:
    Male
    Occupation:
    The first SEO economist:)
    Location:
    Romania
    Sorry but what I can't understand is if those hundreds links are from the same webpage or are G result?

    Anyway I suggest that you use a boot or try a method to automate this...
     
  5. blackouts

    blackouts Regular Member

    Joined:
    Oct 7, 2011
    Messages:
    292
    Likes Received:
    244
    Location:
    BRAZIL!
    Scrapebox