1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

URL list compare tool - bot - windows --> needed

Discussion in 'Hire a Freelancer' started by asiavirtualsolution, Jun 1, 2014.

  1. asiavirtualsolution

    asiavirtualsolution Junior Member

    Joined:
    May 18, 2013
    Messages:
    198
    Likes Received:
    36
    Occupation:
    www.asiavirtualsolutions.com ,Internet Marketing,
    Location:
    Asia
    Home Page:
    I need something that can clean large url list.

    I have a url list, now i scraped another list, but i want only the urls that is new and not already in the old list ( unique urls).

    I want to input: list a + List b = Output should be the difference between the 2
    Option to save the output to txt file

    Conditions:
    • Must be a windows app ( should run on win 8.1
    • Must be 64 bit
    • Should be able to handle url list exceeding 20 million urls ( 10Gb )
    • This is for personal use so no need for any fancy GUI, just need it to be practical

    Drop me a PM if you can do it with your reasonable price and how long it will take to get done, or get me on skype.
     
  2. TehEpidemick

    TehEpidemick BANNED BANNED

    Joined:
    Oct 31, 2011
    Messages:
    778
    Likes Received:
    181
    add me on skype, tehepidemick
     
  3. asiavirtualsolution

    asiavirtualsolution Junior Member

    Joined:
    May 18, 2013
    Messages:
    198
    Likes Received:
    36
    Occupation:
    www.asiavirtualsolutions.com ,Internet Marketing,
    Location:
    Asia
    Home Page:
    Done, waiting for your quote.

     
  4. promark

    promark BANNED BANNED

    Joined:
    Mar 24, 2014
    Messages:
    37
    Likes Received:
    9
    skype me: promark.david
     
  5. asiavirtualsolution

    asiavirtualsolution Junior Member

    Joined:
    May 18, 2013
    Messages:
    198
    Likes Received:
    36
    Occupation:
    www.asiavirtualsolutions.com ,Internet Marketing,
    Location:
    Asia
    Home Page:
    Job has been assigned to : promark
     
  6. davids355

    davids355 Jr. VIP Jr. VIP Premium Member

    Joined:
    Apr 25, 2011
    Messages:
    8,795
    Likes Received:
    6,344
    Home Page:
    Sounds useful, I was thinking about the same thing recently.
     
  7. sohom

    sohom Senior Member

    Joined:
    May 26, 2013
    Messages:
    981
    Likes Received:
    175
    Location:
    not in Past
    Can make this kind of bot
    but 10GB list ummm!
    whats your system config & RAM in your System?
     
    Last edited: Jun 1, 2014
  8. Asif WILSON Khan

    Asif WILSON Khan Executive VIP Premium Member

    Joined:
    Nov 10, 2012
    Messages:
    10,124
    Likes Received:
    28,566
    Gender:
    Male
    Occupation:
    Fun Lovin' Criminal
    Location:
    London
    Home Page:
    10gb list, wow.
    On first reading I was going to say that it can be done with Notepad++ but 10gb I don't know.
     
  9. handmadebots

    handmadebots Senior Member

    Joined:
    Nov 8, 2012
    Messages:
    904
    Likes Received:
    204
    Home Page:
    Interesting. Still, 10 GB is a lot, and it will consume some resources (whatever you do).
     
  10. sockpuppet

    sockpuppet Junior Member

    Joined:
    Nov 7, 2011
    Messages:
    155
    Likes Received:
    145
  11. icomaki

    icomaki Junior Member

    Joined:
    Aug 15, 2013
    Messages:
    191
    Likes Received:
    52
    Wow Promark , Still trying to take people's money by saying u program? Wonder if your ever gonna finish the one u said u started for me? You can either pay me back or I'll open a thread and u won't be able to do this to the people on this form. Thanks