1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[WTB] Looking for someone to fetch an entire website (165000 files + 165000 images)

Discussion in 'Hire a Freelancer' started by H2DAZ, May 5, 2011.

  1. H2DAZ

    H2DAZ Newbie

    Joined:
    Oct 10, 2010
    Messages:
    15
    Likes Received:
    0
    Hello,

    I have a list of 6223 URLS that I need to fetch in order to get the URLS of approximately 165 000 other pages + 165 000 images matching the page (1 image per page). There is approximately 45 GB of data to get only from the images but I need the output of the matching pages too.

    That website is very well protected from crawlers and proxies. It bans complete IP classes VERY easily.

    I need someone who thinks who can do this, and I will pay him for it.

    Just PM me or reply here.

    Thank you
     
  2. mohitjmehta

    mohitjmehta BANNED BANNED

    Joined:
    Sep 17, 2010
    Messages:
    1,018
    Likes Received:
    783
    i m not in race BUT just try to help you
    either use blackwidow or IDM site downloader
     
  3. H2DAZ

    H2DAZ Newbie

    Joined:
    Oct 10, 2010
    Messages:
    15
    Likes Received:
    0
    Thing is a usual crawler won't do the trick, it is highly IP restricted...
     
  4. sam.hunt0710

    sam.hunt0710 Jr. VIP Jr. VIP

    Joined:
    Feb 22, 2011
    Messages:
    3,383
    Likes Received:
    1,230
    Gender:
    Male
    Occupation:
    Owner Of a IT Comapny
    Location:
    India
  5. H2DAZ

    H2DAZ Newbie

    Joined:
    Oct 10, 2010
    Messages:
    15
    Likes Received:
    0
    thank you sam i will look into that
     
  6. H2DAZ

    H2DAZ Newbie

    Joined:
    Oct 10, 2010
    Messages:
    15
    Likes Received:
    0
    the 165000 files + 165000 images URLS have been determined, so i have all the urls needed. I now just need someone able to fetch all that content for me. anyone ?
     
  7. aca2k2

    aca2k2 Registered Member

    Joined:
    Dec 31, 2007
    Messages:
    97
    Likes Received:
    39
    Hope i can, i have done similar things a lot before.

    I have already done some same kind of image fetching online for someone here. Let me know the website and give me some urls set. I will try and let you know whether i could fetch all those 165000 images and files.