1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Need help with...Spider/Crawling web

Discussion in 'General Programming Chat' started by xmonkeys, Mar 19, 2013.

  1. xmonkeys

    xmonkeys Newbie

    Joined:
    Jan 17, 2011
    Messages:
    23
    Likes Received:
    1
    Hello,

    I am looking for someone or someway to crawl free adult sites, and post their videos on my sites. I have access to their csv/dump, but I want to create an automated way of getting them without having to continually manage them.

    Does anyone know of a good spider, or crawler that can get the necessary info?


    thanks.
     
  2. seriousjoker

    seriousjoker Registered Member

    Joined:
    Feb 12, 2013
    Messages:
    75
    Likes Received:
    33
    I can make one for you, if you are intrested.
     
  3. innozemec

    innozemec Jr. VIP Jr. VIP

    Joined:
    Aug 19, 2011
    Messages:
    5,288
    Likes Received:
    1,799
    Location:
    www.Indexification.com
    Home Page:
  4. jazzc

    jazzc Moderator Staff Member Moderator Jr. VIP

    Joined:
    Jan 27, 2009
    Messages:
    2,468
    Likes Received:
    10,143
    Wget, curl, aria2c, teleport pro
     
  5. It Should Be Easy

    It Should Be Easy Newbie

    Joined:
    Mar 18, 2013
    Messages:
    7
    Likes Received:
    0
    Since you said "someway" I am not sure if you are a developer yourself. But the Scrapy framework in Python is great for almost any kind of crawling tasks.