1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Ways to Scrape URLs

Discussion in 'BlackHat Lounge' started by killerseonic, Apr 25, 2013.

  1. killerseonic

    killerseonic Junior Member

    Joined:
    Feb 5, 2013
    Messages:
    132
    Likes Received:
    21
    Location:
    Here + There
    Home Page:
    So I am using Free URL Scraper, its a easy PHP script witch you can modify with filezilla to search only for urls in your language.

    But maybe there are better ways out there to collect urls? *PM also searching for lists..

    cheers
     
  2. YouFeelMeDawg?

    YouFeelMeDawg? BANNED BANNED

    Joined:
    Aug 10, 2011
    Messages:
    266
    Likes Received:
    371
    Ofcourse there are better ways to scrape.

    I would go with scrapy if i was you, or you can go for slybot. Slybot is based on scrapy and scrapely but they do have some cool functions where you can extract generic forms. By the way, slybot actually is a lot better since you can pretty much extract a lot more stuff with scrapely , although I have to say I rather use BeautifulSoup to parse html.
     
  3. divok

    divok Senior Member

    Joined:
    Jul 21, 2010
    Messages:
    1,015
    Likes Received:
    634
    Location:
    http://twitter.com/divok
    That depends for what purpose are you scraping the url's , for link building use scrapebox(manual) otherwise for some other purpose I would use scrapy for my own crawlers .
     
  4. killerseonic

    killerseonic Junior Member

    Joined:
    Feb 5, 2013
    Messages:
    132
    Likes Received:
    21
    Location:
    Here + There
    Home Page:
    Actually I want to scrape urls for my domain valuation website to grow the database :)

    will try slybot now, thx for the suggestion