1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Bulk URL Scrapper (NEEDED)

Discussion in 'Black Hat SEO Tools' started by TheTRUTH, May 20, 2010.

  1. TheTRUTH

    TheTRUTH Regular Member

    Joined:
    Oct 1, 2008
    Messages:
    345
    Likes Received:
    36
    Location:
    In the sky baby!
    I have a huge list of urls.I need a program that will go to all thoose urls and scrap the external links.Does such a program exist
     
    • Thanks Thanks x 1
  2. Scripteen

    Scripteen Elite Member

    Joined:
    Sep 19, 2009
    Messages:
    1,811
    Likes Received:
    1,918
    Home Page:
    Which platform are these sites running on? or all different kinds of pages?
     
  3. *zap*

    *zap* Regular Member

    Joined:
    Apr 7, 2008
    Messages:
    349
    Likes Received:
    98
    Occupation:
    none
    Location:
    Uk
    If these are just standard webpages it would take longer to make a cup of coffee then build a script to do this. just get some one with winautomation or ubot to bang one out
     
  4. Dayum

    Dayum BANNED BANNED

    Joined:
    Mar 22, 2010
    Messages:
    271
    Likes Received:
    90
    scrapebox
     
    • Thanks Thanks x 1
  5. TheTRUTH

    TheTRUTH Regular Member

    Joined:
    Oct 1, 2008
    Messages:
    345
    Likes Received:
    36
    Location:
    In the sky baby!
    Scrapebox only grabs emails.

    These are just websites
     
  6. CoyoteAssassin

    CoyoteAssassin Elite Member

    Joined:
    Jan 3, 2010
    Messages:
    1,862
    Likes Received:
    3,906
    Occupation:
    Full Time IMer
    Location:
    USA
    I am in the process of doing the same type of operation. What do you want to do with the links. Is there content on the pages that you want to scrape, email addresses, or are you just building links? I think there are several approaches to what you want to do.

    Try Extract Link 3.0 and see what it does for you.

    Does this help? Reply with more details and I should be of assistance as this is something I do often. Also, I think the second comment may be correct and you can pay someone $15 on ScriptLance to do the job for you.
     
  7. Sweetfunny

    Sweetfunny Jr. VIP Jr. VIP Premium Member

    Joined:
    Jul 13, 2008
    Messages:
    1,747
    Likes Received:
    5,039
    Location:
    ScrapeBox v2.0
    Home Page:
    • Thanks Thanks x 2
  8. CoyoteAssassin

    CoyoteAssassin Elite Member

    Joined:
    Jan 3, 2010
    Messages:
    1,862
    Likes Received:
    3,906
    Occupation:
    Full Time IMer
    Location:
    USA
    Thanks for jumping in ScrapeBox developer. I hate to ask if you are right (as you know the product better than anyone else), but are you saying that Scrapbox will allow me to dump in a list of URL's and that ScrapBox will go to each link and extract all of the URL's and do this over and over for each link in the initial list?

    If so, that is great to hear. I was just mentioning on here today that I own ScrapBox and have for about 3 months but I have never opened it. I think I've activate it but I thought it without fully knowing what I was getting into. Either way, this will be a great reason to get into it.

    With that said, where do we get the Link Extractor Addon?
     
  9. TheTRUTH

    TheTRUTH Regular Member

    Joined:
    Oct 1, 2008
    Messages:
    345
    Likes Received:
    36
    Location:
    In the sky baby!
    Thanks, I thought i had seen somthing liek that, but then i seen the grab emails button and thought I was mistaken.
     
  10. Dayum

    Dayum BANNED BANNED

    Joined:
    Mar 22, 2010
    Messages:
    271
    Likes Received:
    90
    Open up Scrapebox, go to "Addons" and download it from there.
     
    • Thanks Thanks x 1
  11. CoyoteAssassin

    CoyoteAssassin Elite Member

    Joined:
    Jan 3, 2010
    Messages:
    1,862
    Likes Received:
    3,906
    Occupation:
    Full Time IMer
    Location:
    USA
    I normally do not give out more than one thanks per forum, but I really appreciate everyone who chimed in. Scrapbox has now paid for itself - even if all I use it for is to extract links. I have other methods but I was unable to dump a list into those programs and get the program to extract the links for each page.

    This works like a charm. And for anyone who stumbles upon this tread looking for a solution to extract links, buy scrapbox. not only is it good at a lot of other things, it gets the job done quickly and very easily.

    Thanks again.

    Also, here are the instructions for those who stumble upon the post.


    1. Buy ScrapBox
    2. Click on Addons
    3. Click on Show Available Addons
    4. Once the list populates, click on Link Extractor and click Install
    5. Once the install has taken place, click on Addons > Link Extractor.
    6. In the new window, click on Import/Export and select your method of importing your URL list.
    7. Be sure to click on Internal sites unless you only want External Sites. (Sweet - it would be great if there was an Internal only feature).
    8. With the list of URL's now in the window, click Start
    9. Once finished, click Import/Export and export your links into a .txt file.

    I hope this helps!
     
  12. CoyoteAssassin

    CoyoteAssassin Elite Member

    Joined:
    Jan 3, 2010
    Messages:
    1,862
    Likes Received:
    3,906
    Occupation:
    Full Time IMer
    Location:
    USA
    Also, on another note, you can only use Scrapbox to extract full URL's. For example HTML with the link below (your internal sites) will not extract even with the Internal feature turned on.

    This will do the trick though - http://www. webmaster-toolkit .com/link-extractor.shtml
     
  13. cybermoney

    cybermoney Newbie

    Joined:
    Oct 6, 2010
    Messages:
    2
    Likes Received:
    0
    please help me how to extract url link site for extracting email.thank you