Bulk URL Scrapper (NEEDED)

TheTRUTH

TheTRUTH

Regular Member
Joined
Oct 1, 2008
Messages
344
Reaction score
40
I have a huge list of urls.I need a program that will go to all thoose urls and scrap the external links.Does such a program exist
 
Advertise on BHW
Which platform are these sites running on? or all different kinds of pages?
 
If these are just standard webpages it would take longer to make a cup of coffee then build a script to do this. just get some one with winautomation or ubot to bang one out
 
Scrapebox only grabs emails.

These are just websites
 
I am in the process of doing the same type of operation. What do you want to do with the links. Is there content on the pages that you want to scrape, email addresses, or are you just building links? I think there are several approaches to what you want to do.

Try Extract Link 3.0 and see what it does for you.

Does this help? Reply with more details and I should be of assistance as this is something I do often. Also, I think the second comment may be correct and you can pay someone $15 on ScriptLance to do the job for you.
 
Thanks for jumping in ScrapeBox developer. I hate to ask if you are right (as you know the product better than anyone else), but are you saying that Scrapbox will allow me to dump in a list of URL's and that ScrapBox will go to each link and extract all of the URL's and do this over and over for each link in the initial list?

If so, that is great to hear. I was just mentioning on here today that I own ScrapBox and have for about 3 months but I have never opened it. I think I've activate it but I thought it without fully knowing what I was getting into. Either way, this will be a great reason to get into it.

With that said, where do we get the Link Extractor Addon?
 
Thanks, I thought i had seen somthing liek that, but then i seen the grab emails button and thought I was mistaken.
 
CoyoteAssassin said:
Thanks for jumping in ScrapeBox developer. I hate to ask if you are right (as you know the product better than anyone else), but are you saying that Scrapbox will allow me to dump in a list of URL's and that ScrapBox will go to each link and extract all of the URL's and do this over and over for each link in the initial list?

If so, that is great to hear. I was just mentioning on here today that I own ScrapBox and have for about 3 months but I have never opened it. I think I've activate it but I thought it without fully knowing what I was getting into. Either way, this will be a great reason to get into it.

With that said, where do we get the Link Extractor Addon?
Click to expand...

Open up Scrapebox, go to "Addons" and download it from there.
 
I normally do not give out more than one thanks per forum, but I really appreciate everyone who chimed in. Scrapbox has now paid for itself - even if all I use it for is to extract links. I have other methods but I was unable to dump a list into those programs and get the program to extract the links for each page.

This works like a charm. And for anyone who stumbles upon this tread looking for a solution to extract links, buy scrapbox. not only is it good at a lot of other things, it gets the job done quickly and very easily.

Thanks again.

Also, here are the instructions for those who stumble upon the post.


  1. Buy ScrapBox
  2. Click on Addons
  3. Click on Show Available Addons
  4. Once the list populates, click on Link Extractor and click Install
  5. Once the install has taken place, click on Addons > Link Extractor.
  6. In the new window, click on Import/Export and select your method of importing your URL list.
  7. Be sure to click on Internal sites unless you only want External Sites. (Sweet - it would be great if there was an Internal only feature).
  8. With the list of URL's now in the window, click Start
  9. Once finished, click Import/Export and export your links into a .txt file.

I hope this helps!
 
Also, on another note, you can only use Scrapbox to extract full URL's. For example HTML with the link below (your internal sites) will not extract even with the Internal feature turned on.

This will do the trick though - http://www. webmaster-toolkit .com/link-extractor.shtml
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top