best method of scraping WP sites ?

[r000k]

So guys whats the best way to scrape a wordpress site ? not looking to copy site, basically scrape certain articles and what not

Im sure SB could do it (which I own) or another tool maybee, but I like making my own basic tools to learn , and so I can load up 100 sites, hit one button and have it spit out my desired results in my desired format.

Not all the sites im targeting have sitemaps, so scraping site.com/sitemap.xml wont work
Most sites have different permalink structure's , so no really scraping site.com/page_1 etc.

Is there a slight trick Im missing for easy scraping ?

Ive started writing a (crappish) spider and doing it that way, im about half way through. I guess with that method it will work for non WP sites as well....

Cheers,

 

[Schvamp]

Why would you create your own spider if you have scrapebox?
Scrape sites with WP footprint in SB -> save the list and scrape the list with the 'site:' function that way you will get alot of urls to innerpages -> Send that list to an article grabber or whatever.

 

[sire243]

Best way would be to code it yourself. Use regex since it's faster. What I would do is:

1) Use scrapebox to scrape all the urls under that domain.
2) then just create a simple scraper that goes to a website and scrapes.

Or you could be all complicated and code up a spider, but I wouldn't advise that since it will hog resources like a bitch. (And besides, you're still learning)

 

[r000k]

Why would you create your own spider if you have scrapebox?
Scrape sites with WP footprint in SB -> save the list and scrape the list with the 'site:' function that way you will get alot of urls to innerpages -> Send that list to an article grabber or whatever.

I havnt tried to code in about 10 months, im mainly doing it to brush up on my limited skill set. Plus when Im finished ill just be able to load a list of sites and it spit out the data is csv.

Best way would be to code it yourself. Use regex since it's faster. What I would do is:

1) Use scrapebox to scrape all the urls under that domain.
2) then just create a simple scraper that goes to a website and scrapes.

Or you could be all complicated and code up a spider, but I wouldn't advise that since it will hog resources like a bitch. (And besides, you're still learning)

Yeah the best way woud be to use scrapebox i guess, But I think ill attempt at making a crummy spider, ive got no problems with using regexs an threading and just run it on my vps... if i fail, ill just resort to SB.


thanks for the replies.

 

[sire243]

Maybe this will help you then.

[URL]http://www.makeuseof.com/tag/build-basic-web-crawler-pull-information-website/[/URL]

It's in php, but I'm sure you'll get some ideas.

 

[r000k]

well I got this to work, first I checked if site.com/sitemap.xml was present if not then I spider ed the site

These are the two resources that helped me out if anyone is looking to do the same thing as I was, I had to make a few adjustments, ie change the regex's so it would only spider/scrape the 1 site and add a little threading.

http://www.dreamincode.net/forums/topic/160644-creating-a-web-spider/
http://tech.reboot.pro/showthread.php?tid=320

 

[soundclicktop]

I need soundclick top / program / chart increaser