Data Scraping
- Thread starter blackbob1
- Start date
TZ2011
Senior Member
- Jun 26, 2011
- 1,095
- 1,131
Question is too wide to be answered just like that.
But in general, pretty much everything is doable with curl and regex, there is a plenty of tutorials and half-done scripts on internet to look for, you will need to get the hands "dirty" to make it to work since each site has their structure and you will need to setup regex and specific elements to extract.
You can start from here and see is it that what you need. If you need something simplier, you can try Chrome extension Scraper extension , if you need something more complex you can try with Python (if you are into programing) etc etc it's really up to your task and site that you need to work on.
edit: I think that I misunderstood your question. You are looking to scrape specific elements on website or to put elements from your site in another database ? Can you be more specific ?
But in general, pretty much everything is doable with curl and regex, there is a plenty of tutorials and half-done scripts on internet to look for, you will need to get the hands "dirty" to make it to work since each site has their structure and you will need to setup regex and specific elements to extract.
You can start from here and see is it that what you need. If you need something simplier, you can try Chrome extension Scraper extension , if you need something more complex you can try with Python (if you are into programing) etc etc it's really up to your task and site that you need to work on.
edit: I think that I misunderstood your question. You are looking to scrape specific elements on website or to put elements from your site in another database ? Can you be more specific ?
Last edited:
You may use imacros to get the data and save it into .csv database
Here is a sample code:
Here is a sample code:
Code:
VERSION BUILD=7601105 RECORDER=FX
SET !ERRORIGNORE YES
TAB T=1
CLEAR
URL GOTO=http://www.blackhatworld.com/blackhat-seo/cloaking-content-generators/591267-data-scraping.html
TAG POS=1 TYPE=TITLE ATTR=* EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Data.csvWell, I think you pretty much answered your own question
If the data is "very specific," then it is unlikely that you will find a ready script or software and you will have to build a custom. Go to the programming section here and see if someone is willing to help
Debugger
Junior Member
- Aug 16, 2009
- 174
- 34
I was searching on net.But did not find tuts like one you've.Do you have more links for tuts..more complex thingy..i can code in python or any language ..if you know any please do tell..thanks
thx for this looks perfect I can get bits of it to work but looks like theres problems with the way urls are form I'll post in programming the specific question
competent123
Registered Member
- May 30, 2009
- 55
- 14
arent' normal scrapers good enough to do that, assuming its the usual targets ( wordpress/forums?)
no its products off an ecommerce site to populate Magento, after looking into it I dont think it is possible (with my limited knowledge)
i used to scrape sites daily for title, description, price etc. There's a certain methodology that i developed that worked very well.
1st you will need to scrape all URLs of the pages that you need to scrape data from - as someone mentioned above, iMacros works magic here. you will need to create a macro to scrape the URL's of All the pages that need to be scraped for the info you mentioned (i believe title, desc, etc.), and put those URLS in a CSV file.
2nd step, you will need to create another iMacro to "loop" through the CSV file and go to each one of the URL's in the file and scrape the necessary info into another CSV file.
the key here is that all pages that need to be scraped have the same HTML layout, or else your iMacro will fail out.
PM if you need more help, i can try to help you out.
1st you will need to scrape all URLs of the pages that you need to scrape data from - as someone mentioned above, iMacros works magic here. you will need to create a macro to scrape the URL's of All the pages that need to be scraped for the info you mentioned (i believe title, desc, etc.), and put those URLS in a CSV file.
2nd step, you will need to create another iMacro to "loop" through the CSV file and go to each one of the URL's in the file and scrape the necessary info into another CSV file.
the key here is that all pages that need to be scraped have the same HTML layout, or else your iMacro will fail out.
PM if you need more help, i can try to help you out.