Scraping from a JSON help (novice scraper run into issues)

ad9051 · May 15, 2013

Hello I am trying to scrape the contents from a website, the data I require from the webpage however is hidden in a JSON. I have tried loading the JSON but it returns 10 random users, where as the website contains 1400 total users . Is it still possible to scrape data from JSON's ?

note. this scraping isn't of personal details, no emails, no names or anything like that (just in case anyone questions the morality of the scraping)

Any help or comments would be greatly appreciated.

Kind regards

Alan

MrBlue · May 15, 2013

If you are not afraid of a little server side Javascript take a look at Substack's "json-scrape" package. I use this a lot and it works very well.

Code:

https://github.com/substack/json-scrape

ad9051 · May 15, 2013

Thanks for the quick reply Mr Blue, Is this the only way to scrape from JSON or is there other ways? I am hoping to create a script to scrape the website every x mins on repeat so using a programme to manually scrape the JSON is far from ideal.

Many thanks

Alan

MrBlue · May 15, 2013

There are many ways to skin a cat

Do you have a platform and language preference? Automating your script to run can easily be achieved using cron jobs in Linux. Feel free to pm me the url of the site in question. I'll gladly take a look and give you my opinion.

Cheers!

ad9051 said:
Thanks for the quick reply Mr Blue, Is this the only way to scrape from JSON or is there other ways? I am hoping to create a script to scrape the website every x mins on repeat so using a programme to manually scrape the JSON is far from ideal.

Many thanks

Alan

ad9051 · May 15, 2013

Thanks Mr Blue, Ill send you a DM

MrBlue said:
There are many ways to skin a cat Do you have a platform and language preference? Automating your script to run can easily be achieved using cron jobs in Linux. Feel free to pm me the url of the site in question. I'll gladly take a look and give you my opinion.

Cheers!

ad9051 · May 15, 2013

ad9051 said:
Thanks Mr Blue, Ill send you a DM

Ahh apparently I need to have 15 posts on the forum before I can DM ppl. Not sure if you can dm me? If you can skype chat or email you, if that is agreeable with you. Cheers Alan

Zapdos · May 15, 2013

http://php.net/manual/en/function.json-decode.php
Include that with a cron job and voila.

edit:
Some pseudo-code:

Code:

$url = "http://www.website.com/json";
$contents = json_decode(file_get_contents($url));
$username = $contents['user']['username'];

MrBlue · May 16, 2013

PM sent. Add me to Skype.

Cheers!

ad9051 said:
Ahh apparently I need to have 15 posts on the forum before I can DM ppl. Not sure if you can dm me? If you can skype chat or email you, if that is agreeable with you. Cheers Alan

Scraping from a JSON help (novice scraper run into issues)

ad9051

Registered Member

MrBlue

Senior Member

ad9051

Registered Member

MrBlue

Senior Member

ad9051

Registered Member

ad9051

Registered Member

Zapdos

Power Member

MrBlue

Senior Member

Main Menu

Marketplace

Making Money

BlackHat World