My progress scraping Facebook

H

huyvun

Newbie
Joined
May 1, 2012
Messages
21
Reaction score
2
I have found a way to scrape facebook without any real problems , as a result I have scraped
a bit over 5million records of random facebook users..

The records contain, profile information, friends lists, etc..

The storage system is a local distributed p2p network, using dynamic hash tables..

I'd like to release this software, to increase the rate of scraping..

With as a little as a hundred volunteers, I think we can download most all of the facebook
social graph, inside a week or two...

Anyone think this information might be of use ?

I'd like to post a link to the data i have scraped so far, but I can't post urls in this forum
yet.
 
Advertise on BHW
ok, sorry to be sneaky...
but if you go to my profile, and look at my biography:
i have put the url to some of this data i have mined..

in my profile, under about me.. biography i think..
 
go speak to the guys at sysomos, this data is gold to them.
 
If you are talking about scraping users.

I think, facebook has all the info right there, hxxp: / / www(dot)facebook(dot)com / directory / people /
remove spaces
 
yes that is the global directory, but after about 100 hundred scrapes, facebook scraping detection system
kicks in, and pretty much blocks your efforts..


i have found a way around this..

i can scrape all day long at full speed and not worry about FB stopping me.
 
does anyone here have the right to post urls ?
 
thanks for that,, so as you can see some records are very detailed ( for example user664477291 ,, the number is
their facebook profile id ).
I'm sure there is some interesting uses , if I can eventually download enough user profiles ( e.g more than 500million of them).
Of course the data is massive, but i've written something a bit like google's bigtable, so it's not really an issue ( just the time it takes to crawl ).
I just think its not only unfair, but very dangerous for one company to have access to such data..
Who ever has access to FB's data base, has unimaginable power...
I think this data should be open..
 
Have you been able to scrape business information (place of employment and title) as well as public phone?
 
huyvun said:
thanks for that,, so as you can see some records are very detailed ( for example user664477291 ,, the number is
their facebook profile id ).
I'm sure there is some interesting uses , if I can eventually download enough user profiles ( e.g more than 500million of them).
Of course the data is massive, but i've written something a bit like google's bigtable, so it's not really an issue ( just the time it takes to crawl ).
I just think its not only unfair, but very dangerous for one company to have access to such data..
Who ever has access to FB's data base, has unimaginable power...
I think this data should be open..
Click to expand...

Given the scope limiting elements now built into Facebook, the way it only allows certain user interactions, ratios of friends, numbers in groups, numbers of likes. numbers of requests - the way it monitors business pages and fan pages, nulls links if your friend base is too high and all that other shit - and now wants $5 to "promote a post" when I log in - I'm struggling to see a useful application of a huge list of names and facebook ID's.
Real (none facebook) email addresses YEAH.
Telephone numbers HELL YEAH.

But the data your collecting? What are you going to use it for? What CAN you use it for inside or outside of Facebook?

Genuine question - What is an example of an application for this info?

Scritty
 
Why dont you just like buy some installs and scrape it yourself, i mean after all you say you want just 100 computers scraping at the same time. Getting that many pc's to scrape facebook is easy.
 
I would be very interested in this, I'm working on some projects involving predictions based on Facebook so this would help me immensely!
 
FYI:
If anyone starts testing out the platform, please msg me on skype..
At present you need to be running windows ( any flavor, except vista ).
 
what is the pre-reqs to post links here ?
 
i've created a small screencast of my initial runs ( but i cant post links here),
so please look at the about me in my profile, as i have put the youtube link here ...
btw- the screencast recorder is extremely slow ( looks like 5x slower than it was running at )
the tube watch id is v=83BtDoOLmKY
 
The raw data won't be of any use, you need to compile it in a way that it can be helpful in decision making !
 
Advertise on BHW
You must log in or register to reply here.
Sign up now!

Main Menu

Home Main Forum List Black Hat SEO White Hat SEO BHW Newbie Guide Blogging Black Hat Tools Social Media Downloads

Marketplace

Account Selling Content / Copywriting EBooks / Methods Hosting Misc Proxies For Sale SEO - Link building SEO - Packages Social Media Social Media - Panels Web Design More

Making Money

Affiliate Programs Hire a Freelancer IM Journeys Making Money Pay Per Click Site Flipping Want to Buy

BlackHat World

Dispute Resolution Forum Suggestions & Feedback Introductions Lounge
Back
Top