1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

My progress scraping Facebook

Discussion in 'Black Hat SEO Tools' started by huyvun, Sep 29, 2012.

  1. huyvun

    huyvun Newbie

    Joined:
    May 1, 2012
    Messages:
    21
    Likes Received:
    2
    Occupation:
    coder
    I have found a way to scrape facebook without any real problems , as a result I have scraped
    a bit over 5million records of random facebook users..

    The records contain, profile information, friends lists, etc..

    The storage system is a local distributed p2p network, using dynamic hash tables..

    I'd like to release this software, to increase the rate of scraping..

    With as a little as a hundred volunteers, I think we can download most all of the facebook
    social graph, inside a week or two...

    Anyone think this information might be of use ?

    I'd like to post a link to the data i have scraped so far, but I can't post urls in this forum
    yet.
     
    • Thanks Thanks x 1
  2. silentkill3r

    silentkill3r BANNED BANNED

    Joined:
    Jun 25, 2012
    Messages:
    127
    Likes Received:
    70
    m not sure ! but maybe some people would have been looking for it....
     
  3. huyvun

    huyvun Newbie

    Joined:
    May 1, 2012
    Messages:
    21
    Likes Received:
    2
    Occupation:
    coder
    ok, sorry to be sneaky...
    but if you go to my profile, and look at my biography:
    i have put the url to some of this data i have mined..

    in my profile, under about me.. biography i think..
     
  4. GoTRooT

    GoTRooT Jr. VIP Jr. VIP

    Joined:
    Jun 21, 2010
    Messages:
    511
    Likes Received:
    241
    Occupation:
    Englland
    Location:
    Englland
    go speak to the guys at sysomos, this data is gold to them.
     
  5. blackcatavi

    blackcatavi Registered Member

    Joined:
    Aug 11, 2012
    Messages:
    92
    Likes Received:
    9
    Occupation:
    bot developer, wp expert
    Location:
    China
    If you are talking about scraping users.

    I think, facebook has all the info right there, hxxp: / / www(dot)facebook(dot)com / directory / people /
    remove spaces
     
  6. huyvun

    huyvun Newbie

    Joined:
    May 1, 2012
    Messages:
    21
    Likes Received:
    2
    Occupation:
    coder
    yes that is the global directory, but after about 100 hundred scrapes, facebook scraping detection system
    kicks in, and pretty much blocks your efforts..


    i have found a way around this..

    i can scrape all day long at full speed and not worry about FB stopping me.
     
  7. huyvun

    huyvun Newbie

    Joined:
    May 1, 2012
    Messages:
    21
    Likes Received:
    2
    Occupation:
    coder
    does anyone here have the right to post urls ?
     
  8. Standard Toaster

    Standard Toaster Regular Member

    Joined:
    Aug 29, 2009
    Messages:
    335
    Likes Received:
    190
  9. huyvun

    huyvun Newbie

    Joined:
    May 1, 2012
    Messages:
    21
    Likes Received:
    2
    Occupation:
    coder
    thanks for that,, so as you can see some records are very detailed ( for example user664477291 ,, the number is
    their facebook profile id ).
    I'm sure there is some interesting uses , if I can eventually download enough user profiles ( e.g more than 500million of them).
    Of course the data is massive, but i've written something a bit like google's bigtable, so it's not really an issue ( just the time it takes to crawl ).
    I just think its not only unfair, but very dangerous for one company to have access to such data..
    Who ever has access to FB's data base, has unimaginable power...
    I think this data should be open..
     
  10. ddevil459

    ddevil459 Regular Member

    Joined:
    Nov 8, 2008
    Messages:
    228
    Likes Received:
    46
    Have you been able to scrape business information (place of employment and title) as well as public phone?
     
  11. Scritty

    Scritty Elite Member Premium Member

    Joined:
    May 1, 2010
    Messages:
    2,807
    Likes Received:
    4,496
    Occupation:
    Affiliate Marketer
    Location:
    UK
    Home Page:
    Given the scope limiting elements now built into Facebook, the way it only allows certain user interactions, ratios of friends, numbers in groups, numbers of likes. numbers of requests - the way it monitors business pages and fan pages, nulls links if your friend base is too high and all that other shit - and now wants $5 to "promote a post" when I log in - I'm struggling to see a useful application of a huge list of names and facebook ID's.
    Real (none facebook) email addresses YEAH.
    Telephone numbers HELL YEAH.

    But the data your collecting? What are you going to use it for? What CAN you use it for inside or outside of Facebook?

    Genuine question - What is an example of an application for this info?

    Scritty
     
  12. YouFeelMeDawg?

    YouFeelMeDawg? BANNED BANNED

    Joined:
    Aug 10, 2011
    Messages:
    266
    Likes Received:
    371
    Why dont you just like buy some installs and scrape it yourself, i mean after all you say you want just 100 computers scraping at the same time. Getting that many pc's to scrape facebook is easy.
     
  13. Librish

    Librish Newbie

    Joined:
    Oct 24, 2012
    Messages:
    12
    Likes Received:
    1
    I would be very interested in this, I'm working on some projects involving predictions based on Facebook so this would help me immensely!
     
  14. huyvun

    huyvun Newbie

    Joined:
    May 1, 2012
    Messages:
    21
    Likes Received:
    2
    Occupation:
    coder
    FYI:
    If anyone starts testing out the platform, please msg me on skype..
    At present you need to be running windows ( any flavor, except vista ).
     
  15. huyvun

    huyvun Newbie

    Joined:
    May 1, 2012
    Messages:
    21
    Likes Received:
    2
    Occupation:
    coder
    what is the pre-reqs to post links here ?
     
  16. huyvun

    huyvun Newbie

    Joined:
    May 1, 2012
    Messages:
    21
    Likes Received:
    2
    Occupation:
    coder
    i've created a small screencast of my initial runs ( but i cant post links here),
    so please look at the about me in my profile, as i have put the youtube link here ...
    btw- the screencast recorder is extremely slow ( looks like 5x slower than it was running at )
    the tube watch id is v=83BtDoOLmKY
     
  17. keval007

    keval007 Junior Member

    Joined:
    Jun 12, 2012
    Messages:
    145
    Likes Received:
    26
    Occupation:
    Web Scraper & PHP Developer
    In which language you have made this scraper?
     
  18. huyvun

    huyvun Newbie

    Joined:
    May 1, 2012
    Messages:
    21
    Likes Received:
    2
    Occupation:
    coder
    C and assembler
     
  19. julia adam

    julia adam Jr. VIP Jr. VIP Premium Member

    Joined:
    May 29, 2011
    Messages:
    500
    Likes Received:
    25
    Good info mate! Thanks for sharing.
     
  20. pinakin

    pinakin Newbie

    Joined:
    Aug 5, 2012
    Messages:
    2
    Likes Received:
    0
    The raw data won't be of any use, you need to compile it in a way that it can be helpful in decision making !