1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is Facebook crawling the web too?

Discussion in 'Black Hat SEO' started by bartosimpsonio, Jul 28, 2014.

  1. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    8,829
    Likes Received:
    7,441
    Occupation:
    ZLinky2Buy SEO Services
    Location:
    ⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩⇩
    Home Page:
    Apparently the Facebook user agent (facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) ) is supposed to be a referral hit from within the social network.

    So I ran a test with a hidden page, it's a 1x1 link deep hidden into some random page(single page) on a site. Turns out that hidden link just got a hit from one of their IPs:

    NetRange: 173.252.64.0 - 173.252.127.255
    CIDR: 173.252.64.0/18
    OriginAS: AS32934
    NetName: FACEBOOK-INC
    NetHandle: NET-173-252-64-0-1
    Parent: NET-173-0-0-0-0
    NetType: Direct Assignment
    RegDate: 2011-02-28
    Updated: 2012-02-24
    Ref: http://whois.arin.net/rest/net/NET-173-252-64-0-1

    It's impossible to have that link linked from within FB. Unless of course the target page was linked and FB crawled all links within it.

    Is FB becoming a search engine? It'd make total sense if they did. Scary thought as well, to have everyone live in their artificial little world and forget the once free WWW...
     
  2. prab1996

    prab1996 Elite Member

    Joined:
    Jan 8, 2013
    Messages:
    3,496
    Likes Received:
    2,027
    Occupation:
    your gf's <3 ♥♥♥♥
    Location:
    Prab1996.com
    Home Page:
    there are many sites who send bots to crawl your sites. if fb has sent a bot then it's not a news.
    -=-
     
  3. VoidITSolutions

    VoidITSolutions BANNED BANNED

    Joined:
    Apr 5, 2013
    Messages:
    164
    Likes Received:
    44
    I've noticed during some spam campaigns that Facebook would crawl my site and block urls in it from me spamming the base url. They also check sites for malicious content I believe, don't hold me to it though.
     
    • Thanks Thanks x 1
  4. Automated

    Automated Regular Member

    Joined:
    Jun 7, 2012
    Messages:
    289
    Likes Received:
    123
    Location:
    Online
    Def makes sense and wouldn't surprise me...
     
  5. silvermember

    silvermember Regular Member

    Joined:
    Apr 16, 2013
    Messages:
    243
    Likes Received:
    87
    Location:
    Chained on Earth Gravity
    it does for certain in my opinion since they learned from G and in a way it is a SE anyway at all - why shouldn't they do things what we are doing in order to get along and get paid for our effort!!
    the only different I see is when they are doing it its "Legal"!!
     
  6. SharkServers

    SharkServers Regular Member

    Joined:
    Jun 29, 2014
    Messages:
    380
    Likes Received:
    181
    Occupation:
    Web Hosting
    Location:
    DMCA? Pff! www.SuckMyBallsDM.CA
    Home Page:
    I noticed visits from that IP range yesterday when updating some posts on our company blog. As soon as I hit the "Update" button, a visit from FB owned IP would happen to that page. As the only social plugin I use on the blog is Shareaholic, I suppose it sends some sort of a ping to them when a post is updated, but who knows...