1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How do I scrape 10.000 emails from one tricky page

Discussion in 'Black Hat SEO Tools' started by mindmaster, Aug 23, 2011.

  1. mindmaster

    mindmaster Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 16, 2010
    Messages:
    2,501
    Likes Received:
    1,135
    Location:
    at my new office
    I need to scrape a page on linkedin. It has around 10.000 emails, but has that fricking "Show more comments" (similar to twitter I guess).

    I tried viewing source code, but it shows only 50.

    Would scrapebox be able to scrape all 10.000 and go beyond that 'show more comments' in some way?

    Appreciated.
    Thanks and Rep for who ever helps me.
     
  2. wkrappen91

    wkrappen91 Power Member

    Joined:
    Sep 9, 2010
    Messages:
    588
    Likes Received:
    720
    Location:
    127.0.0.1
    Example page?
    Source code?
    URL?
     
    • Thanks Thanks x 1
  3. Mosquera

    Mosquera Newbie

    Joined:
    Feb 24, 2009
    Messages:
    32
    Likes Received:
    18
    You can try with site:linked/page and adding a few more modifiers that might help. Also, if you tell me the way that comments page are addressed, it can get pretty easy (e.g. comments-page-1, comments-page-2, etc.)
     
    • Thanks Thanks x 1
  4. Gradimir Stankovic

    Gradimir Stankovic Power Member

    Joined:
    Jan 10, 2010
    Messages:
    737
    Likes Received:
    845
    Location:
    404 not found
    Autopager? Imacros?

    PM me the link and I could probably set up some scraper for you...
     
    • Thanks Thanks x 1
  5. mindmaster

    mindmaster Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 16, 2010
    Messages:
    2,501
    Likes Received:
    1,135
    Location:
    at my new office
    The link is a discussion inside a Linkedin group. Even if I post you can see it only if you are part of the group. However here is the source code:
    Code:
    view-source:http://www.linkedin.com/groupItem?trk=group_most_popular-0-b-ttl&gid=3762&view=&qid=08742f44-6eec-4c9f-b565-ad6482ffe61c&goback=%2Egmp_3762&item=3285020&type=member
    The URL dose not change when you clip 'show more comments'.
     
    Last edited: Aug 23, 2011
  6. wkrappen91

    wkrappen91 Power Member

    Joined:
    Sep 9, 2010
    Messages:
    588
    Likes Received:
    720
    Location:
    127.0.0.1
    Firefox -> HTTP Live Headers -> Go to page, scroll down, let comments load, post headers here.
     
    • Thanks Thanks x 1
  7. Mosquera

    Mosquera Newbie

    Joined:
    Feb 24, 2009
    Messages:
    32
    Likes Received:
    18
    Then comments are probably updated on a http post or with AJAX, making it visible only with a connection tamperer (it's prolly updated in the same page with a waiting gif such as youtube, right?)

    I'm afraid I can't help much more as I am not familiar with linkedin and I can't see the way it's done. :(
     
    • Thanks Thanks x 1
  8. bizboy949

    bizboy949 Newbie

    Joined:
    Jun 27, 2011
    Messages:
    29
    Likes Received:
    0
    Very interested in this on how it can be done as well.
     
  9. TheMatrix

    TheMatrix BANNED BANNED

    Joined:
    Dec 20, 2008
    Messages:
    3,444
    Likes Received:
    7,279
    Build a macro to work with SB.

    Just saying, I have no idea how that will work, though.
     
    • Thanks Thanks x 1
  10. mindmaster

    mindmaster Jr. VIP Jr. VIP Premium Member

    Joined:
    Sep 16, 2010
    Messages:
    2,501
    Likes Received:
    1,135
    Location:
    at my new office
    Ok, got it.

    Gyy it was so simple.

    Thanks and reps for those giving me ideas.

    Cheers
     
    Last edited: Aug 24, 2011