1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Scraping data before the new Trump Administration takes it down

Discussion in 'BlackHat Lounge' started by outscrape, Jan 20, 2017.

Thread Status:
Not open for further replies.
  1. outscrape

    outscrape Jr. VIP Jr. VIP

    Joined:
    Nov 23, 2016
    Messages:
    118
    Likes Received:
    76
    In case you're wondering the incredible value of web scraping - this really makes me happy (i mean, sad also).



    "The group was split in two. One half was setting web crawlers upon NOAA web pages that could be easily copied and sent to the Internet Archive. The other was working their way through the harder-to-crack data sets—the ones that fuel pages like the EPA’s incredibly detailed interactive map of greenhouse gas emissions, zoomable down to each high-emitting factory and power plant. “In that case, you have to find a back door,” said Michelle Murphy, a technoscience scholar at the University of Toronto."

    Sometimes web scraping is the only way to rescue data before it's gone forever:


    https://www.wired.com/2017/01/rogue-scientists-race-save-climate-data-trump/?mbid=social_cp_fb_tny
     
  2. Capo Dei Capi

    Capo Dei Capi BANNED BANNED

    Joined:
    Oct 23, 2014
    Messages:
    754
    Likes Received:
    1,732
    Its very important that they save it, if they don't companies and people will be willing to increase their costs. Every company secretly desires to increase their electricity costs from lighting alone upwards of 600%. Major companies like Walmart,UPS,Fedex etc. actually don't want 10 mpg big rigs they secretly want ones that only get 4 mpg its all a lie for PR.
     
  3. Jomasdf

    Jomasdf Jr. VIP Jr. VIP

    Joined:
    Jul 7, 2012
    Messages:
    421
    Likes Received:
    155
    Occupation:
    C# dev
    Location:
    Sweden
    Home Page:
    Um, so screenshot if you feel like it. I'm not a fan of conspiraty theories.
     
  4. JustUs

    JustUs Power Member

    Joined:
    May 6, 2012
    Messages:
    626
    Likes Received:
    582
    Yesterday I had a minor, but painful surgery that involved removing anal warts. This is the second time in my life I have had such a surgery. The first was over twenty years ago. About a week and a half before this surgery I had a cyst removed from my chest. The prop team removed the suture rather than my doctor because the nature of venereal wart removal is such that you are incapacitated for several days, to a week, and I would not have been able to make the appt. with my general physician. Today, the infection from the surgery spread throughout my body and has left me very sore, everywhere. These are the things that matter to me.

    I could really care less if a bunch of extremists are worried that data that is prohibited from being destroyed by law is removed from easy access on some website. First, that data has been replicated by any number of websites; second, the internet archive (Wayback machine) routinely crawls government websites; third, as mentioned, law prohibits the destruction of that data.

    I could care less about your extremism, your rioting because you failed to get a ribbon (win the election) this time, or anything else. I do, however, support removing climate science from the political arena until the information is less fluid.
     
  5. outscrape

    outscrape Jr. VIP Jr. VIP

    Joined:
    Nov 23, 2016
    Messages:
    118
    Likes Received:
    76

    I am sorry to hear about your surgery.

    Regardless of your politics, it's worth recognizing that the internet archive does not contain the entire internet - these people are literally working to make sure that it DOES have this information because it doesn't, not to duplicate it. Specifically - "This data catalog is a place to store data that is difficult or impossible to harvest through web crawlers."
     
  6. JustUs

    JustUs Power Member

    Joined:
    May 6, 2012
    Messages:
    626
    Likes Received:
    582
    My surgery is my problem and not yours or anybody elses. It was listed to provide proportion. The information that you are so concerned about is freely available information that is protected as a matter of law. To claim it will disappear is nothing more than extremist alarmism.

    Back around September I was speaking with one of my friends who is a conspiracy theorist/nut. He asserted that Trump would not win the election because the Obama administration would not allow it. He then asserted that Obama would suspend the Constitution and become a dictator. I told him, as I tell you: I am not interested in your unfounded conspiracy theories. What you are engaging in fits the definition of "Fake News" and "Leftist propaganda." Save it.
     
  7. outscrape

    outscrape Jr. VIP Jr. VIP

    Joined:
    Nov 23, 2016
    Messages:
    118
    Likes Received:
    76
    I'm not interested in having a political conversation, just sharing interesting scraping news :)
     
  8. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,027
    Likes Received:
    10,817
    Occupation:
    WHEREZ MA
    Location:
    BITCOINS AT?
    Home Page:
    Awww c'mon....

    Of all the scraping news in the world, you thought something about Trump and climate change was not gonna stir up political shit?

    Gimme a break.
     
    • Thanks Thanks x 2
  9. pasdoy

    pasdoy Power Member

    Joined:
    Jul 17, 2008
    Messages:
    764
    Likes Received:
    241
    >technoscience

    haaa, guess that's a new field
     
  10. outscrape

    outscrape Jr. VIP Jr. VIP

    Joined:
    Nov 23, 2016
    Messages:
    118
    Likes Received:
    76
    It had never occurred to me that someone would argue that it's absolutely impossible that controversial gov't data will ever be hard to access

    -__-
     
  11. redarrow

    redarrow Elite Member

    Joined:
    Apr 1, 2013
    Messages:
    4,283
    Likes Received:
    978
    This post so boaring just why,?
     
  12. bartosimpsonio

    bartosimpsonio Jr. VIP Jr. VIP Premium Member

    Joined:
    Mar 21, 2013
    Messages:
    12,027
    Likes Received:
    10,817
    Occupation:
    WHEREZ MA
    Location:
    BITCOINS AT?
    Home Page:

    Yeah. If you went ahead and edited the post to remove the political controversy you started there by total accident, that'd be grrrreat.
     
    • Thanks Thanks x 1
  13. outscrape

    outscrape Jr. VIP Jr. VIP

    Joined:
    Nov 23, 2016
    Messages:
    118
    Likes Received:
    76
    seems i can't, but if you'll tell me what part is controversial i'll report it to the mod to remove
     
  14. JustUs

    JustUs Power Member

    Joined:
    May 6, 2012
    Messages:
    626
    Likes Received:
    582
  15. outscrape

    outscrape Jr. VIP Jr. VIP

    Joined:
    Nov 23, 2016
    Messages:
    118
    Likes Received:
    76
    It's a lot easier to scrape data than ask the gub't for it nicely and hope they get back to ya

    This is BHW after all
     
  16. t0mmy

    t0mmy Jr. Executive VIP Jr. VIP

    Joined:
    Jun 5, 2011
    Messages:
    6,565
    Likes Received:
    13,894
    Gender:
    Male
    Location:
    Spain
    Home Page:
    ... and that's all folks!
     
    • Thanks Thanks x 2
Thread Status:
Not open for further replies.