Scraping data before the new Trump Administration takes it down

In case you're wondering the incredible value of web scraping - this really makes me happy (i mean, sad also).



"The group was split in two. One half was setting web crawlers upon NOAA web pages that could be easily copied and sent to the Internet Archive. The other was working their way through the harder-to-crack data sets—the ones that fuel pages like the EPA’s incredibly detailed interactive map of greenhouse gas emissions, zoomable down to each high-emitting factory and power plant. “In that case, you have to find a back door,” said Michelle Murphy, a technoscience scholar at the University of Toronto."

Sometimes web scraping is the only way to rescue data before it's gone forever:


https://www.wired.com/2017/01/rogue-scientists-race-save-climate-data-trump/?mbid=social_cp_fb_tny

Its very important that they save it, if they don't companies and people will be willing to increase their costs. Every company secretly desires to increase their electricity costs from lighting alone upwards of 600%. Major companies like Walmart,UPS,Fedex etc. actually don't want 10 mpg big rigs they secretly want ones that only get 4 mpg its all a lie for PR.

Um, so screenshot if you feel like it. I'm not a fan of conspiraty theories.

Yesterday I had a minor, but painful surgery that involved removing anal warts. This is the second time in my life I have had such a surgery. The first was over twenty years ago. About a week and a half before this surgery I had a cyst removed from my chest. The prop team removed the suture rather than my doctor because the nature of venereal wart removal is such that you are incapacitated for several days, to a week, and I would not have been able to make the appt. with my general physician. Today, the infection from the surgery spread throughout my body and has left me very sore, everywhere. These are the things that matter to me.

I could really care less if a bunch of extremists are worried that data that is prohibited from being destroyed by law is removed from easy access on some website. First, that data has been replicated by any number of websites; second, the internet archive (Wayback machine) routinely crawls government websites; third, as mentioned, law prohibits the destruction of that data.

I could care less about your extremism, your rioting because you failed to get a ribbon (win the election) this time, or anything else. I do, however, support removing climate science from the political arena until the information is less fluid.

JustUs said:

I could really care less if a bunch of extremists are worried that data that is prohibited from being destroyed by law is removed from easy access on some website. First, that data has been replicated by any number of websites; second, the internet archive (Wayback machine) routinely crawls government websites; third, as mentioned, law prohibits the destruction of that data.

Click to expand...

I am sorry to hear about your surgery.

Regardless of your politics, it's worth recognizing that the internet archive does not contain the entire internet - these people are literally working to make sure that it DOES have this information because it doesn't, not to duplicate it. Specifically - "This data catalog is a place to store data that is difficult or impossible to harvest through web crawlers."

outscrape said:

I am sorry to hear about your surgery.

Regardless of your politics, it's worth recognizing that the internet archive does not contain the entire internet - these people are literally working to make sure that it DOES have this information because it doesn't, not to duplicate it. Specifically - "This data catalog is a place to store data that is difficult or impossible to harvest through web crawlers."

Click to expand...

My surgery is my problem and not yours or anybody elses. It was listed to provide proportion. The information that you are so concerned about is freely available information that is protected as a matter of law. To claim it will disappear is nothing more than extremist alarmism.

Back around September I was speaking with one of my friends who is a conspiracy theorist/nut. He asserted that Trump would not win the election because the Obama administration would not allow it. He then asserted that Obama would suspend the Constitution and become a dictator. I told him, as I tell you: I am not interested in your unfounded conspiracy theories. What you are engaging in fits the definition of "Fake News" and "Leftist propaganda." Save it.

I'm not interested in having a political conversation, just sharing interesting scraping news

outscrape said:

I'm not interested in having a political conversation, just sharing interesting scraping news

Click to expand...

Awww c'mon....

Of all the scraping news in the world, you thought something about Trump and climate change was not gonna stir up political shit?

Gimme a break.

>technoscience

haaa, guess that's a new field

bartosimpsonio said:

Awww c'mon....

Of all the scraping news in the world, you thought something about Trump and climate change was not gonna stir up political shit?

Gimme a break.

Click to expand...

It had never occurred to me that someone would argue that it's absolutely impossible that controversial gov't data will ever be hard to access

-__-

This post so boaring just why,?

outscrape said:

It had never occurred to me that someone would argue that it's absolutely impossible that controversial gov't data will ever be hard to access

-__-

Click to expand...

Yeah. If you went ahead and edited the post to remove the political controversy you started there by total accident, that'd be grrrreat.

bartosimpsonio said:

Yeah. If you went ahead and edited the post to remove the political controversy you started there by total accident, that'd be grrrreat.

Click to expand...

seems i can't, but if you'll tell me what part is controversial i'll report it to the mod to remove

outscrape said:

It had never occurred to me that someone would argue that it's absolutely impossible that controversial gov't data will ever be hard to access

-__-

Click to expand...

It had never occurred to me that someone would be so tied up in their own little world of conspiracy that they would not be able to understand relative law (actually not surprising):
https://www.foia.gov/
https://www.archives.gov/records-mgmt/faqs/federal.html
https://www.archives.gov/about/laws

It's a lot easier to scrape data than ask the gub't for it nicely and hope they get back to ya

This is BHW after all

... and that's all folks!