Basic Scraping Guide - How to make money with scraped content

outscrape said:

I'm actually working on building a scraping tool for universal use....but more about that later....

Click to expand...

"Universal" I'd imagine

scraping in spaaaaace...

Hey all,
I made a new thread to answer common scraping problems that people get a lot. Here you go.


Jon

Really nice share! I have been scrapping for a while, just for client's projects, but now I think I will try to use in another way just as a personal project, I liked the idea of a search engine, maybe it generates something, maybe not, but as it is a personal project, I am sure it will be fun!

I have been tinkering with scrapping, and my initial adventures have been with Scrapy [https://scrapy.org/]. This software is free, and fairly easy to use ... if you have some programming experience. If not, it could be daunting, but well worthwhile. It is highly customizable, which can translate to an almost flat learning curve for some, but powerful.

A project I am currently helping with is calling itself "The Poor Man's Facial Recognition and Profiling" solution. It is basically scraping the web like a beast, then uploading the images to Google Photos and utilizing their AI to bundle faces. It has so far been partially successful. One of the goals of our effort is to be able to add an image of someone we do not know and find a bunch of images that have that person in them. If we know who the person is, we name the bundle, which could be more useful for some. It is also a test of Google's definition of the word, "unlimited."

Very nice little guide. Thanks for sharing.

Mark Eternally said:

A project I am currently helping with is calling itself "The Poor Man's Facial Recognition and Profiling" solution. It is basically scraping the web like a beast, then uploading the images to Google Photos and utilizing their AI to bundle faces. It has so far been partially successful. One of the goals of our effort is to be able to add an image of someone we do not know and find a bunch of images that have that person in them. If we know who the person is, we name the bundle, which could be more useful for some. It is also a test of Google's definition of the word, "unlimited."

Click to expand...

What's your end goal for this? Do you have a plan to sell it or sell access or something?

outscrape said:

What's your end goal for this? Do you have a plan to sell it or sell access or something?

Click to expand...

@outscrape our plan is to better understand scraping so we can utilize it in other projects. We primarily focus on security related projects, but that takes us into so many other non-security arenas we have to constantly explore outside our primary focus. For instance, we are very interested in wireless technologies, in particular with regard to security, and to better understand how to address some questions posed we had to first understand real-world usage. This led to some very interesting adventures into human psychology and social human to human interactions as well as human to technology interactions and even technology to human interactions. Honestly, we jumped into the deep end on that one.

No, we do not plan to sell it or sell access to the information. We posted an article on our blog about the project, but did not delve into details. (That site uses google, bing and maybe facebook stat trackers, but there are no hidden affiliate or other such revenue generating code on that blog.)

Based on your posts it seems you do scraping as a profession, is this true? If so, do you do any type of research projects related to scraping? Anything you could share?

Regarding the "Lead Generation" section of the initial post, does anyone know how this is going to be affected by the incoming GDPR law, I assume that scraping personal information such as emails/names and using such information would become illegal, as there is no consent even though it is in the public domain (not so public with LinkedIn where public means "our closed system")? For example scraping contacts details and company information for a business directory or selling it on.

In there UK there are few companies which scrape companies house for instance and list the company directors and what not - would this now be illegal?

Quite honestly, I have read how the GDPR is meant to indirectly be a global "law," however, enforcement will be almost impossible, in particular for entities outside the EU. I am sure some countries will try to play nice and help enforce this, but not all will be so kind.

I am not aware of any such rule for any other continent, but I have not looked for them.

I know this is an older post, but do you offer a service in regards to the information you provided? Thank you.