Best scraper for finding illegal image violations?

Hi guys, I'm developing a software to detect illegal image violations.

I mean, if someone posts a photo and someone steals it, the software will detect the violation.

I'd like more power and need a web scraper capable of scanning at least 100,000 websites.

Any suggestions?

100k websites is not even a drop in the internet.

I don't even think google could do this

riccc said:

Hi guys, I'm developing a software to detect illegal image violations.

I mean, if someone posts a photo and someone steals it, the software will detect the violation.

I'd like more power and need a web scraper capable of scanning at least 100,000 websites.

Any suggestions?

Click to expand...

Ya

riccc said:

Hi guys, I'm developing a software to detect illegal image violations.

I mean, if someone posts a photo and someone steals it, the software will detect the violation.

I'd like more power and need a web scraper capable of scanning at least 100,000 websites.

Any suggestions?

Click to expand...

yandex and Google lens?

Why would you want to scrape again what has already been scraped a million times?

Use commoncrawl archives or AI engines.

I think using commoncrawl archives or AI engines like Google Lens is a great idea, as it would save you from scraping the same data multiple times. However, if you still want to build your own web scraper, you might want to consider using a distributed scraping approach to handle the large volume of websites. Have you thought about using a cloud-based service like AWS or Google Cloud to scale your scraping efforts? Additionally, what kind of image recognition algorithm are you planning to use to detect the violations, and how will you handle false positives or varying image formats?

riccc said:

Hi guys, I'm developing a software to detect illegal image violations.

I mean, if someone posts a photo and someone steals it, the software will detect the violation.

I'd like more power and need a web scraper capable of scanning at least 100,000 websites.

Any suggestions?

Click to expand...

There are so many issues with this, I don't knownwhere to start.

First you'd wanna be the owners of the images , like getty is doing it, else you're open to false dmca claim/abuse yourself etc.

The scraper is the least problem here.

theusualkeysersoze said:

There are so many issues with this, I don't knownwhere to start.

First you'd wanna be the owners of the images , like getty is doing it, else you're open to false dmca claim/abuse yourself etc.

The scraper is the least problem here.

Click to expand...

Don’t lawyers do this for their client too so the lawyer isn’t the owner but they are working in behalf of the ownrr

JohnnyOmm said:

Don’t lawyers do this for their client too so the lawyer isn’t the owner but they are working in behalf of the ownrr

Click to expand...

Yeah, if you have power of attorney, it's a different story.

I'd like more power and need a web scraper capable of scanning at least 100,000 websites.

Click to expand...

Thats silly idea.
Scraper capable of doing what you actually need would require milions USD/month and team of skilled developers.

Here is solution you can implement on your own and that wont break the bank:

Reverse image search - google, bing, yandex.
Check every image you want to monitor every X days using all above. Than check urls you get from reverse image search.
If google/bing didnt find image you are tracking - trust me, you also wont find it with your private scraper.

riccc said:

Hi guys, I'm developing a software to detect illegal image violations.

I mean, if someone posts a photo and someone steals it, the software will detect the violation.

I'd like more power and need a web scraper capable of scanning at least 100,000 websites.

Any suggestions?

Click to expand...

Google Reverse Image Search
Free, uses uploaded images to find the source and where they appear — useful for checking copyright or where images have been reused.

building a crawler for 100k domains is actually the easy part .. matching the images at that scale is the real technical debt tbh . if u rely on md5 or simple hashes u’ll miss 90% of the violations cuz they just resize or compress the file slightly and the hash changes
i built a similar detection engine for a client in the adult tube niche back in 22 and the only way to scale without burning a hole in the server budget was using pHash (perceptual hashing) . it converts the image into a bitwise fingerprint that stays the same even if the photo is cropped or filtered . once u have those fingerprints u can do a massive lookup in a vector db like milvus or pinecone in milliseconds
regarding the infra .. don't crawl the entire internet from scratch . u’ll spend more on proxies than u’ll ever make in settlements . use the common crawl index (index.commoncrawl.org) to extract image tags and source urls from their monthly snapshots . u can pull a list of millions of potential stolen image urls from their public dataset and then run a threaded worker farm to hit just those specific direct links . it saves u from loading the whole page bloat on 100k sites
another thing to watch for is the legal trolling aspect .. if ur system just blasts automated dmca notices based on a bot match u will eventually hit a bad faith wall in a u.s. court . we always implemented a human-in-the-loop verification step for any match with a similarity score over 85% before the automated mailing sequence kicks in .
if u want to build this in python offload the heavy image processing to a rust or c++ extension .. python's gil will choke the moment u try to process 10k images per minute on a single node .