what site is considered too hard to scrape ?

[huyvun]

Any thoughts on what sites are considered far to difficult to scrape..?

Could be because the site has very good security/anti scraping technology ?

Mix of html/Js, too obfuscated ?

Security such as rotating cookies,, detection of browser/curl header orders etc..

 

[tpickett]

Nothing is "too difficult" to scrape. Just have to know how to parse with regex. Even Googles SERPs can be scraped with PHP and cURL...

 

[huyvun]

well, fb for example have measures to analyze frequency of requests, (intervals too),
path of links crawled etc,, and very quickly they detect bot or human, unless scrape software is very complex
(i'm talking when you start scraping thousands of links,, then fb shuts you down)

linkedin looks pretty hard to scrape..
as well ask skype forums

 

[tpickett]

anything can be done with enough proxies from the right location...

 

[huyvun]

the number of proxies is irrelevant to the question. - you're suggesting that the only defense a site has
against scraping is IP detection...

anyways.....

 

[thejake]

The trickiest sites to scrape are the ones with strong authentication, browser-specific script behaviors and asynchronous elements. For example terapeak isn't much fun to scrape without browser automation.

 

[meannn]

anything can be done with enough proxies from the right location...

Nope, if content is made with javascript, you cannot scrape. Javascript is browser based.

 

[huyvun]

agreed... that's what i was thinking too
sites which pull content in via async js..
or going a step further..
js which downloads obfsucated js which re-assembles ( like self modifying code ), at load time,
which in turn pulls content .. this would be almost impossible to scrape, unless you designed a custom-specific scraper ...
even with that,, the site could make use of all sort of anti-debug tricks, like one-time cookie-pads.
code timing,, public key layers.. etc..
with that said.. - if a browser can display it,, then ofcourse there is a way around it.. -

 

[m00j99]

Any thoughts on what sites are considered far to difficult to scrape..?

if a browser can display it,, then ofcourse there is a way around it.. -

so... what exactly is your question now?

 

[huyvun]

want opinions, on what site is considered the hardest to crack..
for example,, if the question was - what is the hardest windows app to reverse engineer, without a doubt it would be skype ..
( the most hardcore anti-debugging techniques ever implemented into an app ).
so want to know, what you guys consider the equivelant of this would be for as a site

 

[-ReX-]

want opinions, on what site is considered the hardest to crack..
for example,, if the question was - what is the hardest windows app to reverse engineer, without a doubt it would be skype ..
( the most hardcore anti-debugging techniques ever implemented into an app ).
so want to know, what you guys consider the equivelant of this would be for as a site

Can't be that hard, some one reverse engineersed it and realised the code to grab the last ip of any user lol.

 

[-Jericho-]

NSA Servers. Bastards just don't want to let us in.