Gscraper Taking Years To Scrape Targets!!!

Hey guys.....

I'm doing some scraping for GSA and its taking AGES to scrape targets...

I'm using footprints from Footprint factory so I have about 30k footprints. I'm running at max 1500 threads with 30 sec timeout...

I'm removing duplicates at scraping so I think that may be a reason its taking some time.


Here are my server details...

PowerUp
2 CPU
60 GB
2048 MB
Unmetered Bandwith

Any thoughts on why its taking so long to complete scrape?

You don't have enough proxies.

WinningIO said:

You don't have enough proxies.

Click to expand...

Im using gscraper's proxy service... the unlimited ones...

keith88 said:

Im using gscraper's proxy service... the unlimited ones...

Click to expand...

they were good , now they are worse than public proxies

divok said:

they were good , now they are worse than public proxies

Click to expand...

So go private? How many?

Two days ago I used Gscraper and 1000 key words with 500 + public proxies at 20 threads to scrape 900,000 sites in about three years hours.

Here is the problem with GScraper, and it is a warning as well, GScraper sends every link and every proxy that you use to China. This means that when you are scraping, or using your own proxies, you are not just scraping for yourself, or keeping the proxies that you use to yourself. Ditto when you post.

If Hawke wants to contest this, I will be more than happy to post the reverse compiled source on line.

Because you are sending all your scraped links and proxies to China, you have a delayed response that slows down the scraping.

JustUs said:

Two days ago I used Gscraper and 1000 key words with 500 + public proxies at 20 threads to scrape 900,000 sites in about three years.

Here is the problem with GScraper, and it is a warning as well, GScraper sends every link and every proxy that you use to China. This means that when you are scraping, or using your own proxies, you are not just scraping for yourself, or keeping the proxies that you use to yourself. Ditto when you post.

If Hawke wants to contest this, I will be more than happy to post the reverse compiled source on line.

Because you are sending all your scraped links and proxies to China, you have a delayed response that slows down the scraping.

Click to expand...

That is a serious accusation. I wonder what the developers have to say about it.

JustUs said:

Two days ago I used Gscraper and 1000 key words with 500 + public proxies at 20 threads to scrape 900,000 sites in about three years.

Here is the problem with GScraper, and it is a warning as well, GScraper sends every link and every proxy that you use to China. This means that when you are scraping, or using your own proxies, you are not just scraping for yourself, or keeping the proxies that you use to yourself. Ditto when you post.

If Hawke wants to contest this, I will be more than happy to post the reverse compiled source on line.

Because you are sending all your scraped links and proxies to China, you have a delayed response that slows down the scraping.

Click to expand...

This is deserving of its own thread. I suggest you start one as this is a very serious accusation and if true, should be out in the open..

JustUs said:

Two days ago I used Gscraper and 1000 key words with 500 + public proxies at 20 threads to scrape 900,000 sites in about three years.

Here is the problem with GScraper, and it is a warning as well, GScraper sends every link and every proxy that you use to China. This means that when you are scraping, or using your own proxies, you are not just scraping for yourself, or keeping the proxies that you use to yourself. Ditto when you post.

If Hawke wants to contest this, I will be more than happy to post the reverse compiled source on line.

Because you are sending all your scraped links and proxies to China, you have a delayed response that slows down the scraping.

Click to expand...

if you are serious and not trolling , you MUST shitlist them .

divok said:

if you are serious and not trolling , you MUST shitlist them .

Click to expand...

Draw you own conclusion as to whether I am bullshiting or trolling.



And if you want to see it in the Visual Basic that it is written in, I can do that too. I converted to C# because I am a C family type of guy.

JustUs said:

Draw you own conclusion as to whether I am bullshiting or trolling.

And if you want to see it in the Visual Basic that it is written in, I can do that too. I converted to C# because I am a C family type of guy.

Click to expand...

What other options do I have?

What does that all mean? Can you explain a little more on what it is theyre doing and what the implications are? Not everyone here is a programmer.

mudbutt said:

What does that all mean? Can you explain a little more on what it is theyre doing and what the implications are? Not everyone here is a programmer.

Click to expand...

with my limited knowledge and if this is actually a method in gscraper . Then it is encrypting some data and returning it to some other variable or function .
http://msdn.microsoft.com/en-us/library/system.security.cryptography.cryptostream(v=vs.110).aspx

I may be completely wrong . Still it doesn't prove anything .

JustUs said:

Draw you own conclusion as to whether I am bullshiting or trolling.

And if you want to see it in the Visual Basic that it is written in, I can do that too. I converted to C# because I am a C family type of guy.

Click to expand...

I am not a coder but if you are talking about DESCryptoServiceProvider then I suggest you look it up.

If not could you please explain how you drew your conclusion. I am interested.

JustUs said:

Draw you own conclusion as to whether I am bullshiting or trolling.

And if you want to see it in the Visual Basic that it is written in, I can do that too. I converted to C# because I am a C family type of guy.

Click to expand...

Well. The function encrypts string_0 and string_1. That doesn't say much without tracing the code back and checking what the result returned from smethod_3 is being used for. I don't see it being sent anywhere in that function, from where is smethod_3 being called? If someone has a valid license I could take a look and intercept the data. It would take like 10 minutes to verify the claims anyway.

hatemachine said:

Well. The function encrypts string_0 and string_1. That doesn't say much without tracing the code back and checking what the result returned from smethod_3 is being used for. I don't see it being sent anywhere in that function, from where is smethod_3 being called? If someone has a valid license I could take a look and intercept the data. It would take like 10 minutes to verify the claims anyway.

Click to expand...

It would be interesting.

Maybe somebody could run it through a network sniffer/packet analyzer and see if it is sending anything that way.

I think it's insane that people use someone else's tool to scrape google when you can easily build a scraper yourself with a spare afternoon and some persistence.

To help you guys out, here:

- look up php curl
- open firefox and turn off javascript and cookies
- visit google and do a search
- grab the url
- write your php curl script to use private proxies and visit that url
- use simple html dom to grab the links

voila, you now have a scraper.

Interesting theory of yours, about GScraper. That code doesn't confirm it.

hatemachine said:

Well. The function encrypts string_0 and string_1. That doesn't say much without tracing the code back and checking what the result returned from smethod_3 is being used for. I don't see it being sent anywhere in that function, from where is smethod_3 being called? If someone has a valid license I could take a look and intercept the data. It would take like 10 minutes to verify the claims anyway.

Click to expand...

You would be mistaken in some of your assumptions. That particular function is used in a couple of different ways depending on where the function is called from. Called from one place the method encrypts the URL, called from another it encrypts the proxy, called from a third it seeks permission to run. There are two separate keys that are used, and I could show them as well. Also, the program is tied to the computer through a few different hardware aspects.

BUT, and a big one, is that I did not put that piece of code up to debate what it means or how it is used. I put it up to demonstrate that I have, in fact, reverse compiled GScraper and just might have a good idea of what I am speaking about. You can cross test what I am speaking about with Wireshark and see the amount of traffic going to China. By putting that small piece of code up that does not reveal anything critical about the program, I have allowed others to some means to determine if I am speaking about what I know.

The code is not for sale or view unless one can demonstrate a need. A want is not good enough, but it is enough to put Hawke on warning. I have privately taken this to his attention before.

So please everyone, save the nit picking and fire up Wireshark to see where data is going when it is encrypted (208.98.62.34) and then ask yourself why the data would be encrypted?

If I had built such a tool for sale I would be definitely do what is being accused in this thread. It is trivial to do and gets others to do work for you.