Gscraper Taking Years To Scrape Targets!!!

[JustUs]

What other options do I have?

You could always reverse engineer the program yourself.

If you are speaking of scrapers, while I have not reverse engineered ScrapeBox, I have never been given any reason to believe that this occurs in ScrapeBox from unexplained and excessive encrypted traffic.

 

[JustUs]

Interesting theory of yours, about GScraper. That code doesn't confirm it.

Your problem, not mine. I have given one way of verifying it, and another beat me to it - using a packet sniffer to see where the traffic is going.

You might also want to ask why registry keys that allow dotnet tracing are deleted when the program starts.

You see, I have nothing to prove to you.

 

[JustUs]

I am not a coder but if you are talking about DESCryptoServiceProvider then I suggest you look it up.

If not could you please explain how you drew your conclusion. I am interested.

I reverse compiled the program when I noticed excessive unexplained encrypted traffic to this:

I have known about it for over a year. I Have suggested that it be stopped. It has not.

If you keep in mind what the program does, it is a fairly good program.

 

[hatemachine]

You would be mistaken in some of your assumptions. That particular function is used in a couple of different ways depending on where the function is called from. Called from one place the method encrypts the URL, called from another it encrypts the proxy, called from a third it seeks permission to run. There are two separate keys that are used, and I could show them as well. Also, the program is tied to the computer through a few different hardware aspects.

BUT, and a big one, is that I did not put that piece of code up to debate what it means or how it is used. I put it up to demonstrate that I have, in fact, reverse compiled GScraper and just might have a good idea of what I am speaking about. You can cross test what I am speaking about with Wireshark and see the amount of traffic going to China. By putting that small piece of code up that does not reveal anything critical about the program, I have allowed others to some means to determine if I am speaking about what I know.

The code is not for sale or view unless one can demonstrate a need. A want is not good enough, but it is enough to put Hawke on warning. I have privately taken this to his attention before.

So please everyone, save the nit picking and fire up Wireshark to see where data is going when it is encrypted (208.98.62.34) and then ask yourself why the data would be encrypted?

No nit picking going on. Just posting that function isn't enough proof. And RE'ing a .NET app is nothing to be proud of, it's a cake. You do not need to be a wizard to run unpacking tools or a decompiler. I'm not trying to attack you. In fact I actually believe you, but you'll have to post actual proof. If you have the keys AND the encrypted data, why don't you just decrypt it and post what's being sent to their server? Ten minutes of work to solve the mystery of Gscraper.

 

[hatemachine]

Your problem, not mine. I have given one way of verifying it, and another beat me to it - using a packet sniffer to see where the traffic is going.

You might also want to ask why registry keys that allow dotnet tracing are deleted when the program starts.

You see, I have nothing to prove to you.

Disabling JIT by removing the registery keys is nothing bad, many do it.

 

[JustUs]

No nit picking going on. Just posting that function isn't enough proof. And RE'ing a .NET app is nothing to be proud of, it's a cake. You do not need to be a wizard to run unpacking tools or a decompiler. I'm not trying to attack you. In fact I actually believe you, but you'll have to post actual proof. If you have the keys AND the encrypted data, why don't you just decrypt it and post what's being sent to their server? Ten minutes of work to solve the mystery of Gscraper.

Because I am trying to get the problem resolved. The author will know his own work.

I have already told you what the data is.

BTW, it is far more than a few minutes work.

 

[Phunky]

Well this thread got incredibly interesting...quickly. Thanks for the heads up JustUs

 

[JustUs]

Disabling JIT by removing the registery keys is nothing bad, many do it.

Til it creates problems with other programs that are running and being traced, fortunately VS rebuilds those keys. Nor does deleting the registry keys disable JIT. MSIL is based on JIT. Nor does RedGate properly decompile the program.

BTW: The notes are mine as I was working through the program.

 

[hatemachine]

Til it creates problems with other programs that are running and being traced, fortunately VS rebuilds those keys. Nor does deleting the registry keys disable JIT. MSIL is based on JIT. Nor does RedGate properly decompile the program.

BTW: The notes are mine as I was working through the program.

I am not sure why you are talking about MSIL. I was saying that those keys are removed to get rid of the Just In Time debugger which creates annoying windows and can screw with certain apps. Here's an example:

http://social.msdn.microsoft.com/Fo...ting-rid-of-justintime-debugger?forum=vsdebug

See the registry values? They're the same. Some have issues with them, and by removing those values you end up having less problems for your customers.

I am aware that RG can't properly decompile it, as it is a huge project and you'd have to change alot of the UI stuff to make it work. Can you actually compile it? I assume based on the bp's you set, that you can.

Because I am trying to get the problem resolved. The author will know his own work.

I have already told you what the data is.

BTW, it is far more than a few minutes work.

I don't see why it would take more than a few minutes? We know what encryption method is used ( DES ) and all you need is the initialization vector and the key which was used to encrypt the data, also you have the encrypted data shown in Wireshark. Now all you have to do is write a function that decrypts the data using the initialization vector and the key that was passed to the function that encrypts the data. I wouldn't suggest you to use Wireshark though. Fiddler is prefered when dealing with the HTTP protocol as it is much easier to use.

 

[PrinceVisi]

I am waiting for a Final Answer to these accusations as they seem very serious.

 

[JustUs]

I am not sure why you are talking about MSIL. I was saying that those keys are removed to get rid of the Just In Time debugger which creates annoying windows and can screw with certain apps. Here's an example:

http://social.msdn.microsoft.com/Fo...ting-rid-of-justintime-debugger?forum=vsdebug

See the registry values? They're the same. Some have issues with them, and by removing those values you end up having less problems for your customers.

I am aware that RG can't properly decompile it, as it is a huge project and you'd have to change alot of the UI stuff to make it work. Can you actually compile it? I assume based on the bp's you set, that you can.



I don't see why it would take more than a few minutes? We know what encryption method is used ( DES ) and all you need is the initialization vector and the key which was used to encrypt the data, also you have the encrypted data shown in Wireshark. Now all you have to do is write a function that decrypts the data using the initialization vector and the key that was passed to the function that encrypts the data. I wouldn't suggest you to use Wireshark though. Fiddler is prefered when dealing with the HTTP protocol as it is much easier to use.

1. See image below.
2. The program is more twisted than Chinese arithmetic. Somewhere between 60 and 80 percent of the program is devoted to encryption and all of it multi level redirection/polymorphism and nested functions.

Oh, Forgot:
msil - microsoft intermediate language.

Msil is what dotnet programs compile to.

 

[hatemachine]

1. See image below.
2. The program is more twisted than Chinese arithmetic. Somewhere between 60 and 80 percent of the program is devoted to encryption and all of it multi level redirection/polymorphism and nested functions.

Oh, Forgot:
msil - microsoft intermediate language.

Msil is what dotnet programs compile to.

Obviously the code is not going to be fully recovered, no one likes getting their work stolen. They used an obfuscator to scramble their code which is totally understandable from a developers point of view. You should try to reverse mine, you'd poke your eyes out; but that doesn't mean that I am hiding something. I know what MSIL is, but it has nothing to do with the debugger breaking certain apps. Anyway, to the main point; it is indeed strange that they would encrypt the URL and the Proxy. Care to post that code too? Certainly interested to hear the sellers response on this. Good find.

 

[JustUs]

Obviously the code is not going to be fully recovered, no one likes getting their work stolen. They used an obfuscator to scramble their code which is totally understandable from a developers point of view. You should try to reverse mine, you'd poke your eyes out; but that doesn't mean that I am hiding something. I know what MSIL is, but it has nothing to do with the debugger breaking certain apps. Anyway, to the main point; it is indeed strange that they would encrypt the URL and the Proxy. Care to post that code too? Certainly interested to hear the sellers response on this. Good find.

1. The man used the commercial edition of DotFusicator. Besides VS removing the symbols during a release build the commercial version of DotFusicator removes the few that VS leaves and then obfuscates what is left and adds dummy classes to further confuse the issue. It was not something that reverse engineered overnight! It was something that I was concerned enough that I spent roughly 30 days reverse engineering and about that long walking through it. Your point about the UI is on the money.

2. There is nothing that cannot be decompiled. It takes a lot of work, but if someone is sufficiently motivated it will be decompiled. We are not talking about cracking here.

3. I have my own homegrown scraper based on dotNet. If I attempt to run it when GScraper is running, it will throw an exception. I also have several other helper programs based on dotNet that fail if GScraper is running. The nature of a dotNet program is that .net uses hungry algorithms. At the same time, I have no problem running programs that use .net when run alongside a program that does not use .net.

4. While I could over the course of a couple of dozen posts show the code that encrypts the URL and proxys, I do think that it would be revealing more of Hawkes code than should be. So the answer is no. Let us see if Hawke fixes the problem.

5. I understand why people obfuscate code, esp. .net. That does not bother me. The other stuff I have seen does.

6. Notice the message on the bottom of the image under the error list, "Build Succeeded." The code, as you put it, is fully recovered.

 

[hatemachine]

1. The man used the commercial edition of DotFusicator. Besides VS removing the symbols during a release build the commercial version of DotFusicator removes the few that VS leaves and then obfuscates what is left and adds dummy classes to further confuse the issue. It was not something that reverse engineered overnight! It was something that I was concerned enough that I spent roughly 30 days reverse engineering and about that long walking through it. Your point about the UI is on the money.

2. There is nothing that cannot be decompiled. It takes a lot of work, but if someone is sufficiently motivated it will be decompiled. We are not talking about cracking here.

3. I have my own homegrown scraper based on dotNet. If I attempt to run it when GScraper is running, it will throw an exception. I also have several other helper programs based on dotNet that fail if GScraper is running. The nature of a dotNet program is that .net uses hungry algorithms. At the same time, I have no problem running programs that use .net when run alongside a program that does not use .net.

4. While I could over the course of a couple of dozen posts show the code that encrypts the URL and proxys, I do think that it would be revealing more of Hawkes code than should be. So the answer is no. Let us see if Hawke fixes the problem.

5. I understand why people obfuscate code, esp. .net. That does not bother me. The other stuff I have seen does.

6. Notice the message on the bottom of the image under the error list, "Build Succeeded." The code, as you put it, is fully recovered.

Awesome. Now we'll just have to wait for Hawkes!

 

[JustUs]

Awesome. Now we'll just have to wait for Hawkes!

Thank you for understanding. I think you would enjoy having the same courtesy extended to you if it were your program!

 

[Wilson Grant Fisk]

Thank you for understanding. I think you would enjoy having the same courtesy extended to you if it were your program!

Good work, it will be interesting to see how the developer responds.

 

[umerjutt00]

Wow this have been turned into an amazing thread.let's see what happens

 

[JasonS]

I've already send PM to Hawkes three days ago regarding the scraping urls issue. But till yet there is not any response from him.
I wonder if he would care to response here as well.

 

[Snckr]

Well, what the actual fuck.

I've scraped well over 100,000,000 URLs with Gscraper this month alone (I have their quarterly proxy service and have used them for a long time) and I've only had good things to say, but it has been going down from 250k links scraped/min to only around 100k and I've been seeing lagging issues as well (nothing changed on the dedi, other things like GSA, SB, CB, CS, etc work fine).

Let's see how this unwinds.

 

[Snckr]

Well. The function encrypts string_0 and string_1. That doesn't say much without tracing the code back and checking what the result returned from smethod_3 is being used for. I don't see it being sent anywhere in that function, from where is smethod_3 being called? If someone has a valid license I could take a look and intercept the data. It would take like 10 minutes to verify the claims anyway.

I have a license and the proxy service, shoot me a PM for my dedi information.