Questions about multi-threaded selenium bot

[plau889]

Hello,

I'm a C# developer, but I'm a complete rookie in the realm of bots...

I would like to develop a bot that would be able to:
1. Open a new chrome window, navigate to a site, authenticate with a user+pass combination, and save this session. It should be able to do this 25 times (for example), and this would end up with 25 saved sessions, each with its own set of cookies/cache, and no link between the instances.
2. I would like to "recall" the saved sessions, so if I navigate back to the site, it should be logged in with the user+pass that was used. Now the bot should be able to do certain things, like scraping some data, posting comments etc.
3. The bot should be able to use a proxy for each chrome instance.

Now I know that there are a gazillion bot programmers on this forum, who would gladly do something similar for me, but I want to build it myself, otherwise I wouldn't learn. I also want to know the code inside-out, so if it's needed, I can quickly patch it.

I also need this bot to be secure, meaning that the site that I would use it one, should not see these bot activities (anti-fingerprinting).

Do you think this could be achieved in C# + Selenium?
Are there any github projects similar to this, that would cover some of the points?

 

[adverts]

1) selenium session data can be extracted via API and stored.
2) chromedriver/geckodriver both support custom profiles where you can rebuild your extract session data
3) both drivers allow binding to proxies
Anti-fingerprinting goes well beyond just session and proxies. Behavior is where 99% of people get caught.
You'll probably need to sink about 200-300 developmental hours to get proficient.

 

[gimme4free]

Simply create a new instance of a driver with it's own profile:

ChromeOptions TmpOptions = new ChromeOptions();
TmpOptions.AddArgument(("user-data-dir=" 
                + (My.Application.Info.DirectoryPath + ("\\profiles\\" + "InstanceX"))));
ChromeDriverService TmpService = ChromeDriverService.CreateDefaultService();
ChromeDriver MyDriver = new ChromeDriver(TmpService, TmpOptions);

 

[net307]

Even with new profile if you use selenium you can be detected, selenium also leaves footprints

 

[Jibri Wright]

In order to use Selenium, you're gonna have to get rid of the webdriver flag it sets to true.

Before (in browser console window):

> navigator.webdriver
true
Change (in selenium):

// C#
var options = new ChromeOptions();
options.AddExcludedArguments(new List<string>() { "enable-automation" });

// Python
options.add_experimental_option("excludeSwitches", ['enable-automation'])
After (in browser console window):

> navigator.webdriver
undefined

The webdriver flag is set to true when you're automating through a webdriver and that's how sites can tell that you're automating. This will set that flag back to undefined

 

[dotaboy]

In order to use Selenium, you're gonna have to get rid of the webdriver flag it sets to true.

Before (in browser console window):

> navigator.webdriver
true
Change (in selenium):

// C#
var options = new ChromeOptions();
options.AddExcludedArguments(new List<string>() { "enable-automation" });

// Python
options.add_experimental_option("excludeSwitches", ['enable-automation'])
After (in browser console window):

> navigator.webdriver
undefined

The webdriver flag is set to true when you're automating through a webdriver and that's how sites can tell that you're automating. This will set that flag back to undefined

The information you provided here is priceless. I am using selenium for a while without doing that but luckily I haven't been caught. I will adapt this information to my system

 

[rootjazz]

In order to use Selenium, you're gonna have to get rid of the webdriver flag it sets to true.

Before (in browser console window):

> navigator.webdriver
true
Change (in selenium):

// C#
var options = new ChromeOptions();
options.AddExcludedArguments(new List<string>() { "enable-automation" });

// Python
options.add_experimental_option("excludeSwitches", ['enable-automation'])
After (in browser console window):

> navigator.webdriver
undefined

The webdriver flag is set to true when you're automating through a webdriver and that's how sites can tell that you're automating. This will set that flag back to undefined

Is ONE way of many they detect you.

Some other ways listed here:

https://intoli.com/blog/making-chrome-headless-undetectable/

 

[Jibri Wright]

Is ONE way of many they detect you.

Some other ways listed here:

https://intoli.com/blog/making-chrome-headless-undetectable/

Lol I'm hip. I've read that article a while ago. But the safest method is to not use headless and automate the browser. Not only that but you have to change the navigator.platform, navigator.maxTouchPoints, etc. The list goes on. I've actually done this where I created some self-injecting Javascript that modifies all of the getter requests. Completely undetectable.