Twitter automation test (300 accounts, 6 control groups)

[dohyung97022]

Have you ever got change password?

Not yet, the test hasn't even started.
But in my pervious test, I never got that.
Only ran for a week tho.

https://www.blackhatworld.com/seo/twitter-automization-test-4-factors-32-accounts.1461652/

 

[dohyung97022]

Why use odroid n2l and not a Raspberry?

Cheap with more computing power.

All odroid n2l are Connected to your home network ?

Yup. Proxy server, automation server installed in odroid is all connected in my home network.

 

[imccafey]

You must have worked in the industry,
CGNAT? I have never heard of that before.

I do know that some cheap sim providers have a smaller pool of ip addresses.
Ended up paying more money, XD

In some capacity, yes. So basically these IPs are in use by several hundred users at a time, most half decent platforms know not to ban these IPs because of one specific user infraction otherwise they run the risk of banning access of multiple users on that IP. It's quite different from buying multiple IPs specifically.

 

[DropVS]

Cheap with more computing power.



Yup. Proxy server, automation server installed in odroid is all connected in my home network.

Hum.. why you do not connect two or more modems to your odroid?
Why One modem per odroid ?

What modem do you use ?

 

[Metug]

Not yet, the test hasn't even started.
But in my pervious test, I never got that.
Only ran for a week tho.

https://www.blackhatworld.com/seo/twitter-automization-test-4-factors-32-accounts.1461652/

I think its about following too fast. Thanks.

 

[dohyung97022]

Hum.. why you do not connect two or more modems to your odroid?
Why One modem per odroid ?

What modem do you use ?

I think you misunderstood me.
Every server is connected to a single modem via wifi.

I don't know the model name, but my internet provider installed it.
Looks kinda expensive.

Or did you mean huawei lte usb modems for the proxy?

 

[DropVS]

I think you misunderstood me.
Every server is connected to a single modem via wifi.

I don't know the model name, but my internet provider installed it.
Looks kinda expensive.

Or did you mean huawei lte usb modems for the proxy?

yes, talking about lte usb modems.
I thought that you connect usb modems to your odroid so my question why you use two odroid ? With one you cant use two or more usb modems?

 

[dohyung97022]

yes, talking about lte usb modems.
I thought that you connect usb modems to your odroid so my question why you use two odroid ? With one you cant use two or more usb modems?

Actually, for the lte proxy server,
I use a raspbarry pi.
And I do have multiple lte usb modems connected to one raspbarry.

When using odroid, or any other computer, It seems that it cannot handle more than 1 usb modems.
Well it can connect to it, and ifconfig shows all two connections, but 3proxy cannot traffic it somehow.
Only the raspbarry worked.
Do not know why, but I have tested it for a long time, so I am sure about that.

The odroids are automation servers.

 

[DropVS]

Actually, for the lte proxy server,
I use a raspbarry pi.
And I do have multiple lte usb modems connected to one raspbarry.

When using odroid, or any other computer, It seems that it cannot handle more than 1 usb modems.
Well it can connect to it, and ifconfig shows all two connections, but 3proxy cannot traffic it somehow.
Only the raspbarry worked.
Do not know why, but I have tested it for a long time, so I am sure about that.

The odroids are automation servers.

Ah sorry. I didnt understood it . I realy thought that you use odroid as proxy server.

So you use raspberry (what version / gb ram? ) for proxy connect to your home internet and with usb lte for proxy.
Your odroid connects to your raspberry by wifi? to your proxy?

 

[dohyung97022]

6/20
Report

Sad to say, I had some issues today.
So still no test results, Delayed again... XD

Problems
1. Odroid freezing issue.
2. Memory allocation issue.
3. Fan problems.

1. Odroid freezing issue

So I installed my automation server on the odroid machines, and ran it.
After an hour and a half, It stopped working.
I was wondering what happened and checked datadog to see this error.

This error is caused by selenium chromedriver, so I do not know the cause of it.
It does not specify that the error is caused in my code, so I had a bump.
Anyways, when my code hits an error, it abandons the work, kills the thread, and tries a new one.

But shortly after It tried a new action, it stopped logging.

I could not ssh into my server, I could not vnc into it.
So I checked it with a hdmi cable.

The screen showed up fine.
But I could not see any mouse movement, the caps lock was not working.
The machine just froze.


2. Memory allocation

After the first freezing problem, I have changed some of my code to use less memory.
I had this error pop up.

The odroid n2l has 4 GB to work on, so I had to delete some features to make it more slim.

There were some threads that was running tasks like sending phone screens to my frontend.
There was also image gif logs that I was sending to aws S3.
I know that images consume a lot of memory. So it had to be deleted.
Removed those code today. I am planning to run tests again tomorrow.

I did pick up with some memory profiling tools that can be used in python.
Maybe I will try that too.

3. Fan problems

The odroids were quite hot, but the fans were not spinning at all.
It seems that the odroid fan limit was 65 degree, But the machine freezes before it hits the limit.
I have tried changing the odroid fan limit to 35 degrees.
But I think I have accidently broke the kernel.
The fan does not seem to spin at all.
I think I will reinstall the os tomorrow and try it again.

Links if you are interested.
https://forum.odroid.com/viewtopic.php?f=181&t=46747

Alternative Solution?

I am not sure if my code is the problem, or my hardware is.

But if your hardware cannot handle it,
rather than making your code better, just smash more computing power.
This is a common joke among programmers.
But sometimes it could be a solution. XD

Genius mentions it in video.

Hopefully I have a mini-s beelink machine,
which is a little bit better than the odroid n2l.
I installed my server in it, and is currently running automations.

It seems promising,
It has been working for 3 hours. and no error log yet.
The problem is, I do not have another beelink machine to run 2 concurrently.


What to do.
I will keep the beelink machine running all night.

If it fails with a memory allocation,
my code is definitely the problem.

If it succeeds, I can buy another machine,
but you would have to wait until it arrives.

Either way, I will try to solve the fan problem and the memory problem in my odroid,
and retry it tomorrow.
If that works out, I will just go as originally planned.

Another hard day.
But I aint no quitter.

See ya tomorrow.

 

[dohyung97022]

Ah sorry. I didnt understood it . I realy thought that you use odroid as proxy server.

So you use raspberry (what version / gb ram? ) for proxy connect to your home internet and with usb lte for proxy.
Your odroid connects to your raspberry by wifi? to your proxy?

I did miss communicate too. No need to be sorry. XD

huawei E353/E3131 (USB modem)

raspbarry pi 3 model b+ (1GB ram)(proxy server)(wifi to local network) -> open http proxy to port 3128 in local network.

odroid n2l (4GB ram)(wifi to local network) -> connect to http proxy to port 3128 in home network.

How I connected usb modem.
https://www.blackhatworld.com/seo/diy-how-to-create-your-own-4g-proxy.1234185/post-15914299

 

[DropVS]

I did miss communicate too. No need to be sorry. XD

huawei E353/E3131 (USB modem)

raspbarry pi 3 model b+ (1GB ram)(proxy server)(wifi to local network) -> open http proxy to port 3128 in local network.

odroid n2l (4GB ram)(wifi to local network) -> connect to http proxy to port 3128 in home network.

How I connected usb modem.
https://www.blackhatworld.com/seo/diy-how-to-create-your-own-4g-proxy.1234185/post-15914299

Im trying to build all the setup first before start.

So you have huawei E353/E3131 (USB modem). Do you know if E3131 is equals to E3131A? Where do you bought it ? ebay? Any price range ?
Odroid is for automation but 4gb its fine for 300 accounts automations?

About automation, do you run 24/7 the accounts? Or try to follow human behaviors like only use twitter from 9am to 9pm?

 

[dohyung97022]

Im trying to build all the setup first before start.

So you have huawei E353/E3131 (USB modem). Do you know if E3131 is equals to E3131A? Where do you bought it ? ebay? Any price range ?
Odroid is for automation but 4gb its fine for 300 accounts automations?

About automation, do you run 24/7 the accounts? Or try to follow human behaviors like only use twitter from 9am to 9pm?

I bought it from aliexpress.
I don't think bhw likes me putting selling links here.
So you would have to search for
HUAWEI E3272 s-153

For odroid, I am having some problems.
as you can see in the report above.

Problems
1. Odroid freezing issue.
2. Memory allocation issue.
3. Fan problems.

I did also state how the automation works in report 6/19 in this thread.

Good luck.

 

[DropVS]

I bought it from aliexpress.
I don't think bhw likes me putting selling links here.
So you would have to search for
HUAWEI E3272 s-153

For odroid, I am having some problems.
as you can see in the report above.

I did also state how the automation works in report 6/19 in this thread.

Good luck.

And about using a Ryzen 5800h CPU with 16gb ram to automation, its overskill?

 

[dohyung97022]

And about using a Ryzen 5800h CPU with 16gb ram to automation, its overskill?

Its just a tradeoff if you can scale faster or not.
If the machine is expensive, you would have to earn more money in order to expand.
No need to be too worried about it.

 

[DropVS]

Its just a tradeoff if you can scale faster or not.
If the machine is expensive, you would have to earn more money in order to expand.
No need to be too worried about it.

Interesting.. I need other machine for it. Because of what I'll invest some money and looking for a mini pc.

Do you know or do you tried if you can connect your usb modem to a vmware (or other ) to main machine?

I mean , my ideia is buy a machine with for example 16gb ram and install proxmox.

In proxmox I will install a linux shell for modem and proxy.
and other VM with windows 11 (what do you use?) for automation .

What do you think?

 

[dohyung97022]

Interesting.. I need other machine for it. Because of what I'll invest some money and looking for a mini pc.

Do you know or do you tried if you can connect your usb modem to a vmware (or other ) to main machine?

I mean , my ideia is buy a machine with for example 16gb ram and install proxmox.

In proxmox I will install a linux shell for modem and proxy.
and other VM with windows 11 (what do you use?) for automation .

What do you think?

Don't know about proxmox,

but docker seems to have a way to connect usb to its pods.
https://stackoverflow.com/questions...to-give-access-to-a-host-usb-or-serial-device
And there could be this other problem, where your proxy software does not work in VM environments.

This is the kind of stuff you or I will never know, until you try out

 

[dohyung97022]

6/21
Report

I have found out the why the odroids were freezing.
I was watching system metrics, while running my program.

I saw chromium browser pop up as the highest memory usage.
total of 3.22 / 3.62 GB of my ram was being used.
And 30% of 3.62 GB was being used by chrome.

I thought my code was the problem, but if this was true,
we should have seen "python3 main.py" at the top of the screen, not chrome.

I also measured with the module memory profiler to be perfectly sure it was not my code.
https://pypi.org/project/memory-profiler/
Thread memory usage graph

So if you see the graph above,
the black graph is the memory used in total.
Colored graphs are memory used in threads.
4 automation ran in this graph.

If there was a memory leak in my code,
the second and the third "inbetween automation" memory usage sould go higher than before.
It sould look somewhat like a stair.
If so, the second time it ran automation, it holds onto more memory than before.
But that was not the case.

How to get multi-thread memory usage graph in python.

mprof run --multiprocess --include-children main.py
mprof plot

---

The shitty thing about chrome is that it keeps increasing memory usage as time goes on.

Chromes memory usage keeps going up 8.4% -> 22.6% -> 30.3%.

I think this is because Chrome saves stuff like images in the browsers cache.
My automation code keeps on surfing and sees a lot of images, and it goes into memory.
But when testing in my macbook, the memory ramps up and comes down after some time.
So there must be a cache limit that the browser can store,
But my odroid cannot hit the limit, and exceeds the total memory.


What to do next
So what should we do...

I can

1. Optimize odroid os to use lesser memory.
The idle odroid ubuntu mate memory useage is 1.5 GB.
If I can lower that, it may hit the ram limit of chrome and get lower.

2. Use a better machine.

Option 1, I think I will break the os, while trying XD
So I will go with option 2.

Besides the beelink machine is running perfectly with no problem at all.
So I will order another machine.

May take some time to arrive, cuz it's coming from china tho.
See you next time.

 

[DropVS]

For your tests and automation you never use browser? only commands lines? like call to api? or you have a OS with GUI to run chrome wihout headless?

 

[dohyung97022]

For your tests and automation you never use browser? only commands lines? like call to api? or you have a OS with GUI to run chrome wihout headless?

Selenium with undetected chromedriver uses the chrome browser, not the api.

I do not run headless.
The previous test with 30 accs I have tho.