Yes, people have built systems like this, but it’s important to be clear about what is actually feasible, what is extremely hard, and where the real limits are.
I’ll break it down at a high level, without getting into anything sketchy or brittle.
First, the big picture.
What you are describing is not really “an LLM controlling phones.” It is a distributed device automation system where phones expose a control surface, a central server coordinates tasks, and an LLM is used for reasoning and decision making on top.
That distinction matters because LLMs are bad at low level control and very good at high level intent.
Now, feasibility by platform.
On Android, this is realistically achievable. Android allows much deeper automation through accessibility services, instrumentation, and device management tools. Many internal QA teams already run large Android device farms where apps are installed, screens are read, and inputs are simulated remotely. Adding an LLM on top to decide what action to take next is doable, though still nontrivial.
On iOS, this becomes the hard wall. Apple heavily restricts automation and remote control. Outside of Apple’s own testing tools and tightly controlled enterprise setups, you cannot fully control arbitrary apps or system behavior. You can do limited UI testing on simulators or supervised devices, but a true iOS phone farm with full interaction is not realistically possible at scale without running into platform restrictions.
Cross platform at the same depth is therefore not symmetric. Any serious system ends up being Android first, iOS second, with different capabilities.
Architecture wise, the systems that work usually look like this.
Each phone runs a lightweight local agent. This agent is not smart. It can capture screen state, execute basic input actions, report device status, and enforce safety rules. Think of it as a remote hand, not a brain.
A central orchestration server manages devices. It assigns tasks, routes commands, handles failures, and keeps devices healthy. This is where scaling and reliability live.
The LLM sits above that. It never clicks buttons directly. It receives structured state, decides intent, and outputs high level actions like “open app,” “navigate to screen,” or “retry step.” Those are translated by deterministic code into device actions.
If you try to let the LLM directly control taps, timing, and gestures, the system will be unstable and expensive very quickly.
Now the honest difficulty assessment.
Technically possible on Android. Yes.
Technically clean. No.
Operationally expensive. Very.
iOS parity. Mostly no.
Most teams that succeed in this space are not building a general AI phone controller. They are automating a very narrow workflow, with guardrails everywhere, and using the LLM only where flexibility is required.
If your goal is research, QA automation, or internal tooling, this can make sense. If the goal is large scale generic interaction across many consumer apps, the platform owners have designed the systems specifically to resist that.
One final practical note.
Before building a phone farm, many teams discover that emulators, simulators, or direct API integrations solve eighty percent of the problem with twenty percent of the effort. Physical devices are usually the last resort.
If you want, you can tell me what kind of interaction you actually need the phones to perform at a high level, and I can help you sanity check whether this architecture is justified or if there is a simpler path.
