A $7 ESP32 lets an AI drive a real iPhone: it can see the screen and tap it, no jailbreak, no developer account, no Mac

What this project does

esp32-iphone-agent does something that sounds like magic: it lets an AI model genuinely "use" a real iPhone. It uses a $7 ESP32-S3 development board as the "hands" and AirPlay screen mirroring as the "eyes" — the AI sees the phone's live screen through AirPlay, and the ESP32 plugged into the phone's USB port is recognized as a mouse and keyboard so it can tap, swipe, and type. The brain can be any OpenAI-compatible vision model, whether a cloud API or a model you run locally. The crucial part is that none of this needs a Mac, an Apple developer account, or a jailbreak. You just plug the board into the iPhone, give the model an instruction like "open Clock and set a 7 AM daily alarm", and it will find the app, turn the time wheel, tick the repeat days, and save — every step's screenshot and tap location is recorded so you can review it afterward.

Why it is trending right now

The project only went open source on October 11, 2026, and reached nearly 570 stars in two days, the classic "the demo is so wild it goes viral" type. I think the core reason is one sentence: it pulled "AI controlling a real phone", previously a lofty endeavor, down to a cost an ordinary person can reproduce in a single weekend — a $7 board plus ready-made code. Phone automation used to mean either the vendor's own UI testing framework (needs a Mac and a developer account) or private APIs after jailbreaking, both with high barriers and compliance risk. This project uses AirPlay mirroring plus USB HID, a "wild" path that bypasses every barrier, and it works stably; in the demo video, drawing a smiley face and setting an alarm flow seamlessly, so it spread virally.

Technical highlights

Its design is clever and splits into three layers. The first layer is the "eyes": the ESP32 itself does not see; instead the phone mirrors its screen via AirPlay, and the project's receiver displays the live picture in a web UI so the model can understand the current state. The second layer is the "hands": an ESP32-S3 plugged into the phone's USB port is flashed with firmware that emulates a standard HID mouse and keyboard, so it performs real physical taps, swipes, and text input — not some accessibility simulation, but hardware input the system recognizes at a low level. The third layer is the "brain": any OpenAI-compatible vision model. The project sends the current screenshot plus the task description to the model, the model returns "what to do next", and the agent loops: execute, screenshot, decide again, until the task is done. It also keeps a "plan wall" and "step wall" where each step's screenshot overlays the tap or stroke trajectory, making debugging obvious at a glance. The whole thing is written in Python with a simple web interface, so the barrier is relatively low.

Who it is for

Two groups should play with this most. One is developers in the AI Agent and "computer-use" space who want to study how to let models drive real mobile devices without being locked into Apple's ecosystem. The other is hardware and Raspberry Pi tinkerers, for whom a $7 cost plus clear firmware and scripts makes a perfect weekend project. Automation test engineers can also use it as a low-cost real-device interaction check. If you are a pure software developer with no hardware and no interest in buying a board, it is for now just an eye-opening demo; but the moment you want an agent that can touch a real phone, this is about the cheapest starting point there is.

Quick try

You need an ESP32-S3 board (such as the Xiao ESP32S3) and an iPhone, then:

git clone https://github.com/onehumanbeing/esp32-iphone-agent.git
cd esp32-iphone-agent
pip install -r requirements.txt

Flash the firmware to the board per the README, mirror the phone to the receiver via AirPlay, and open the web UI to start giving the model instructions.

How it compares to alternatives

In phone automation, the traditional routes are basically two. Apple's own XCUITest / UI Automation is powerful but requires a Mac and a paid developer account, and can only test first-party apps. Android's ADB route is more open but unavailable to iPhone users. Private APIs after jailbreaking go deeper but void your warranty and break on every system upgrade. esp32-iphone-agent plays a completely different game: it never touches the system internals, instead taking the universal path of "watch the screen plus emulate hardware input", so it works across iOS versions and across apps without any privilege. The trade-off is that it depends on AirPlay mirroring clarity and the model's vision ability, and may occasionally mis-tap on complex UI — but that is exactly the space where the agent loop can self-correct. In short, it is the minimal viable solution for "turning a real phone into an interface any vision model can drive".