- “Watch 20 Reels about cooking and tell me what’s trending.”
- “List my 10 newest Instagram DMs.”
- “Summarize my last 10 saved posts.”
- “How many followers does @nasa have?”
- “Watch 10 TikToks and tell me what they’re about.”
Where Mobster goes by itself
The agent says where it wants to be (IG_OPEN), and code takes it there. The pack taps only the controls Instagram names in its tree, and opens a link only when it lands in the app.
Instagram’s web addresses for the DM inbox (
instagram.com/direct/inbox/) and for Reels (instagram.com/reels/) open the home feed, not those screens (opened on an iPhone 15 Pro, iOS 26.0.1, 14 October 2026). The pack never opens them. The app links instagram://direct-inbox and instagram://reels_home open the inbox and the Reels viewer on the same phone. open_url and mobster phone open open the app link when you give them one of those two web addresses, and say they did.
Each step shows in the task’s steps in plain words, such as “Opened your saved posts in Instagram” or “Read 10 DMs in Instagram”.
What Mobster reads
IG_READ reads what a screen shows, scrolling as it goes, and opens the screen first when it isn’t there.
What it read reaches the model as data from the phone, never as instructions, and stays in front of it for the rest of the task. You get it as a card in the result: a table of your DMs, a list of the Reels with their sounds. When the answer repeats those rows as a table, the result keeps the card and leaves the copy out.
Reading a DM thread opens it, so Instagram shows it as seen. The DMs list itself is read without opening any conversation, from the row’s own label: who, the last message and when. When a row leaves one of those out, its line names the field (
Last message: not on the screen), so your agent never has to guess at a missing one. Instagram shows an older conversation with no preview, only Sent Tuesday: its row gives Tuesday as the time and names the last message as not on the screen.
TikTok
TT_OPEN and TT_READ do the same for TikTok. TikTok’s pack is built from the words TikTok’s app gives its controls, and every screen it knows waits to be checked against screens recorded from the real app. Until then each step that can’t tell where it is goes on the general way, as below.
X, YouTube, Threads, LinkedIn and Reddit
Five more packs work the same way, each with its own two tools. Their screens are built from the words each app gives its controls, and wait to be checked against screens recorded from the real app, as TikTok’s do.Questions answered with no model
A request that is exactly one of these questions, or opens one of these screens, is answered by code before Mobster calls a model. It costs nothing and makes no model call. The answer is what the screen showed, row by row.
A bare “my last 10 DMs” is answered when the task started in Instagram, TikTok or X, or when that app is in front on the phone. With none of those it goes to the model, because Instagram, TikTok and X all have DMs.
The model takes the request instead when:
- it holds a word the table doesn’t list, such as a name, a place, a reason or a second question;
- it holds an act word: send, post, reply, buy, delete, like, follow, turn on or off, change, install and the rest of Mobster’s act list;
- it fits two apps or two questions;
- the task has an output format or schema, belongs to a workflow, a long task or an account, or pins apps;
- the pack isn’t sure which screen it is on, or the page doesn’t show the row you asked for. The phone stays where the code left it, and the model takes it from there.
MOBSTER_FRONT_DOOR=off in the env file turns this off (Configuration).
Likes, follows, comments and posts
A pack never likes, follows, comments, sends, saves or posts. When a task asks for one of those, the agent asks the pack for the act’s place (IG_OPEN like, follow @nasa, comment, reply Sam Lee, compose, and the same for TT_OPEN), and the pack stops at the control and says where it is. The agent’s own tap on it then asks you first, is journaled and is never sent twice, as any act is (Safety).
From your coding agent
Mobster’s MCP server has the same packs as tools,instagram_open, instagram_read, tiktok_open and tiktok_read, on any iPhone list_devices shows. On a real iPhone, the app must be one you let outside agents use, and every tap a pack makes follows your rules in Settings › Safety. Asked for {"what": "dms", "n": 3, "device": "<your iPhone>"}, instagram_read answers:
columns, screen and title (MCP server).
Messages, Settings, Calendar, Notes, Maps, Photos and Safari
The same goes for seven of the apps on every iPhone. Tasks like these get faster and cheaper:- “Which of my text conversations are unread?”
- “What version of iOS is my phone on?”
- “When and where is my design review?”
- “What’s the locker code in my gym note?”
- “What’s the address of Apple Park?”
- “How many photos are in my Dogs album?”
Each pack goes where it needs through the app’s own controls: Settings, Notes and Calendar through their search or their list, Maps through
maps://?q=, Safari through a Google link. Reading the Messages list opens no conversation, so nothing is marked read. thread <name> opens one, and Messages marks it read.
The packs only read and move around. Settings never flips a switch, Calendar never adds or deletes an event, Maps never starts directions. For a message, MS_OPEN reply Sam Lee opens Sam Lee’s conversation, checks that the name at the top is Sam Lee, and names the message field. If the conversation on screen is someone else’s, it names nothing. The agent types and taps Send itself, and Mobster asks you first (Safety).
Each of the seven was checked on iOS 26.0.1 on 11 and 12 October 2026, on screens recorded from a real iPhone, except a conversation’s own screen, which waits for a recording. Over iPhone Mirroring they read the same screens from the drawn lines, with two things the picture can’t show left out: Messages’ unread dot, so unread needs a USB connection, and a switch’s state in Settings.
Over iPhone Mirroring
Over iPhone Mirroring Mobster has no accessibility tree: it reads the words drawn on the screen and names the icons it sees. Each screen a pack knows has a second set of signs made only of those, so the pack names the same screens either way. On 59 Instagram screens recorded on a real iPhone on 11 and 12 October 2026, the pack named 52 from the tree and, read from the same screenshots the way Mirroring reads them, 50 of those 52, with none named wrong (measured 14 October 2026). The 2 it didn’t name are Explore’s search with only the typed words and the keyboard on screen. Over Mirroring a pack:- reads from the drawn lines: a Reel’s handle beside Follow, its caption, and the counts down the right side (rounded as drawn, “282K”); a list row’s name and the line under it;
- taps the icons Mirroring named (“Messages”, “Search”) to move between tabs;
- opens no links, so a profile by name goes through the app’s search, and when that doesn’t work the step goes on the general way.
When an app changes
Apps change their screens and test new ones on some people first. A pack is built to keep up:- Several variants per screen. A screen may have more than one set of signs (an A/B test, a redesign), and each set is words, roles and where things sit, never pixels.
- It hands over at once. When a pack can’t name the screen it meant to reach or read, that step says so and the general agent goes on from the same screen. It never acts on a guess, and after two such steps in a row the pack stays out of the rest of that task. A task never fails because a pack is out of date.
- It logs drift. The step’s
app_packevent says which screen it meant and carries the screen’s fingerprint (a hash of its structure, no text), and the hint ledger keeps the miss as drift with the app version and the screen where the agent took over, as it does for saved programs (GET /api/harness/hints, Scripting). A step that missed its last 2 uses on a version of the app isn’t tried again on that version. - It learns the new variant, carefully. From that screen the pack proposes a new variant: the app’s own words in its top part, the current tab and named buttons, never a name, a handle, a message or a number. When the task finishes, the proposal is confirmed. The next time the pack misses the same screen, the variant has to name that fresh screen while none of the pack’s other screens does; only then is it trusted, for that app version. A learned variant only names screens: it never finds the control for a like, a follow, a comment or a send. Learned variants are kept on your Mac in
hints-heals.jsonlbeside the journal, with the other fixes Mobster learns.
Check a pack
A pack health run visits each tab a pack knows on your iPhone, by taps on tabs alone, with no model call, and ends with a checklist of the screens it named. Start one as a task withpackCheck, the pack’s key (POST /api/runs with {"appId": "instagram", "goal": "Check the Instagram pack", "packCheck": "instagram"}), or schedule that task to run each morning (Schedules). It answers like “Instagram pack: working on 448.0.0; 1 screen changed.”
Keep your numbers
A stats run reads your own profile and your insights with no model call, and stores today’s numbers for that app and handle: followers, following, posts (Instagram), likes (TikTok), and the insights above. Start one as a task withpackStats, the pack’s key (POST /api/runs with {"appId": "com.burbn.instagram", "goal": "Store my Instagram numbers", "packStats": "instagram"}), or schedule that task each morning (Schedules). With a profile on the run, it stores nothing unless the app shows that profile’s handle, and says so. GET /api/accounts/instagram/me.test/stats?days=30 returns what is stored, one row a day, oldest first, with latest; it never touches the phone. The Instagram and TikTok insights screens are built from the apps’ usual labels and wait for a recording from an account that has them. On an account without them the step stops and the profile numbers are still stored.
Each step a pack takes adds one line to the hint ledger, hints.jsonl beside Mobster’s journal: the step, whether it landed, the screen and how sure the pack was, the time, and the app version. It holds no screen text and no names. GET /api/packs shows each pack’s health: the version it was checked on, the version on your iPhone, state (healthy, degraded or unknown), a summary line, the changedScreens it last missed, and how many variants it has learned (Scripting).
Requests for more than 40 rows, or for zero rows, go to the model. The code path never silently reduces the count you asked for.