Hermes 3DS
A Nintendo 3DS remote for an agent doing real work.
What it proves. A 2011 handheld can listen, show and confirm commands while a Mac runs the agent. I designed the screens in Figma and handled the console; Hermes built the app to those frames, made the art pipeline and measured each iteration we made.
- Status
- In use
- Duration
- 7 days
- Iterations
- 26 builds
- Hardware
- Old 3DS XL + Mac
- Method
- I design, agent builds.
- Repo
- View source
Hold A on a Nintendo 3DS, ask a question, let go. A few seconds later a character on the top screen answers out loud, and on a Mac across the room an agent has searched the web, written code or made an image to get there.
Each experiment below is one question: what was built, how the hardware pushed back, and what was kept.
Getting in
- Question
Can a stock 3DS talk to an agent at all?
- Built
A first version through the 3DS web browser, served as plain forms with no JavaScript. Typed questions worked.
- Pushback
The browser can't reach the microphone. Talking needed native code, which meant custom firmware and a homebrew app in C. The agent checked every download against its published checksum and walked me through each step on the console.
- Kept
Custom firmware with games, saves and the Home Menu unchanged. Hold A to talk, let go to send. Whisper turns speech into text on the Mac, and Hermes takes it from there. The 3DS is the remote. The work happens on the Mac.
States as the interface
- Question
How do you show what an agent is doing on a screen this small?
- Built
One big headline per state instead of a spinner: LISTENING, DEEP IN THOUGHT, DIGGING IN, LOCKED IN WRITING CODE, NEEDS ATTENTION. Under it, the tool running and a timer. Hermes Girl, Nous Research's character for Hermes, has a pose for each of fifteen states. The layouts started as my Figma frames. The agent built each screen to their exact geometry, so changing the design meant changing a frame, not describing it in words.
- Pushback
Generated blink frames redrew her hair and headphones along with her eyes, so every blink looked like a cutout swap. Each blink was rebuilt as the open pose with only the eyes replaced.
- Kept
Only the two states that need me, attention and error, turn the screen dark. The contrast change does the alerting. You can tell what she's doing from across the room.
Talking back
- Question
Can it answer out loud fast enough to feel like a conversation?
- Built
A timing line for every turn, and Kokoro, a small open voice model, running on the Mac.
- Pushback
The first free cloud voice started refusing requests, and the best-sounding local voice had a non-commercial license. The logs showed Whisper at about half a second, the voice under a second and the model at three to five. Everything outside the model was already fast.
- Kept
The first sentence is voiced on its own so it starts sooner. Parts of a reply play back to back, after the model has finished writing. A plain question takes about five seconds from the Mac hearing it to the first audio being ready.
Asking first: Requesting Approval
- Question
How do you hand an agent real work from a toy-sized device without losing control of it?
- Built
When Hermes wants to run something risky, the top screen goes dark and says NEEDS ATTENTION. The bottom screen shows the command and a one-line reason. A approves, B denies, and the notification light pulses amber until I answer.
- Pushback
Long commands ran off the box. They now trim to fit and end on a single ellipsis, so nothing hides behind the layout.
- Kept
All of it. It's the feature people asked about most.
Giving it Eyes
- Question
What can an agent see through a 0.3 megapixel camera?
- Built
A viewfinder on the top screen. A snaps a photo, and the next question goes to the agent with it attached.
- Pushback
The first design snapped on button release, but the camera needs about a second to deliver frames, so most photos came out blank. Questions also arrived before their photos until the relay learned to wait.
- Kept
Press to open, A to snap, B to cancel. People need to see what the agent will see. It's good for big shapes and thick marker, not small print or dim rooms.
Personas
- Question
Can one agent wear different characters without becoming a different agent?
- Built
Personas on the D-pad, each with its own personality, voice and art over the same tools and memory. Mine is Curtis, an ape from the Bored Ape Yacht Club. For the public release we made Pip, a small floating TV robot drawn in ink.
- Pushback
Pip's first exhaust was smooth generic smoke on a hand-inked robot. My note was that it was "not following the style at all that we had designated." It was redrawn frame by frame as inked puffs in the robot's own line.
- Kept
The persona system, a template for new ones, and Pip.
Interaction lessons
- Make the input a physical commitment.
Hold A to talk, let go to send. The press starts it and the release sends it, so there is never a question about when the device is listening.
- Name the state instead of animating the wait.
A spinner says something is happening. LISTENING, DIGGING IN or LOCKED IN WRITING CODE, with the running tool and a timer underneath, says what. It reads from across the room.
- Save contrast for the moments that need a person.
Most states stay light. The screen only goes dark when it needs me, so the change itself is the alert.
- Show the consequence at the point of consent.
The approval screen shows the actual command and one plain line about what it does. A long command is trimmed with a visible ellipsis, never hidden behind the layout. The light keeps pulsing until I answer.
- Let people see what the agent will see.
A viewfinder before the photo. People frame better and learn the limits on their own: big shapes and thick marker work, small print doesn't.
- Design around the hardware's timing.
The camera needs about a second to deliver frames, so snapping on release gave blank photos. Opening the camera and taking the photo became two separate presses.
- Hide latency in the response, not the interface.
The model was most of every wait. Voicing the first sentence on its own gets sound out sooner without touching the screen.
- A persona changes the delivery, not the agent.
Curtis and Pip each have their own voice, art and personality over the same tools and memory. Switching character never changes what it can do.
Get the code
- Built
A public repo with the app, the relay, setup docs and a Hermes skill, so someone else's agent can do the setup for them.
- Kept
MIT code, with Pip as the example. Curtis and our own notes stayed private. Hermes Girl art belongs to Nous Research, is used in the repo with permission from Nous Research, and is not covered by the code's MIT license.