Connect people with the world,
so they look forward, not down.
Meta's mission was building the future of human connection. We needed compelling use cases to find product-market fit on display glasses.
ROLEI led the end-to-end multimodal AI design leveraging photo input and OCR to help people understand and take actions on what they see, including standardizing the human-AI interaction model and gesture vocabulary, contributing to design system, and creative direction for onboarding, education, and commercial artwork.
OUTCOMELaunched at Meta Connect 2025. The interaction model my engineer and I designed was filed as a patent, with me as first inventor.
A restaurant abroad. A menu you can't read.
How can wearers consume translations on a transparent display the size of a postage stamp?
If the wearer can't discover the gesture, can't navigate, and can't read the translation — the AI feels broken regardless of whether the translation model is accurate.
Consumer
Discovery
Wearers didn't know zoom and pan were possible on the 600 × 600px display.
Gesture friction
Captouch swipe to zoom was easily mistaken as taps. Head-IMU panning felt jittery.
Legibility
The additive display, contrast, and image resolution can make it hard to read texts.
Technical
Natural invocation
Corresponding visual response templates only fired on rigid commands — wearers had to remember exact trigger phrases for the AI to show rich visual responses.
Image resolution ↔ latency
Sharper text on the 600 × 600px display needs higher-resolution renders and deeper zoom — heavier processing, slower response.
System understanding ≠ human understanding
OCR groups real-world text by adjacency — orientation, line breaks, and layout work against it.
What you see ≠ what you hear
1 OCR input feeds 2 separate models. The rendered translation includes over-detected noise the LLM's audio response cuts — and even on clean input, 2 models means 2 different wordings.
A translation you can
read, move through, and act on —
with nothing to touch.
Every design decision held to 3 principles: intuitive, easy, quick.
Result format
How might we show the translated result on the display and in the AI's audio response — what's the relationship between the two?
Navigation
How might we enable wearers to move through the translated content — with no direct-manipulation surface like a touchscreen?
Follow-up actions
How might we offer the right actions once the translation lands to help wearers reach their goal?
The devil is in the details.
Understanding was never the goal —
acting on it was.
A translation is only useful if you can map it back to the world in front of you. I made the call on intuition and competitor analysis, we built it, then validated with UXR.
Two ways to put a translation on a display you can’t touch. I took the one that keeps your eyes in one place.
- Can transfer attention to the display entirely, with no extra eye movement to scan and compare between capture preview and translation
- Can’t scale to include image
- Might be challenging to read depending on the color of the documents and the background environment users are in
Machine picks by proximity;
human follows a path.
Next, how do you move through it — on a screen this small, with nothing to touch? Constant pinch-pan-zoom is exhausting, so I bet on removing the manual work with Smart text selection.
Text selection may sound simple. But in reality it's much messier than you can imagine. Real-world text has various orientation, linebreak, and layout. Jiaqian and I relentlessly refined the calculation so the system identifies orientation, proximity, and chooses the next text block aligned with human reading order. The calculation also reduces panning to the minimum to avoid dizziness.
Two ways to get from one line to the next with nothing to touch. Both auto-zoom; they disagree about what moves — the image, or the selection.
- Same mental model as most of the image panning mechanism: fixed viewport + moving image
- Less confident about which line item is going to be zoomed in
- When text boxes are really close to each other, how does the bounding box work
Flexibility was the design —
not the fallback.
Not everyone agreed on leaning this hard on automation. Leadership pushed to keep manual free pan-and-zoom. We didn't settle it by winning; we made the modes complement each other: smart selection for the common path, free pan-and-zoom whenever you want it.
How it works.
Ask generically by natural invocation or specify with your finger.
Pinch and drag to pan where you want. Wrist roll to zoom in on the section. Swipe to consume each line.
Reveal the original language or shift long-form text into system template tuned for 600 × 600 legibility. Other follow up actions are designed and scoped in future milestones.
Resolve gesture conflict, provide contextual education
Shipped, patented, recognized.
I’d go upstream sooner.
If I did it again, I'd build a tighter feedback loop with the ML, OCR, and Camera teams earlier. Some of my design decisions were band-aids for upstream model and camera limitations — earlier collaboration would have let me shape UX-relevant model behavior directly, instead of designing around it downstream.