EYOX
Eyox Scan capture work, explained for Ike
For Ike, from Ruben

Eyox Scan capture: the eye model and the move-closer hint

Ike, this page shows what Ruben built and decided with agent sessions for the Eyox Scan capture flow, and how it looks. Three stacked pull requests carry the work. You are the requested reviewer of the first one.

1

The camera moved to VisionCamera 5

The Scan capture path runs on VisionCamera 5, which gives the app live camera frames on iOS. The Data app's capture is retired. #1091, VisionCamera 5 2

The phone recognises the eye

While the optician aims, a small Core ML model on the iPhone finds the eye in the framing oval and measures how wide it is. iOS 17 and later. #1092, on-device eye model 3

The optician is told to move closer

When the eye is too small in the oval, the band under the oval says “Kom dichterbij” (“Move closer”). It never blocks Record. #1093, move-closer hint

Why

A specialist has less detail to grade when the eye is small in the frame. Before this work nothing on the phone recognised the eye: the “close enough” check was always true, and its “Move closer to camera” text never showed.

How it ships

Behind two remote switches. The model runs first in shadow: nothing shows, and the app records what the hint would have done. The hint goes on, first in one store, only when that data shows it helps.

2

VisionCamera 5

PR #1091 Jira: Move the Scan capture path to VisionCamera 5 head d2074062 base main reviewer: Ike

The Scan capture path moved from VisionCamera 4.7.3 to 5.2.4 and keeps what capture does on main: 4K at 30 fps, stabilisation off, mp4, no audio, the same view and the same torch levels. VisionCamera 5 is a rewrite on Nitro (sessions, outputs, a controller), not a version bump.

The session, before and after

<CameraCapture>
-  <Camera device format fps photo video torch torchLevel />
-  cameraRef.startRecording({ videoCodec, ... })
+  <Camera device outputs=[preview, video, ...extraOutputs]
+          constraints=captureConstraints() />
+  during the 3 s countdown: videoOutput.createRecorder()
+  after it: recorder.startRecording(onFinished, onError)
+  torch: controller.enableTorchWithStrength(level)
+         or controller.setTorchMode(on | off)

Session choices live in one place: captureConstraints in packages/camera/src/captureSession.ts.

Torch

The 220-line VisionCamera 4 patch is gone. JS clamps the torch level to 0.01 to 1, and a 10-line patch keeps the fallback to the highest available level when the iPhone is too hot.

Android torch fix

On VisionCamera 5.2.4 the old torch-off on pause did nothing: it reflected into a VisionCamera 4 class that no longer exists. The patch now maps a paused host to lifecycle CREATED in CustomLifecycle.kt, so the torch goes off when the app goes to the background. Kotlin only; iOS and the JS bundle are unchanged. Test: androidTorchOffOnPause.test.ts.

iOS guidance

VisionCamera 5 has no preview snapshot on iOS. The iOS snapshot analysis never had an analyser behind it and always settled on the same result, so iOS now shows that result without polling. The oval and align line look as before.

Guidance pauses from the Record press to the end of the recording, without a zoom reset.

A seam for the eye model

CameraCapture takes extraOutputs, appended after the preview and video outputs. The eye model pull request adds its frame output there. No frame output and no worklets in this pull request.

Data app capture retired

The shared camera code no longer compiles against the Data app's VisionCamera 4. Its capture route now shows “Capture is not available”, and the Data app imports only the camera-free parts of the shared packages.

How it was tested

  • Android: on the emulator, with the full Maestro suite in the Dev build. All flows passed, none retried.
  • No physical Android device check: all store capture devices are iPhones.
  • iPhone: Ruben ran the full device test script on TestFlight Dev 1.6.0 (127). It passed.
  • Local on d2074062: Scan Jest 147 suites and 1,618 tests, typecheck, lint with 0 errors, format check. GitHub PR checks are green.
  • Size: the iOS app payload grows by 3.6 MB unpacked (37.2 to 40.8 MB), almost all of it VisionCamera 5 and Nitro in the main binary.

What follows

3

The on-device eye model

PR #1092 Jira: Scan: recognise the eye and iris on the phone while the optician aims head 4cfb8277 base: the #1091 branch new: scan-eye-model flag

While the optician aims, a small trained model reads camera frames on the iPhone and reports three things. It is a capture aid: it says nothing about the eye's condition.

Eye in ovalyes / noWhether an eye is inside the framing oval.
Eye width% of ovalThe horizontal extent of the visible eye opening, iris included, as a share of the oval width. This drives the hint.
Iris diameter% of ovalAs a share of the oval width. The hint uses it only as a check.

When several eye areas are found, the one that contains iris wins, then the one nearest the oval centre.

Where it runs

  1. VisionCamera 5 frame outputScan iOS 17 and later, preview-sized, upright frames.
  2. Worklet on the frame output's threadLets one frame through every 250 ms while aiming (four per second), one analysis at a time.
  3. @eyox/eye-frame-capture (Nitro)An upright copy in memory, at most 960 px. No file is written.
  4. EyeSegmenterModule (Swift, Core ML)A square around the oval, 1.1 times its width. A 256 px pass; a 384 px pass when there is no eye or a very small one; a zoom pass for eyes under 60 px.
  5. capture-flow eyeModelReads the eye and iris from the label map: eye in oval, eye width, iris diameter. The move-closer hint takes it from there.

The model is off from the Record press through the countdown and the recording. The frame output stays attached, so the camera session is not reconfigured at Record. Dev builds add a readout line and a “Detection on/off” overlay; store builds show neither.

Size and speed

6.6 MBModelCore ML ML Program, fp16. LR-ASPP MobileNetV3-Large, three classes: background, eye opening, iris. One package for 256 and 384 px inputs.
+6.86 MBDownload+7.37 MB installed, Dev build 128 against 127. The ticket allows 10 MB.
2 msModel timePlus 6 ms decode, on an iPhone 17 on the earlier snapshot path. On camera frames: still to measure.
4 / sAnalysesOne every 250 ms while aiming. The criterion: at least one per second, each at most 100 ms.

Locked: the weights are fixed in the build. No learning on the phone, no download at runtime. The package traces to one training run, exp_562_student_v1, in its metadata. A new model is a new package in a reviewed pull request.

Considered options, and why each lost

  • Apple Vision face landmarksNo extra size, but unusable close up (Ruben's iPhone test of a Dev build).
  • An OpenCV iris circle finderOn the same build it often never found the iris, and it added 7.3 MB to the download and 17.0 MB installed.
  • MediaPipe Face MeshFails on about 88% of Eyox close-ups.
  • The desktop eye segmenter in eyox-ml-training (DeepLabV3-ResNet101)199 MB, about 110 ms per frame on a Mac GPU, and no phone export.
  • Analysis on a serverRuled out: no frame leaves the phone for this analysis.
  • Guidance snapshots instead of camera framesEach analysis cost a JPEG encode, write, read and decode, and VisionCamera 5 has no iOS snapshot. Ruben decided that live analysis runs on frames.
  • Other variants of the same model, kept as optionsMobileNetV3-Small (2.27 MB) is level within the confidence intervals, but the chosen model was integrated and is slightly better on close-ups. A student without pretrained backbone weights is level on close-ups and weaker on the smallest far eyes. int8 weights (3.44 MB) need iOS 16.
Chosen: one small trained eye and iris segmentation model inside the Scan iOS app, run with Core ML on camera frames, iOS 17 and later, as a capture aid. It won because the rule-based and off-the-shelf options failed on Eyox close-ups, while this one is small and fast and keeps every frame on the phone.

Why iOS 17 and later

Below iOS 17, VisionCamera 5 cannot give the frame output smaller buffers (full 4K frames instead), and it is not established that older iOS accepts a frame output next to the movie file output that records the clip. Scan supports iOS 15.1, so iPhones on iOS 15 and 16 get no model and no hint.

Android deferred

LiteRT versions of the same weights exist at 256 and 384 px (int8 3.9 MB, fp16 6.7 MB each). Nothing has run on an Android phone, and there is no Android test phone for this work.

Consequences to know

  • The frame output joins the capture session on iOS 17 and later. If iOS refused it next to the movie file output, recording would fail. Not seen on a device.
  • The worklets Babel plugin transforms the whole Scan bundle; the gesture handler builds with worklets on iOS.
  • Each model version adds about 6.6 MB of binary to the repository.

Still to check on an iPhone with iOS 17 or later

Time per analysis on camera frames; a clip against main with the same resolution and frame rate; the eye with the whole face in the frame; flicker at the bar; the hint copy on an iPhone SE size at large text.

4

The move-closer hint

PR #1093 Jira: Scan: tell the optician to move closer when the eye is too far from the camera head c4eb9ad4 base: the #1092 branch in review: scan-move-closer-hint flag and shadow fields

When the eye is too small in the oval, the optician sees “Kom dichterbij” (EN “Move closer”) in the band under the oval, in place of the align line. A dark pill with an amber expand icon and white text. It hides during the countdown and the recording, and it never blocks Record.

50%

The bar

The eye's width as a share of the oval width. The hint shows only once the judged share is below 47% (the bar minus 3 points), so it does not flicker at the edge. The judged share is the median of the last three readings.

0.5 s

Timing

Reliable readings below the bar in a row for at least 0.5 s, and at least two of them. At four per second: readings at 0, 250 and 500 ms.

0.25 to 0.90

Reliable iris range

A reading is reliable with an eye in the oval, an iris, and an iris diameter of 0.25 to 0.90 of the eye's width. Unreliable readings below the bar change nothing.

1.5 s

Stale guard

After 1.5 s without a reliable reading the hint hides. It also hides at once without an eye, on the first reliable reading at or above the bar, and after two unreliable ones in a row at or above it.

Try it: the hint rules at four readings per second

A drawn mock of the capture screen. The eye is an illustration. The simulator runs a copy of the rules in moveCloser/hint.ts with the numbers in thresholds.ts, on one reading every 250 ms.

Eye model: analysing

Drawn mock of the Scan capture screen with the hint as built (variant A). The “Model” tag at the top is a page annotation, not app UI.

40%

Move it below 47% to see the hint come up after 0.5 s; above 50% it goes at once.

Mode
Copy
Hint
hidden
Hint mode: shown on screen
Below-bar streak
0 readings
needs 2 readings and 500 ms
Last reliable reading
none
hides after 1.5 s without one

Judged share (median of the last three) against the bar

Filled bar: the judged share. Dot: the latest reading.

The rules, in the order the code checks them

  1. 1No eye in the oval: hide at once and start over.
  2. 2Stale guard: hide after 1.5 s without a reliable reading.
  3. 3A reliable reading at or above the bar (50%): hide at once.
  4. 4Unreliable readings at or above the bar: hide after two in a row; one alone changes nothing.
  5. 5An unreliable reading below the bar (a blink, a bad iris): hold, change nothing.
  6. 6A reliable reading with the median below 47%: count it. Show once the count lasts 0.5 s with at least two readings.
  7. 7A reliable reading with the median between 47% and 50%: a shown hint stays, a count that has not shown yet starts over.
  8. 8Record press or another eye: start over. Record always works; the model is off until the recording ends.

Readings, newest first

TimeReadingJudgedRuleHint

What capture_eye_finished keeps for the last Record press

Press Record on the mock to see the values sent for that recording.

Settings still open, to tune from shadow data

  • The 1.5 s stale guard is about six missed readings at four per second; 0.75 to 1 s would keep it inside the one-second hide criterion.
  • The median over three readings.
  • One reading without an eye hides the hint at once.
  • PostHog gets the raw last reading at the press; the median the hint judged is one of the new shadow fields.

Out of scope

5

Guardrails

The model and the hint help the optician record. They are held to five lines, in the code and in both pull requests.

Capture aid only

It tells the optician how to record. Device pathway “No” in both pull requests.

Says nothing about the eye

No grade, score, finding or referral advice. It measures position and size in the oval, nothing about the eye's condition.

Never blocks or judges

Record always works. The clip and the still the specialist grades are unchanged: no masking, no cropping.

Off while recording

From the Record press through the countdown and the recording, the model does not run and the hint does not show.

No frame leaves the phone

No network call in the eye model code or the frame module. PostHog gets derived numbers only, never an image.

Where the line comes from: the regulatory advice Eyox received is that capture hints are instructions for use as long as they say nothing about the eye, and that the software must not steer the decision. A use that changes the graded recording or says something about the eye needs a new decision.

6

Rollout with flags

Jira: Scan: switch on the move-closer hint in the stores only when shadow data shows it helps

The model and the hint reach the stores behind two PostHog flags, switched per store login. Both exist and are off. The hint goes on only after two checks on shadow data.

  1. Phase 1Shadow scan-eye-modelscan-move-closer-hint The model runs while the optician aims. Nothing shows. The app records what the hint would have done.
  2. CheckTechnical After at least one week of shadow: no crashes or capture regressions, the model ran on at least 95% of iPhone recordings, and 95% of analyses finished within 250 ms.
  3. CheckValue After at least 20 shadow eye recordings: the would-be hint at the Record press, compared with the specialist's verdict and any recapture.
  4. Phase 2 and 3Hint, one store, then all scan-eye-modelscan-move-closer-hint The hint shows, first for one store, then for all stores. Each switch-on is recorded on the rollout ticket.
  5. AfterClean up scan-eye-model The hint flag is removed. The model flag stays for good as the kill switch: off stops the model and the hint without an app release.

Store volume is low (2 to 9 scan submissions a week), so one week of shadow shows whether the model runs well on store phones, but the value check counts recordings, not weeks.

scan-eye-model

In #1092. On: the model runs (shadow). Off: the camera works as on main with VisionCamera 5. No model is looked up, no frame output exists, nothing of the model shows. Permanent: the kill switch.

scan-move-closer-hint in review

In #1093. On: the hint shows on top of the model. No effect while the model flag is off. Temporary: removed after the full rollout.

  • Both flags are off by default, and count as off when they cannot load: offline at launch, a failed fetch, or cached flags of another user on a shared store phone. The app trusts only flags that a reload returned for the user signed in now.
  • The flags are read once when the capture screen opens. A change never switches anything while the optician aims; it applies at the next capture screen.

Try it: the flag simulator

Set in PostHog
On this phone

The open capture screen runs inoff

The optician sees
On the phone
Recorded

Considered and rejected

  • Switching the hint on directly at release: nothing would show whether the hint helps or nags before every optician sees it.
  • Gathering the rollout data only from internal Dev builds: Dev builds are not representative of store use (other people, phones, light and stores).

A consequence

A flag change changes the app's behaviour in the stores without a new build or review. Who is allowed to change the flags, and how a change is recorded, belongs with the release process.

7

Telemetry

Everything the checks need rides on one existing event, capture_eye_finished (in PostHog: scan:capture_eye_finished), sent once per eye when the capture screen closes. Derived numbers only: no frame, image or eye condition. Fields that end in _at_record hold one entry per recording that started.

On main #1092, eye model timing New on #1092 with the flag #1093, hint at the press New on #1093 (in review)

Hover over or tap a field to see what it means. Click a field to pin it.

An example eye in shadow mode, with every field of the finished stack. The values are made up for illustration; they are not measurements.

8

The PR stack and how to review it

Three pull requests in one GitHub stack (stack #1096). Each one's base is the branch below it, so each shows only its own commits.

main
#1091VisionCamera 5d2074062 #1092On-device eye model4cfb8277 #1093Move-closer hintc4eb9ad4
Waiting off main: the Android lens-type fix (Scan Android: on VisionCamera 5, every lens inside a multi-lens camera reports its type as unknown). Its pull request opens against main after #1091 merges.

#1091 VisionCamera 5

Jira: Move the Scan capture path to VisionCamera 5. Base main. Requested reviewer: Ike.

Holds the camera move, the torch patch with the Android torch-off fix, the iOS guidance change, the extraOutputs seam, the Data capture retirement, the fixes from your review, and a merge of main.

Worth a close look

  • packages/camera/src/
  • captureSession.ts
  • hooks/useCamera.ts
  • packages/capture-flow/src/
  • CameraCapture.tsx
  • apps/eyox-scan-mobile-app/patches/
  • react-native-vision-camera+5.2.4.patch

#1092 On-device eye model

Jira: Scan: recognise the eye and iris on the phone while the optician aims. Base: the #1091 branch.

Holds the frame output and its 4 Hz gate, the Nitro frame copy, the Core ML module and model package, the reading of the label map, the scan-eye-model flag and the new eye_model_* fields.

Worth a close look

  • packages/capture-flow/src/
  • eyeModel/useEyeFrameOutput.ios.ts
  • eyeModel/scheduler.ts, reading.ts
  • captureTiming.ts
  • packages/
  • eye-frame-capture/
  • apps/eyox-scan-mobile-app/ios/EyoxScanMobile/
  • EyeSegmenterModule.swift
  • apps/eyox-scan-mobile-app/src/services/
  • eyeModelFlags.ts

#1093 Move-closer hint

Jira: Scan: tell the optician to move closer when the eye is too far from the camera. Base: the #1092 branch.

Holds the hint rules and thresholds, the band under the oval, the Dev tuning controls, the scan-move-closer-hint flag, shadow mode and the move_closer_* fields.

Worth a close look

  • packages/capture-flow/src/moveCloser/
  • hint.ts
  • thresholds.ts
  • useMoveCloserHint.ts
  • MoveCloserBand.tsx

How to review it

  1. Start at the bottom with #1091. It targets main.
  2. Because the three are linked as a GitHub stack, the main PR workflows run on every pull request in it, not only on #1091. The Dev builds (TestFlight Dev, Play internal) carry the device evidence.
  3. Merge bottom up: #1091 first. GitHub then retargets #1092 to main, and #1093 follows #1092. Each lands on main as its own squash commit; a pull request is never merged into the branch below it.
  4. When #1091 changes, it is merged forward into #1092 and then #1093 with a merge commit. The stacked branches are never force-pushed.

Links