oak-service/docs/plans/2026-02-01-facial-recognition-design.md

Facial Recognition: OAK-D + Coral Edge TPU

Add face detection and recognition to the oak-service spatial pipeline.

Architecture

OAK-D Lite (Myriad X)              Coral Edge TPU              Host (Pi 5)
──────────────────────              ──────────────              ───────────
yolov6-nano spatial                 ssd_mobilenet_v2_face       crop person bbox
  → person bboxes                     → face bboxes             cosine similarity
  → spatial coords (X,Y,Z)         arcface/facenet edgetpu      vs SQLite DB
  → RGB frames                       → 128-dim embedding        → name + confidence

Per detection cycle (~0.5s):

1. OAK-D outputs person detections + spatial coords + RGB frame (unchanged)
2. Host crops upper-body region from RGB for each person bbox
3. Coral runs face detection on crop (ssd_mobilenet_v2_face edgetpu)
4. If face found, crop face, resize to model input, run embedding via Coral
5. Host compares embedding against SQLite DB (cosine similarity)
6. Attach recognized_name + recognition_confidence to detection

Setup: Coral Runtime

Install pycoral + tflite-runtime in the oak-service venv:

pip install tflite-runtime pycoral

Download Edge TPU models:

• ssd_mobilenet_v2_face_quant_postprocess_edgetpu.tflite
• face embedding model (facenet or arcface quantized for edgetpu)

Models stored in oak-service/models/ directory.

SQLite Face Database

Path: configurable, default faces.db in service directory.

CREATE TABLE faces (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    embedding BLOB NOT NULL,
    enrolled_at REAL NOT NULL,
    source TEXT
);
CREATE INDEX idx_faces_name ON faces(name);

• Multiple embeddings per person (different angles/lighting)
• Embedding stored as packed float32 bytes
• Matching: cosine similarity, threshold ~0.5 for positive match
• Best match across all embeddings for a name wins

API Changes

New endpoints:

• POST /faces/enroll — multipart: name + photo, or name + use current frame
• GET /faces — list enrolled names with embedding count
• DELETE /faces/{name} — remove person from DB

Modified responses:

• /presence adds: recognized_name, recognition_confidence
• /detections adds per-detection: recognized_name, recognition_confidence

Files

• oak_service_spatial.py — add Coral face pipeline to detection loop
• models/ — Edge TPU model files
• faces.db — SQLite database (created on first run)

Verification

1. Install Coral runtime, verify device detected
2. Download face models, verify inference runs
3. Enroll a face via API
4. Test recognition: stand in front of camera, check /presence for name
5. Test unknown: different person, should show "unknown"