81 lines
2.8 KiB
Markdown
81 lines
2.8 KiB
Markdown
# Facial Recognition: OAK-D + Coral Edge TPU
|
|
|
|
Add face detection and recognition to the oak-service spatial pipeline.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
OAK-D Lite (Myriad X) Coral Edge TPU Host (Pi 5)
|
|
────────────────────── ────────────── ───────────
|
|
yolov6-nano spatial ssd_mobilenet_v2_face crop person bbox
|
|
→ person bboxes → face bboxes cosine similarity
|
|
→ spatial coords (X,Y,Z) arcface/facenet edgetpu vs SQLite DB
|
|
→ RGB frames → 128-dim embedding → name + confidence
|
|
```
|
|
|
|
Per detection cycle (~0.5s):
|
|
1. OAK-D outputs person detections + spatial coords + RGB frame (unchanged)
|
|
2. Host crops upper-body region from RGB for each person bbox
|
|
3. Coral runs face detection on crop (ssd_mobilenet_v2_face edgetpu)
|
|
4. If face found, crop face, resize to model input, run embedding via Coral
|
|
5. Host compares embedding against SQLite DB (cosine similarity)
|
|
6. Attach recognized_name + recognition_confidence to detection
|
|
|
|
## Setup: Coral Runtime
|
|
|
|
Install pycoral + tflite-runtime in the oak-service venv:
|
|
```bash
|
|
pip install tflite-runtime pycoral
|
|
```
|
|
|
|
Download Edge TPU models:
|
|
- ssd_mobilenet_v2_face_quant_postprocess_edgetpu.tflite
|
|
- face embedding model (facenet or arcface quantized for edgetpu)
|
|
|
|
Models stored in oak-service/models/ directory.
|
|
|
|
## SQLite Face Database
|
|
|
|
Path: configurable, default `faces.db` in service directory.
|
|
|
|
```sql
|
|
CREATE TABLE faces (
|
|
id INTEGER PRIMARY KEY,
|
|
name TEXT NOT NULL,
|
|
embedding BLOB NOT NULL,
|
|
enrolled_at REAL NOT NULL,
|
|
source TEXT
|
|
);
|
|
CREATE INDEX idx_faces_name ON faces(name);
|
|
```
|
|
|
|
- Multiple embeddings per person (different angles/lighting)
|
|
- Embedding stored as packed float32 bytes
|
|
- Matching: cosine similarity, threshold ~0.5 for positive match
|
|
- Best match across all embeddings for a name wins
|
|
|
|
## API Changes
|
|
|
|
New endpoints:
|
|
- `POST /faces/enroll` — multipart: name + photo, or name + use current frame
|
|
- `GET /faces` — list enrolled names with embedding count
|
|
- `DELETE /faces/{name}` — remove person from DB
|
|
|
|
Modified responses:
|
|
- `/presence` adds: recognized_name, recognition_confidence
|
|
- `/detections` adds per-detection: recognized_name, recognition_confidence
|
|
|
|
## Files
|
|
|
|
- `oak_service_spatial.py` — add Coral face pipeline to detection loop
|
|
- `models/` — Edge TPU model files
|
|
- `faces.db` — SQLite database (created on first run)
|
|
|
|
## Verification
|
|
|
|
1. Install Coral runtime, verify device detected
|
|
2. Download face models, verify inference runs
|
|
3. Enroll a face via API
|
|
4. Test recognition: stand in front of camera, check /presence for name
|
|
5. Test unknown: different person, should show "unknown"
|