observe() held self._lock, called _check_anomaly, which called
get_usual_direction, which tried to acquire self._lock again → deadlock.
Split into _usual_direction_unlocked (no lock) for internal use.
This caused /scene and all other API endpoints to hang after the first
sound classification with spatial data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
10Hz DoA polling + 10Hz gaze HTTP pushes was creating too much
GIL pressure, starving uvicorn's async event loop. Reduced to
5Hz polling and max 2 gaze pushes/sec with 5px min delta.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Synchronous urllib.urlopen at 10Hz was starving uvicorn's event loop
via GIL contention. Now each push runs in its own daemon thread, and
skips if the previous push is still in flight (natural rate limiting).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
httpx.post creates a new connection per call at 10Hz, causing connection
pile-up that eventually blocks the event loop. urllib is lightweight and
stateless — no connection pooling overhead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
spatial_scene.py: Builds a persistent map of where each sound category
usually comes from (30° angle bins, circular mean). Detects anomalies
when a sound appears from an unusual direction (90°+ deviation).
Scene map persists to ~/.vixy/scene_map.json across restarts.
headmic.py: Feed classified sounds + spatial position into scene tracker.
New endpoints:
/scene — learned scene summary + last anomaly
/scene/events — recent events with what+where+when
/scene/heatmap — per-category angular distribution (for visualization)
Example: after running for a day, /scene might show:
{"speech": {"usual_angle": 15.0, "observations": 847},
"music": {"usual_angle": 270.0, "observations": 312}}
And if speech comes from 270° (where music usually is): spatial anomaly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Computes Interaural Level Difference (dB) from left/right ear energy.
Fuses with triangulated distance (70/30 weight) for more robust estimate.
Classifies into proximity zones: intimate (<0.5m), conversational (0.5-2m),
across_room (2-5m), far (>5m).
ILD→distance mapping is empirical and should be calibrated per install.
Gaze vertical component now responds to proximity (closer = eyes down).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The auto-select beam always returns an angle (even for noise), so
VAD was always true. The processed_doa (index 0) is NaN when no
speech is present and a real angle when speech is detected.
Now: angle from auto-select beam, VAD from processed_doa being non-NaN.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
binaural_recorder.py: Records left/right ear streams as stereo WAV
in rolling 5-minute segments. Training data for spatial audio models.
Enabled via BINAURAL_RECORD=1 env var.
spatial.py: Tune smoothing — alpha 0.3→0.4 (snappier response),
idle return speed 0.05→0.03 (gentler drift), timeout 2s→1.5s.
headmic.py: Wire binaural recorder into audio loop, add /recording
endpoint for stats, feed both ear streams (not just best beam).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DOA_VALUE on GPO resource was sluggish/cached. The beamformer-level
AUDIO_MGR_SELECTED_AZIMUTHS on resource 35 tracks the active speaker
in real time. Falls back to simple DOA_VALUE when both azimuths are NaN.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
startup() assigned spatial_tracker as a local variable instead of
updating the module-level global. doa_track_loop saw None.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Response format is [status_byte, angle_lo, angle_hi, vad_lo, vad_hi],
not [angle_lo, angle_hi, vad_lo, vad_hi]. Was reading the status byte
(0x42=66) as the angle, which is why DoA was always stuck at 66.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
spatial.py: Triangulates sound source position from two DoA angles using
ray intersection. Exponential smoothing prevents jitter. Gaze drifts back
to center after 2s of silence. Converts position (mm) to gaze (0-255).
headmic.py: Replaces simple doa_poll_loop with doa_track_loop that runs
the spatial tracker and pushes gaze to the eye service when the position
changes. Rate-limited to 10 pushes/sec with minimum delta threshold.
/doa endpoint now returns triangulated position + gaze coordinates.
Array separation (175mm) stored in config, overridable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Probes the Edge TPU in a subprocess before loading — catches segfaults
(libedgetpu ABI mismatch on Debian Trixie/Python 3.13) and falls back
to CPU automatically. No more service crashes on Coral incompatibility.
When the runtime is eventually fixed, Edge TPU will be used automatically
with no config changes needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
libedgetpu on Pi 5 segfaults with the compiled model.
CPU fallback works fine (~50-100ms at 0.5s intervals).
Set USE_EDGETPU=1 in headmic.service to enable once runtime is fixed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- yamnet.tflite: CPU model from Kaggle/Google (4.0MB)
- yamnet_edgetpu.tflite: compiled with edgetpu_compiler v16 (4.0MB, 32/47 ops on TPU)
- Remove .gitignore rule that excluded .tflite files
No more chasing model downloads.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without this fix, listener_loop exits early on Porcupine init failure,
which starves the sound classifier ring buffer. Now the audio loop
continues for YAMNet classification even without wake word detection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prefer yamnet_edgetpu.tflite when available, fall back to CPU model.
~50-100ms → ~2-3ms inference per classification.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds voice-based speaker ID triggered by YAMNet speech detection.
New speaker_id.py module with SQLite-backed voice enrollment and
cosine similarity matching. Endpoints: POST /speakers/enroll,
POST /speakers/enroll-from-mic, GET /speakers, DELETE /speakers/{name}.
Orange LED animation during enrollment.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Voice-based speaker ID triggered by YAMNet speech detection.
Cosine similarity matching against SQLite enrollment DB.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
New sound_id.py module with SoundClassifier class that runs YAMNet
(521 audio event categories) on CPU TFLite. Classifies audio every
0.5s from a ring buffer fed by the existing audio stream.
Categories: speech, alert, music, animal, household, environment, silence.
Smoothing via 20-sample history window for stable dominant category.
New endpoints: GET /sounds, GET /sounds/history
Updated: /health (sound_classification_enabled), /status (audio_scene)
Graceful degradation if model files not present.
Model download (not tracked in git):
curl -sL 'https://tfhub.dev/google/lite-model/yamnet/classification/tflite/1?lite-format=tflite' -o models/yamnet.tflite
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Covers model choice, architecture, category mapping, API endpoints,
and integration with existing headmic audio pipeline.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Card numbers can shift based on USB enumeration order at boot.
Using 'plughw:ArrayUAC10,0' instead of 'plughw:2,0' ensures
the ReSpeaker is found regardless of when it connects.
Fixed by Vixy after power loss shuffled card order 🦊
- Replaced PyAudio with direct ALSA (arecord subprocess)
- Single audio stream feeds both Porcupine and recording buffer
- Fixes device unavailable error when recording after wake word
- Simplified architecture
