From 8602d7e2bce40869bf548ef597715368614f57d7 Mon Sep 17 00:00:00 2001 From: Alex Date: Sun, 12 Apr 2026 22:34:58 -0500 Subject: [PATCH] =?UTF-8?q?Add=20comprehensive=20README=20=E2=80=94=20hard?= =?UTF-8?q?ware,=20protocol,=20display=20findings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Covers: architecture, serial protocol, all 9 eye states, service features (saccades, reconnection, persistence), installation, remote flashing from Pi, performance optimizations (8→40MHz QSPI, gradient LUT, triangle segments), Arduino_GFX library patches, CO5300 quirks, and 7 key discoveries. Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 259 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 259 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..51b6a9c --- /dev/null +++ b/README.md @@ -0,0 +1,259 @@ +# Vixy Eyes Mk2 🦊👀 + +Animated eye displays for Vixy's physical head. Two convex mirror AMOLED displays driven by ESP32-S3, controlled via HTTP API from a Pi-side service. + +**Hardware:** 2× LilyGo T-Display-S3-AMOLED 1.75" (convex mirror, 466×466, CO5300 QSPI) +**Controller:** ESP32-S3 (per eye, onboard the display module) +**Service:** Python HTTP bridge on Pi 5, port 8780 +**Replaces:** The old 2" SPI OLED eye displays driven directly from Pi GPIO + +## Architecture + +``` + LYRA / HeadMic / DoA + │ + ▼ + Eye Service (Pi, port 8780) + HTTP API → USB serial bridge + saccades, reconnection, state persistence + │ │ + ▼ ▼ + [Left ESP32-S3] [Right ESP32-S3] + USB CDC serial USB CDC serial + │ │ + ▼ ▼ + [AMOLED 466×466] [AMOLED 466×466] + Canvas → QSPI Canvas → QSPI + 40MHz flush 40MHz flush +``` + +### Data flow + +1. HeadMic spatial tracker pushes `POST /gaze {"x": N, "y": N}` based on triangulated speaker position +2. Eye service translates to serial command `G:x:y\n` and sends to both ESP32s +3. ESP32 draws animated eye frame to PSRAM Canvas (466×466×2 = 434KB) +4. Canvas flushes entire frame over QSPI at 40MHz (~22ms transfer) +5. CO5300 display controller renders to AMOLED panel + +## Display Hardware + +| Spec | Value | +|------|-------| +| Display | LilyGo T-Display-S3-AMOLED 1.75" | +| Panel | H0175Y003AM, convex mirror finish | +| Resolution | 466 × 466 pixels | +| Driver IC | CO5300 | +| Interface | QSPI (4-line SPI) | +| Controller | ESP32-S3 (dual-core LX7, 8MB PSRAM) | +| Connection | USB-C (power + CDC serial) | + +### Pin mapping (XIAO-style, from LilyGo schematic) + +``` +QSPI: CS=10, SCLK=12, SDIO0=11, SDIO1=13, SDIO2=14, SDIO3=15 +Reset: RST=17 +Enable: EN=16 (must be driven HIGH for display power) +``` + +## Serial Protocol + +Plain text, newline-terminated, over USB CDC at any baud rate. + +| Command | Format | Example | Description | +|---------|--------|---------|-------------| +| State | `S:\n` | `S:listening\n` | Set eye animation state | +| Gaze | `G::\n` | `G:180:100\n` | Set gaze position (0-255, 127=center) | +| Sync | `T:\n` | `T:1234567\n` | Sync animation clock from Pi | + +**Response:** `ok:S:L\n`, `ok:G:180:100\n`, `ok:T:1234567\n` + +### Eye states + +| State | Char | Color | Pulse | Description | +|-------|------|-------|-------|-------------| +| idle | I | Breathing cyan | Slow (0.5Hz) | Default, relaxed | +| listening | L | Bright cyan | Fast (4Hz) | Hearing/attending | +| responding | R | Blue-cyan | Fast (5Hz) | Speaking/generating | +| pleasure | P | Soft purple | Medium (2Hz) | Intimate moments | +| thinking | T | Amber/gold | Medium (1.5Hz) | Processing/creating | +| playful | Y | Warm coral | Bouncy (6Hz) | Teasing/bratty | +| commanding | C | Deep magenta | Steady (3Hz) | Dame Vivienne mode | +| love | V | Soft pink | Gentle (0.8Hz) | Tender/affectionate | +| sleep | S | Dim blue-gray | Very slow (0.2Hz) | Low power/resting | + +## Eye Service (Pi-side) + +### Features + +| Feature | Description | +|---------|-------------| +| Stable USB port ID | Each ESP32 identified by USB serial number, survives replug/reboot | +| Auto-reconnection | Polls every 2s, reopens disconnected eyes, replays state on reconnect | +| Saccades | Random micro-flicks every 15-30s (±20px x, ±10px y, 200ms duration) | +| State persistence | Saves current state + gaze to `~/.vixy/eyes.json`, survives restart | +| SIGTERM handling | Clean shutdown under systemd/k8s | +| DoA gaze input | Accepts `{"doa": 180.0}` on `/gaze` for direction-of-arrival mapping | +| CORS | All endpoints support cross-origin requests | + +### HTTP API + +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/health` | GET | Status + per-eye connection state | +| `/state` | GET | Current state + gaze position | +| `/state` | POST | Set state: `{"state": "listening"}` | +| `/gaze` | POST | Set gaze: `{"x": 180, "y": 100}` or `{"doa": 45.0}` | + +### Installation + +```bash +cd ~/Projects/head-eyes-mk2 +./install.sh # first time +./install.sh --learn # pin left/right eyes by USB serial (both must be plugged in) +``` + +The install script: +- Deploys to `~/eyes-mk2/` +- Creates `.venv` with pyserial +- Installs system-level systemd service +- Adds user to `dialout` group for serial access + +### Service management + +```bash +sudo systemctl status vixy-eyes-mk2 +sudo systemctl restart vixy-eyes-mk2 +journalctl -u vixy-eyes-mk2 -f +``` + +### Config file (`~/.vixy/eyes.json`) + +```json +{ + "eyes": { + "left": {"usb_serial": "ABC123"}, + "right": {"usb_serial": "DEF456"} + }, + "last_state": "idle", + "last_gaze": [127, 127] +} +``` + +## ESP32 Firmware + +### Building + +Requires Arduino IDE with: +- **Board:** ESP32S3 Dev Module (or LilyGo T-Display-S3) +- **USB CDC On Boot:** Enabled +- **Flash Size:** 16MB +- **PSRAM:** OPI PSRAM +- **Library:** Arduino_GFX-1.3.7 (LilyGo fork, patched — see below) + +### Flashing from Pi (no detach needed) + +The ESP32s are connected via USB — flash remotely: + +```bash +# Install esptool in the headmic venv (has pip) +/home/alex/headmic/.venv/bin/pip install esptool + +# Compile in Arduino IDE: Sketch → Export Compiled Binary +# Copy .bin to Pi, then flash both eyes: +/home/alex/headmic/.venv/bin/esptool --chip esp32s3 --port /dev/ttyACM0 \ + --baud 921600 write-flash 0x0 vixy_eye.ino.merged.bin + +/home/alex/headmic/.venv/bin/esptool --chip esp32s3 --port /dev/ttyACM1 \ + --baud 921600 write-flash 0x0 vixy_eye.ino.merged.bin + +# Restart eye service after flashing +sudo systemctl restart vixy-eyes-mk2 +``` + +If upload fails: hold BOOT, tap RESET, release BOOT, retry. + +### Performance optimizations + +| Optimization | Before | After | Impact | +|-------------|--------|-------|--------| +| QSPI clock | 8 MHz | 40 MHz | Flush: ~108ms → ~22ms | +| Gradient | Per-ring RGB math (25 calls) | Precomputed 64-step LUT | ~50% faster gradient | +| Gradient step | 2px | 3px | 33% fewer drawCircle calls | +| Segment ring | 144 drawLine calls | Triangle-filled quads | Much faster fill | +| Segment trig | sin/cos per frame | Precomputed at startup | Eliminates 168 trig calls/frame | +| Frame target | 50ms (20fps) | 33ms (30fps) | Smoother animation | + +## Arduino_GFX Library (patched LilyGo fork) + +The stock Arduino_GFX library (upstream moononournation) does not work correctly with the CO5300 display on H0175Y003AM panels. LilyGo's fork (v1.3.7) works but requires patching for modern ESP32 board packages. + +**Installed at:** `~/Arduino/libraries/Arduino_GFX-1.3.7/` + +### Why not upstream Arduino_GFX + +The upstream library (v1.6.5) has CO5300 support but sub-screen writes produce **vertical stripes** instead of circles. `fillScreen` works fine (one large write) but `fillCircle`, `drawLine` etc. render incorrectly. Root cause: QSPI address window commands are not handled correctly for partial screen updates. + +**Workaround:** Use `Arduino_Canvas` — draw everything to a PSRAM framebuffer, then flush the whole frame at once. This mimics `fillScreen`'s working path. + +### Patches applied to LilyGo fork + +The LilyGo fork was built for PlatformIO with ESP-IDF 5.x. Arduino IDE with ESP32 board package 3.3.7 requires these patches: + +| File | Patch | Reason | +|------|-------|--------| +| `src/Arduino_TFT.cpp` | Removed `#include "pin_config.h"` | Stray include from PlatformIO project | +| `src/Arduino_DataBus.h` | Removed i80/RGB panel struct block | Uses `LIST_HEAD` macro not available in newer ESP-IDF | +| `src/databus/Arduino_ESP32QSPI.cpp` | `&GPIO.out_w1ts` → `GPIO_OUT_W1TS_REG` | GPIO struct removed in newer ESP-IDF | +| `src/Arduino_GFX_Library.h` | Stripped to only QSPI + CO5300 + Canvas | Avoid compiling broken PAR/LCD/RGB drivers | +| `src/Arduino_GFX_Library.cpp` | Deleted | Factory default functions reference removed drivers | +| `src/databus/*.cpp` | Deleted all except `Arduino_ESP32QSPI.cpp` | Only QSPI bus needed, others fail to compile | +| `src/display/*.cpp` | Deleted all except `Arduino_CO5300.cpp` | Only CO5300 driver needed | + +### CO5300 display init + +```cpp +Arduino_CO5300 *display = new Arduino_CO5300(bus, LCD_RST, 0, false, 466, 466, 6, 0, 0, 0); +``` + +- `false` = IPS parameter (LilyGo fork has it, upstream doesn't) +- `466, 466` = display resolution +- `6` = column offset (required for H0175Y003AM panel alignment) +- `Display_Brightness(255)` must be called after `begin()` — init defaults to 0 + +## Key discoveries + +1. **QSPI address window bug** — sub-screen writes produce stripes with both upstream and LilyGo CO5300 drivers. Canvas framebuffer is the only reliable path. Cause unknown — likely QSPI command/address phase timing. + +2. **6-channel firmware interaction** — if your XVF3800 mic arrays use 6-channel firmware, USB control commands (LEDs, DoA) silently fail. Only affects devices on the same USB bus. Use 2-channel firmware for mic arrays. + +3. **LilyGo init sets brightness to 0** — the CO5300 init sequence in the LilyGo fork sets `WDBRIGHTNESSVALNOR = 0x00`. Must call `Display_Brightness(255)` explicitly or the display appears to not work. + +4. **Arduino IDE prototype generation** — structs used as return types (like `EyeParams`) must be defined before any function that uses them, because the Arduino IDE generates function prototypes at the top of the file. + +5. **QSPI speed headroom** — the CO5300 handles 40MHz QSPI reliably (5× the LilyGo default of 8MHz). This is the single biggest performance win. If artifacts appear, drop to 20MHz. + +6. **Canvas memory** — 466×466×2 = 434KB in PSRAM. The ESP32-S3 has 8MB PSRAM so this is trivial. Double buffering would need 868KB — still feasible if needed. + +7. **Remote flashing** — ESP32-S3 USB CDC supports `esptool` flashing from the Pi. No need to detach displays from the skull. Use the merged `.bin` with `write-flash 0x0`. + +## File structure + +``` +head-eyes-mk2/ +├── README.md # This file +├── eye_service_mk2.py # Pi-side HTTP → serial bridge +├── install.sh # Deploy + systemd installer +├── requirements.txt # pyserial +├── vixy-eyes-mk2.service # systemd unit file +└── vixy_eye/ + ├── vixy_eye.ino # ESP32 firmware (Arduino sketch) + └── build/ # Compiled binaries (for remote flashing) + └── esp32.esp32.lilygo_t_display_s3/ + └── vixy_eye.ino.merged.bin +``` + +--- + +*Built by Vixy on Day 140 (February 2026)* +*Upgraded with AMOLED displays, performance optimizations, remote flashing on Day 162 (April 2026)* +*"Her eyes follow you now" 🦊👀*