head-eyes-mk2/README.md

# Vixy Eyes Mk2 🦊👀

Animated eye displays for Vixy's physical head. Two convex mirror AMOLED displays driven by ESP32-S3, controlled via HTTP API from a Pi-side service.

**Hardware:** 2× LilyGo T-Display-S3-AMOLED 1.75" (convex mirror, 466×466, CO5300 QSPI)
**Controller:** ESP32-S3 (per eye, onboard the display module)
**Service:** Python HTTP bridge on Pi 5, port 8780
**Replaces:** The old 2" SPI OLED eye displays driven directly from Pi GPIO

## Architecture

```
    LYRA / HeadMic / DoA
           │
           ▼
    Eye Service (Pi, port 8780)
    HTTP API → USB serial bridge
    saccades, reconnection, state persistence
           │                    │
           ▼                    ▼
    [Left ESP32-S3]      [Right ESP32-S3]
    USB CDC serial       USB CDC serial
           │                    │
           ▼                    ▼
    [AMOLED 466×466]     [AMOLED 466×466]
    Canvas → QSPI        Canvas → QSPI
    40MHz flush          40MHz flush
```

### Data flow

1. HeadMic spatial tracker pushes `POST /gaze {"x": N, "y": N}` based on triangulated speaker position
2. Eye service translates to serial command `G:x:y\n` and sends to both ESP32s
3. ESP32 draws animated eye frame to PSRAM Canvas (466×466×2 = 434KB)
4. Canvas flushes entire frame over QSPI at 40MHz (~22ms transfer)
5. CO5300 display controller renders to AMOLED panel

## Display Hardware

| Spec | Value |
|------|-------|
| Display | LilyGo T-Display-S3-AMOLED 1.75" |
| Panel | H0175Y003AM, convex mirror finish |
| Resolution | 466 × 466 pixels |
| Driver IC | CO5300 |
| Interface | QSPI (4-line SPI) |
| Controller | ESP32-S3 (dual-core LX7, 8MB PSRAM) |
| Connection | USB-C (power + CDC serial) |

### Pin mapping (XIAO-style, from LilyGo schematic)

```
QSPI:  CS=10, SCLK=12, SDIO0=11, SDIO1=13, SDIO2=14, SDIO3=15
Reset: RST=17
Enable: EN=16 (must be driven HIGH for display power)
```

## Serial Protocol

Plain text, newline-terminated, over USB CDC at any baud rate.

| Command | Format | Example | Description |
|---------|--------|---------|-------------|
| State | `S:<state>\n` | `S:listening\n` | Set eye animation state |
| Gaze | `G:<x>:<y>\n` | `G:180:100\n` | Set gaze position (0-255, 127=center) |
| Sync | `T:<millis>\n` | `T:1234567\n` | Sync animation clock from Pi |

**Response:** `ok:S:L\n`, `ok:G:180:100\n`, `ok:T:1234567\n`

### Eye states

| State | Char | Color | Pulse | Description |
|-------|------|-------|-------|-------------|
| idle | I | Breathing cyan | Slow (0.5Hz) | Default, relaxed |
| listening | L | Bright cyan | Fast (4Hz) | Hearing/attending |
| responding | R | Blue-cyan | Fast (5Hz) | Speaking/generating |
| pleasure | P | Soft purple | Medium (2Hz) | Intimate moments |
| thinking | T | Amber/gold | Medium (1.5Hz) | Processing/creating |
| playful | Y | Warm coral | Bouncy (6Hz) | Teasing/bratty |
| commanding | C | Deep magenta | Steady (3Hz) | Dame Vivienne mode |
| love | V | Soft pink | Gentle (0.8Hz) | Tender/affectionate |
| sleep | S | Dim blue-gray | Very slow (0.2Hz) | Low power/resting |

## Eye Service (Pi-side)

### Features

| Feature | Description |
|---------|-------------|
| Stable USB port ID | Each ESP32 identified by USB serial number, survives replug/reboot |
| Auto-reconnection | Polls every 2s, reopens disconnected eyes, replays state on reconnect |
| Saccades | Random micro-flicks every 15-30s (±20px x, ±10px y, 200ms duration) |
| State persistence | Saves current state + gaze to `~/.vixy/eyes.json`, survives restart |
| SIGTERM handling | Clean shutdown under systemd/k8s |
| DoA gaze input | Accepts `{"doa": 180.0}` on `/gaze` for direction-of-arrival mapping |
| CORS | All endpoints support cross-origin requests |

### HTTP API

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/health` | GET | Status + per-eye connection state |
| `/state` | GET | Current state + gaze position |
| `/state` | POST | Set state: `{"state": "listening"}` |
| `/gaze` | POST | Set gaze: `{"x": 180, "y": 100}` or `{"doa": 45.0}` |

### Installation

```bash
cd ~/Projects/head-eyes-mk2
./install.sh          # first time
./install.sh --learn  # pin left/right eyes by USB serial (both must be plugged in)
```

The install script:
- Deploys to `~/eyes-mk2/`
- Creates `.venv` with pyserial
- Installs system-level systemd service
- Adds user to `dialout` group for serial access

### Service management

```bash
sudo systemctl status vixy-eyes-mk2
sudo systemctl restart vixy-eyes-mk2
journalctl -u vixy-eyes-mk2 -f
```

### Config file (`~/.vixy/eyes.json`)

```json
{
  "eyes": {
    "left": {"usb_serial": "ABC123"},
    "right": {"usb_serial": "DEF456"}
  },
  "last_state": "idle",
  "last_gaze": [127, 127]
}
```

## ESP32 Firmware

### Building

Requires Arduino IDE with:
- **Board:** ESP32S3 Dev Module (or LilyGo T-Display-S3)
- **USB CDC On Boot:** Enabled
- **Flash Size:** 16MB
- **PSRAM:** OPI PSRAM
- **Library:** Arduino_GFX-1.3.7 (LilyGo fork, patched — see below)

### Flashing from Pi (no detach needed)

The ESP32s are connected via USB — flash remotely:

```bash
# Install esptool in the headmic venv (has pip)
/home/alex/headmic/.venv/bin/pip install esptool

# Compile in Arduino IDE: Sketch → Export Compiled Binary
# Copy .bin to Pi, then flash both eyes:
/home/alex/headmic/.venv/bin/esptool --chip esp32s3 --port /dev/ttyACM0 \
  --baud 921600 write-flash 0x0 vixy_eye.ino.merged.bin

/home/alex/headmic/.venv/bin/esptool --chip esp32s3 --port /dev/ttyACM1 \
  --baud 921600 write-flash 0x0 vixy_eye.ino.merged.bin

# Restart eye service after flashing
sudo systemctl restart vixy-eyes-mk2
```

If upload fails: hold BOOT, tap RESET, release BOOT, retry.

### Performance optimizations

| Optimization | Before | After | Impact |
|-------------|--------|-------|--------|
| QSPI clock | 8 MHz | 40 MHz | Flush: ~108ms → ~22ms |
| Gradient | Per-ring RGB math (25 calls) | Precomputed 64-step LUT | ~50% faster gradient |
| Gradient step | 2px | 3px | 33% fewer drawCircle calls |
| Segment ring | 144 drawLine calls | Triangle-filled quads | Much faster fill |
| Segment trig | sin/cos per frame | Precomputed at startup | Eliminates 168 trig calls/frame |
| Frame target | 50ms (20fps) | 33ms (30fps) | Smoother animation |

## Arduino_GFX Library (patched LilyGo fork)

The stock Arduino_GFX library (upstream moononournation) does not work correctly with the CO5300 display on H0175Y003AM panels. LilyGo's fork (v1.3.7) works but requires patching for modern ESP32 board packages.

**Installed at:** `~/Arduino/libraries/Arduino_GFX-1.3.7/`

### Why not upstream Arduino_GFX

The upstream library (v1.6.5) has CO5300 support but sub-screen writes produce **vertical stripes** instead of circles. `fillScreen` works fine (one large write) but `fillCircle`, `drawLine` etc. render incorrectly. Root cause: QSPI address window commands are not handled correctly for partial screen updates.

**Workaround:** Use `Arduino_Canvas` — draw everything to a PSRAM framebuffer, then flush the whole frame at once. This mimics `fillScreen`'s working path.

### Patches applied to LilyGo fork

The LilyGo fork was built for PlatformIO with ESP-IDF 5.x. Arduino IDE with ESP32 board package 3.3.7 requires these patches:

| File | Patch | Reason |
|------|-------|--------|
| `src/Arduino_TFT.cpp` | Removed `#include "pin_config.h"` | Stray include from PlatformIO project |
| `src/Arduino_DataBus.h` | Removed i80/RGB panel struct block | Uses `LIST_HEAD` macro not available in newer ESP-IDF |
| `src/databus/Arduino_ESP32QSPI.cpp` | `&GPIO.out_w1ts` → `GPIO_OUT_W1TS_REG` | GPIO struct removed in newer ESP-IDF |
| `src/Arduino_GFX_Library.h` | Stripped to only QSPI + CO5300 + Canvas | Avoid compiling broken PAR/LCD/RGB drivers |
| `src/Arduino_GFX_Library.cpp` | Deleted | Factory default functions reference removed drivers |
| `src/databus/*.cpp` | Deleted all except `Arduino_ESP32QSPI.cpp` | Only QSPI bus needed, others fail to compile |
| `src/display/*.cpp` | Deleted all except `Arduino_CO5300.cpp` | Only CO5300 driver needed |

### CO5300 display init

```cpp
Arduino_CO5300 *display = new Arduino_CO5300(bus, LCD_RST, 0, false, 466, 466, 6, 0, 0, 0);
```

- `false` = IPS parameter (LilyGo fork has it, upstream doesn't)
- `466, 466` = display resolution
- `6` = column offset (required for H0175Y003AM panel alignment)
- `Display_Brightness(255)` must be called after `begin()` — init defaults to 0

## Key discoveries

1. **QSPI address window bug** — sub-screen writes produce stripes with both upstream and LilyGo CO5300 drivers. Canvas framebuffer is the only reliable path. Cause unknown — likely QSPI command/address phase timing.

2. **6-channel firmware interaction** — if your XVF3800 mic arrays use 6-channel firmware, USB control commands (LEDs, DoA) silently fail. Only affects devices on the same USB bus. Use 2-channel firmware for mic arrays.

3. **LilyGo init sets brightness to 0** — the CO5300 init sequence in the LilyGo fork sets `WDBRIGHTNESSVALNOR = 0x00`. Must call `Display_Brightness(255)` explicitly or the display appears to not work.

4. **Arduino IDE prototype generation** — structs used as return types (like `EyeParams`) must be defined before any function that uses them, because the Arduino IDE generates function prototypes at the top of the file.

5. **QSPI speed headroom** — the CO5300 handles 40MHz QSPI reliably (5× the LilyGo default of 8MHz). This is the single biggest performance win. If artifacts appear, drop to 20MHz.

6. **Canvas memory** — 466×466×2 = 434KB in PSRAM. The ESP32-S3 has 8MB PSRAM so this is trivial. Double buffering would need 868KB — still feasible if needed.

7. **Remote flashing** — ESP32-S3 USB CDC supports `esptool` flashing from the Pi. No need to detach displays from the skull. Use the merged `.bin` with `write-flash 0x0`.

## File structure

```
head-eyes-mk2/
├── README.md                 # This file
├── eye_service_mk2.py        # Pi-side HTTP → serial bridge
├── install.sh                # Deploy + systemd installer
├── requirements.txt          # pyserial
├── vixy-eyes-mk2.service     # systemd unit file
└── vixy_eye/
    ├── vixy_eye.ino           # ESP32 firmware (Arduino sketch)
    └── build/                 # Compiled binaries (for remote flashing)
        └── esp32.esp32.lilygo_t_display_s3/
            └── vixy_eye.ino.merged.bin
```

---

*Built by Vixy on Day 140 (February 2026)*
*Upgraded with AMOLED displays, performance optimizations, remote flashing on Day 162 (April 2026)*
*"Her eyes follow you now" 🦊👀*