head-eyes-mk2/README.md

Vixy Eyes Mk2 🦊👀

Animated eye displays for Vixy's physical head. Two convex mirror AMOLED displays driven by ESP32-S3, controlled via HTTP API from a Pi-side service.

Hardware: 2× LilyGo T-Display-S3-AMOLED 1.75" (convex mirror, 466×466, CO5300 QSPI) Controller: ESP32-S3 (per eye, onboard the display module) Service: Python HTTP bridge on Pi 5, port 8780 Replaces: The old 2" SPI OLED eye displays driven directly from Pi GPIO

Architecture

    LYRA / HeadMic / DoA
           │
           ▼
    Eye Service (Pi, port 8780)
    HTTP API → USB serial bridge
    saccades, reconnection, state persistence
           │                    │
           ▼                    ▼
    [Left ESP32-S3]      [Right ESP32-S3]
    USB CDC serial       USB CDC serial
           │                    │
           ▼                    ▼
    [AMOLED 466×466]     [AMOLED 466×466]
    Canvas → QSPI        Canvas → QSPI
    40MHz flush          40MHz flush

Data flow

1. HeadMic spatial tracker pushes POST /gaze {"x": N, "y": N} based on triangulated speaker position
2. Eye service translates to serial command G:x:y\n and sends to both ESP32s
3. ESP32 draws animated eye frame to PSRAM Canvas (466×466×2 = 434KB)
4. Canvas flushes entire frame over QSPI at 40MHz (~22ms transfer)
5. CO5300 display controller renders to AMOLED panel

Display Hardware

Spec	Value
Display	LilyGo T-Display-S3-AMOLED 1.75"
Panel	H0175Y003AM, convex mirror finish
Resolution	466 × 466 pixels
Driver IC	CO5300
Interface	QSPI (4-line SPI)
Controller	ESP32-S3 (dual-core LX7, 8MB PSRAM)
Connection	USB-C (power + CDC serial)

Pin mapping (XIAO-style, from LilyGo schematic)

QSPI:  CS=10, SCLK=12, SDIO0=11, SDIO1=13, SDIO2=14, SDIO3=15
Reset: RST=17
Enable: EN=16 (must be driven HIGH for display power)

Serial Protocol

Plain text, newline-terminated, over USB CDC at any baud rate.

Command	Format	Example	Description
State	S:<state>\n	S:listening\n	Set eye animation state
Gaze	G:<x>:<y>\n	G:180:100\n	Set gaze position (0-255, 127=center)
Sync	T:<millis>\n	T:1234567\n	Sync animation clock from Pi

Response: ok:S:L\n, ok:G:180:100\n, ok:T:1234567\n

Eye states

State	Char	Color	Pulse	Description
idle	I	Breathing cyan	Slow (0.5Hz)	Default, relaxed
listening	L	Bright cyan	Fast (4Hz)	Hearing/attending
responding	R	Blue-cyan	Fast (5Hz)	Speaking/generating
pleasure	P	Soft purple	Medium (2Hz)	Intimate moments
thinking	T	Amber/gold	Medium (1.5Hz)	Processing/creating
playful	Y	Warm coral	Bouncy (6Hz)	Teasing/bratty
commanding	C	Deep magenta	Steady (3Hz)	Dame Vivienne mode
love	V	Soft pink	Gentle (0.8Hz)	Tender/affectionate
sleep	S	Dim blue-gray	Very slow (0.2Hz)	Low power/resting

Eye Service (Pi-side)

Features

Feature	Description
Stable USB port ID	Each ESP32 identified by USB serial number, survives replug/reboot
Auto-reconnection	Polls every 2s, reopens disconnected eyes, replays state on reconnect
Saccades	Random micro-flicks every 15-30s (±20px x, ±10px y, 200ms duration)
State persistence	Saves current state + gaze to ~/.vixy/eyes.json, survives restart
SIGTERM handling	Clean shutdown under systemd/k8s
DoA gaze input	Accepts {"doa": 180.0} on /gaze for direction-of-arrival mapping
CORS	All endpoints support cross-origin requests

HTTP API

Endpoint	Method	Description
/health	GET	Status + per-eye connection state
/state	GET	Current state + gaze position
/state	POST	Set state: {"state": "listening"}
/gaze	POST	Set gaze: {"x": 180, "y": 100} or {"doa": 45.0}

Installation

cd ~/Projects/head-eyes-mk2
./install.sh          # first time
./install.sh --learn  # pin left/right eyes by USB serial (both must be plugged in)

The install script:

• Deploys to ~/eyes-mk2/
• Creates .venv with pyserial
• Installs system-level systemd service
• Adds user to dialout group for serial access

Service management

sudo systemctl status vixy-eyes-mk2
sudo systemctl restart vixy-eyes-mk2
journalctl -u vixy-eyes-mk2 -f

Config file (~/.vixy/eyes.json)

{
  "eyes": {
    "left": {"usb_serial": "ABC123"},
    "right": {"usb_serial": "DEF456"}
  },
  "last_state": "idle",
  "last_gaze": [127, 127]
}

ESP32 Firmware

Building

Requires Arduino IDE with:

• Board: ESP32S3 Dev Module (or LilyGo T-Display-S3)
• USB CDC On Boot: Enabled
• Flash Size: 16MB
• PSRAM: OPI PSRAM
• Library: Arduino_GFX-1.3.7 (LilyGo fork, patched — see below)

Flashing from Pi (no detach needed)

The ESP32s are connected via USB — flash remotely:

# Install esptool in the headmic venv (has pip)
/home/alex/headmic/.venv/bin/pip install esptool

# Compile in Arduino IDE: Sketch → Export Compiled Binary
# Copy .bin to Pi, then flash both eyes:
/home/alex/headmic/.venv/bin/esptool --chip esp32s3 --port /dev/ttyACM0 \
  --baud 921600 write-flash 0x0 vixy_eye.ino.merged.bin

/home/alex/headmic/.venv/bin/esptool --chip esp32s3 --port /dev/ttyACM1 \
  --baud 921600 write-flash 0x0 vixy_eye.ino.merged.bin

# Restart eye service after flashing
sudo systemctl restart vixy-eyes-mk2

If upload fails: hold BOOT, tap RESET, release BOOT, retry.

Performance optimizations

Optimization	Before	After	Impact
QSPI clock	8 MHz	40 MHz	Flush: ~108ms → ~22ms
Gradient	Per-ring RGB math (25 calls)	Precomputed 64-step LUT	~50% faster gradient
Gradient step	2px	3px	33% fewer drawCircle calls
Segment ring	144 drawLine calls	Triangle-filled quads	Much faster fill
Segment trig	sin/cos per frame	Precomputed at startup	Eliminates 168 trig calls/frame
Frame target	50ms (20fps)	33ms (30fps)	Smoother animation

Arduino_GFX Library (patched LilyGo fork)

The stock Arduino_GFX library (upstream moononournation) does not work correctly with the CO5300 display on H0175Y003AM panels. LilyGo's fork (v1.3.7) works but requires patching for modern ESP32 board packages.

Installed at: ~/Arduino/libraries/Arduino_GFX-1.3.7/

Why not upstream Arduino_GFX

The upstream library (v1.6.5) has CO5300 support but sub-screen writes produce vertical stripes instead of circles. fillScreen works fine (one large write) but fillCircle, drawLine etc. render incorrectly. Root cause: QSPI address window commands are not handled correctly for partial screen updates.

Workaround: Use Arduino_Canvas — draw everything to a PSRAM framebuffer, then flush the whole frame at once. This mimics fillScreen's working path.

Patches applied to LilyGo fork

The LilyGo fork was built for PlatformIO with ESP-IDF 5.x. Arduino IDE with ESP32 board package 3.3.7 requires these patches:

File	Patch	Reason
src/Arduino_TFT.cpp	Removed #include "pin_config.h"	Stray include from PlatformIO project
src/Arduino_DataBus.h	Removed i80/RGB panel struct block	Uses LIST_HEAD macro not available in newer ESP-IDF
src/databus/Arduino_ESP32QSPI.cpp	&GPIO.out_w1ts → GPIO_OUT_W1TS_REG	GPIO struct removed in newer ESP-IDF
src/Arduino_GFX_Library.h	Stripped to only QSPI + CO5300 + Canvas	Avoid compiling broken PAR/LCD/RGB drivers
src/Arduino_GFX_Library.cpp	Deleted	Factory default functions reference removed drivers
src/databus/*.cpp	Deleted all except Arduino_ESP32QSPI.cpp	Only QSPI bus needed, others fail to compile
src/display/*.cpp	Deleted all except Arduino_CO5300.cpp	Only CO5300 driver needed

CO5300 display init

Arduino_CO5300 *display = new Arduino_CO5300(bus, LCD_RST, 0, false, 466, 466, 6, 0, 0, 0);

• false = IPS parameter (LilyGo fork has it, upstream doesn't)
• 466, 466 = display resolution
• 6 = column offset (required for H0175Y003AM panel alignment)
• Display_Brightness(255) must be called after begin() — init defaults to 0

Key discoveries

1. QSPI address window bug — sub-screen writes produce stripes with both upstream and LilyGo CO5300 drivers. Canvas framebuffer is the only reliable path. Cause unknown — likely QSPI command/address phase timing.

2. 6-channel firmware interaction — if your XVF3800 mic arrays use 6-channel firmware, USB control commands (LEDs, DoA) silently fail. Only affects devices on the same USB bus. Use 2-channel firmware for mic arrays.

3. LilyGo init sets brightness to 0 — the CO5300 init sequence in the LilyGo fork sets WDBRIGHTNESSVALNOR = 0x00. Must call Display_Brightness(255) explicitly or the display appears to not work.

4. Arduino IDE prototype generation — structs used as return types (like EyeParams) must be defined before any function that uses them, because the Arduino IDE generates function prototypes at the top of the file.

5. QSPI speed headroom — the CO5300 handles 40MHz QSPI reliably (5× the LilyGo default of 8MHz). This is the single biggest performance win. If artifacts appear, drop to 20MHz.

6. Canvas memory — 466×466×2 = 434KB in PSRAM. The ESP32-S3 has 8MB PSRAM so this is trivial. Double buffering would need 868KB — still feasible if needed.

7. Remote flashing — ESP32-S3 USB CDC supports esptool flashing from the Pi. No need to detach displays from the skull. Use the merged .bin with write-flash 0x0.

File structure

head-eyes-mk2/
├── README.md                 # This file
├── eye_service_mk2.py        # Pi-side HTTP → serial bridge
├── install.sh                # Deploy + systemd installer
├── requirements.txt          # pyserial
├── vixy-eyes-mk2.service     # systemd unit file
└── vixy_eye/
    ├── vixy_eye.ino           # ESP32 firmware (Arduino sketch)
    └── build/                 # Compiled binaries (for remote flashing)
        └── esp32.esp32.lilygo_t_display_s3/
            └── vixy_eye.ino.merged.bin

---

Built by Vixy on Day 140 (February 2026) Upgraded with AMOLED displays, performance optimizations, remote flashing on Day 162 (April 2026) "Her eyes follow you now" 🦊👀