Fruit fly connectome on an ESP32-S3

Overview

A port of part of a real brain to an ESP32-S3: 48,311 neurons and 9,462,135 connections from the male fruit fly connectome, MaleCNS v1.0, compressed to 13.5MiB in flash. Nothing is trained: a spider on an OLED drives the fly’s own looming detectors, and its escape neurons decide when it jumps.

What it does

• Holds 48,311 neurons and 9,462,135 connections from the male fruit fly connectome, MaleCNS v1.0, with the synaptic contact count of every connection kept exactly. [1]
• Escapes a spider on the OLED: the spider’s growing size drives the fly’s looming detectors, LC4 and LPLC2, and when the escape neurons on that side, including DNp01, get active enough, the fly leaps away. [2]
• No model was trained: the port ships a small text file naming the input and output neurons and one threshold, and the behavior comes from the fly’s own wiring. [3]
• The graph is 13.5MiB compressed in flash with blocks cached in PSRAM, and a new decision comes about every 1.7 s. [4]

Use-case ideas

• Without a display the demo still runs and reports over serial. [5]

Parts and requirements

• esp32:esp32 core 3.3.10
• Adafruit SH110X, Adafruit GFX Library and Adafruit BusIO
• Requirements source: [8]

Setup effort

Two commands: fetch_model.sh fly downloads and verifies the graph, circuit and gold; deploy.sh fly checks the bundle, runs the host gold, compiles, then flashes the graph and the firmware. [10]

Why it's interesting

This demo puts part of a real brain on a microcontroller: 48,311 neurons and 9,462,135 connections from the male fruit fly connectome (MaleCNS v1.0), with the synaptic contact count of every connection kept exactly. The README says the graph is 13.5 MiB compressed in flash, and every simulation step runs through all connections on both ESP32-S3 cores using the chip's SIMD instructions. Nothing is trained. A spider growing on screen drives the fly's own looming detectors, LC4 and LPLC2, and when escape neurons such as DNp01, the giant fiber, get active enough the fly leaps away. The author checked that stimulating those detectors activates the escape neurons 30 to 100 times more than random visual neurons. It is a selected part of the nervous system, not the whole fly.

Source: github.com/slvDev/esp32-ai#porting-part-of-the-fruit-fly-connectome-to-esp32

Hardware

• ESP32-S3 N16R8
• OLED

More ESP32 builds

Credits and links

• slvDev: Creator
• FlyEM Project at HHMI Janelia, University of Cambridge, MRC Laboratory of Molecular Biology and Google Research: MaleCNS v1.0 connectome, used under CC BY 4.0

• Fly firmware README  ↗
• What is ported (docs/fly-connectome)  ↗
• Fly model on Hugging Face  ↗

Sources

1. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/README.md?plain=1#L67-L69
2. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/README.md?plain=1#L89-L93
3. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/README.md?plain=1#L95-L97
4. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/README.md?plain=1#L83-L84
5. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/firmware/esp32_fly/README.md?plain=1#L53-L54
6. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/firmware/esp32_fly/README.md?plain=1#L3-L4
7. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/firmware/esp32_fly/README.md?plain=1#L44-L54
8. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/firmware/esp32_fly/README.md?plain=1#L31-L36
9. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/firmware/esp32_fly/README.md?plain=1#L27
10. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/firmware/esp32_fly/README.md?plain=1#L19-L24
11. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/README.md?plain=1#L73-L75
12. github.com/slvDev/esp32-ai/blob/52ebcbbd905270592c1d415efd1128d04d0fa365/firmware/esp32_fly/README.md?plain=1#L97-L99