EEG-to-Image Generation

Overview

This project, carried out at A*STAR’s Institute for Infocomm Research, is a replication and enhancement of GWIT (Guess What I Think), a recent EEG-to-image model that conditions a frozen Stable Diffusion backbone with EEG-derived features through a ControlNet adapter. The published code did not reproduce the paper’s results, so the core contribution here was identifying and fixing the implementation bugs, recovering image quality and semantic accuracy on par with the original work, and establishing GWIT as a clean baseline for future research.

Background

Reconstructing what a person sees from brain activity is a central goal in brain-computer interface research, with applications across healthcare. EEG is attractive because it is non-invasive, portable, and low cost, but its signals are noisy and have a low signal-to-noise ratio, which makes image reconstruction hard. Many prior approaches leaned on older GANs or on intricate pipelines with multiple alignment losses and large pretraining; GWIT stands out for reaching comparable results with far less machinery.

Architecture

The pipeline replicates GWIT and has four main parts:

• EEG image decoder: a pretrained LSTM trained with triplet loss maps raw EEG into a 128-dimensional embedding. A KNN classifier on those embeddings predicts a text label, producing a coarse control prompt of the form “an image of [label]”.
• Projection network: a 1D convolutional network maps the raw EEG into the same latent space as the noisy image, providing fine-grained conditioning.
• ControlNet adapter: a trainable copy of the latent diffusion UNet encoder, fed the EEG-based control through a zero-initialized convolution so it does not disrupt the frozen backbone early in training.
• Stable Diffusion 2.1 base: a frozen latent diffusion model that generates the final image, decoded by a pretrained VAE.

Two evaluation settings were compared: guess mode, where generation relies on EEG alone, and no-guess mode, where the EEG-predicted text label is also supplied as coarse control.

Datasets

Experiments used the EEGCVPR40 and ThoughtViz datasets in the preprocessed forms released by the GWIT authors. EEGCVPR40 holds EEG from six participants viewing images across 40 ImageNet categories, recorded with a 128-channel system at 1 kHz, split 8:1:1 for train, validation, and test. ThoughtViz contributes a subset covering 10 visual classes, standardized following EEGStyleGAN-ADA.

Debugging and Evaluation

The original implementation produced images that often did not match their ground-truth class, despite EEG features being separable enough for a KNN classifier to reach high label accuracy. Isolating the decoder confirmed it was sound: pairing the KNN-predicted labels with Stable Diffusion (no ControlNet) yielded images that clearly matched the target class, which localized the fault to the ControlNet conditioning path.

Quality and semantics were measured with Inception Score (IS), Frechet Inception Distance (FID), LPIPS, and GA, a 50-way top-1 accuracy that checks whether generated images keep the correct class semantics.

Results

In no-guess mode, the bug fixes produced large gains across every metric (table below): semantic accuracy (GA) jumped roughly 47x, enough to match or slightly beat the original paper.

Guess mode, which uses EEG alone without the predicted label, remained hard and did not improve meaningfully, underscoring how much the coarse semantic label contributes to coherent generation.

Insights

Beyond replication, the project examined assumptions baked into current EEG-to-image work:

• Exact reconstruction is an unrealistic target: EEG reflects a subject’s brief, attention-dependent perception, so it often encodes only partial semantics of a scene. A better goal is to capture the perceived essence rather than reproduce the original image pixel for pixel.
• Single-label datasets cap what models can express: benchmarks like EEGCVPR40 and ThoughtViz assign one label per image, so a decoder that predicts a single label cannot represent multi-object scenes the brain may have perceived.
• Distribution metrics can mislead: because outputs are not meant to match the source exactly, FID and IS are imperfect here, and semantic measures such as GA align better with the real objective.

These point toward richer multi-label semantic representations from EEG, more efficient generative backbones, and evaluation centered on semantic correctness.

Key Technologies

• GWIT, ControlNet, Stable Diffusion 2.1: streamlined EEG-conditioned latent diffusion pipeline
• LSTM EEG decoder with KNN labeling: coarse semantic control from raw EEG
• PyTorch: training and evaluation of the ControlNet adapter and projection network
• IS, FID, LPIPS, GA: image-quality and semantic-accuracy metrics

References

1. E. Lopez, L. Sigillo, F. Colonnese, M. Panella, and D. Comminiello. “Guess What I Think: Streamlined EEG-to-image Generation with Latent Diffusion Models.” ICASSP 2025. arXiv:2410.02780.