Do Vision Model See Like the Brain? A Comparison Across EEG Encoding Model

Problem

This preprint addresses the gap in understanding how convolutional neural networks (CNNs) and vision transformers model human visual system responses, specifically through the lens of EEG encoding models. The authors aim to elucidate the differences in performance between these architectures when tasked with predicting EEG responses to visual stimuli.

Method

The study compares six CNN architectures and two vision transformers using data from ten participants who viewed 200 natural images. The evaluation metric employed is the Pearson correlation coefficient (r) between model predictions and actual EEG responses. The authors test various token representations, including the CLS token alone and the CLS token combined with all patch tokens. Controlled comparisons are made between MoCo-v1 and ResNet-50, which share a matched architecture, as well as between CLIP-RN50 and CLIP-ViT-B/32, which have a matched objective.

Results

The results indicate that CNNs exhibit the strongest correlation with EEG responses at the earliest layers, with performance diminishing at deeper layers. In contrast, vision transformers show peak correlations at the deepest blocks. Specifically, the peak correlations for the CLS token with all patch tokens are reported as follows: CLIP-ViT-B/32 achieves r=0.640, while DINOv2-ViT-B/14 reaches r=0.656. For pooled representations, the correlation coefficients range approximately from 0.48 to 0.51. The comparison between MoCo-v1 and ResNet-50 yields nearly identical results, with r=0.673 and r=0.670, respectively. Additionally, CLIP-RN50 and CLIP-ViT-B/32 diverge in performance until the preservation of patch tokens is considered.

Limitations

The authors do not report any limitations in their study. However, the absence of reported limitations may suggest a need for further exploration of potential confounding factors or the generalizability of the findings across different datasets or populations.

Why it matters

This work has significant implications for the development of models that more accurately reflect human visual processing. By comparing CNNs and vision transformers in the context of EEG responses, the findings could inform future research in neural encoding models and enhance the design of AI systems that mimic human cognitive functions. Understanding these differences may also lead to improved interpretability and performance in applications such as computer vision and human-computer interaction.