Filter:

Multi-session, multi-task neural decoding from distinct cell-types and brain regions

Neuroscience

Transformers

Mehdi Azabou⁺, Krystal Xuejing Pan⁺, Vinam Arora, Ian Knight, Eva L. Dyer^*, Blake Richards^*

⁺Contributed equally as co-first authors

In preparation

GraphFM: A Scalable Framework For Multi-Graph Pretraining

Geometric Deep Learning

Transformers

Divyansha Lachi, Mehdi Azabou, Vinam Arora, Eva L. Dyer

Preprint, 2024

Abstract: Graph neural networks are typically trained on individual datasets, often requiring highly specialized models and extensive hyperparameter tuning. This datasetspecific approach arises because each graph dataset often has unique node features and diverse connectivity structures, making it difficult to build a generalist model. To address these challenges, we introduce a scalable multi-graph multi-task pretraining approach specifically tailored for node classification tasks across diverse graph datasets from different domains. Our method, Graph Foundation Model (GraphFM), leverages a Perceiver-based encoder that employs learned latent tokens to compress domain-specific features into a common latent space. This approach enhances the model's ability to generalize across different graphs and allows for scaling across diverse data. We demonstrate the efficacy of our approach by training a model on 152 different graph datasets comprising over 7.4 million nodes and 189 million edges, establishing the first set of scaling laws for multi-graph pretraining on datasets spanning many domains (e.g., molecules, citation and product graphs). Our results show that pretraining on a diverse array of real and synthetic graphs improves the model's adaptability and stability, while performing competitively with state-of-the-art specialist models. This work illustrates that multi-graph pretraining can significantly reduce the burden imposed by the current graph training paradigm, unlocking new capabilities for the field of graph neural networks by creating a single generalist model that performs competitively across a wide range of datasets and tasks.

Paper arXiv

BibTex

@Misc{lachi2024graphfm,
Title={GraphFM: A Scalable Framework For Multi-Graph Pretraining}, 
Author={Divyansha Lachi And Mehdi Azabou And Vinam Arora And Eva Dyer},
Year={2024},
Eprint={2407.11907},
ArchivePrefix={ArXiv},
PrimaryClass={Cs.LG},
Url={Https://Arxiv.Org/Abs/2407.11907}, 
}

APA

Lachi, D., Azabou, M., Arora, V. & Dyer, E. L. (2024). GraphFM: A Scalable Framework For Multi-Graph Pretraining.  ArXiv 2407.11907

Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution

Neuroscience

Transformers

Yizi Zhang, Yanchen Wang, Donato Jimenez-Beneto, Zixuan Wang, Mehdi Azabou, Blake Richards, Olivier Winter, The International Brain Laboratory, Eva Dyer, Liam Paninski, Cole Hurwitz

Neural Information Processing Systems (NeurIPS), 2024

Abstract: Neuroscience research has made immense progress over the last decade, but our understanding of the brain remains fragmented and piecemeal: the dream of probing an arbitrary brain region and automatically reading out the information encoded in its neural activity remains out of reach. In this work, we build towards a first foundation model for neural spiking data that can solve a diverse set of tasks across multiple brain areas. We introduce a novel self-supervised modeling approach for population activity in which the model alternates between masking out and reconstructing neural activity across different time steps, neurons, and brain regions. To evaluate our approach, we design unsupervised and supervised prediction tasks using the International Brain Laboratory repeated site dataset, which is comprised of Neuropixels recordings targeting the same brain locations across 48 animals and experimental sessions. The prediction tasks include single-neuron and region-level activity prediction, forward prediction, and behavior decoding. We demonstrate that our multi-task-masking (MtM) approach significantly improves the performance of current state-of-the-art population models and enables multitask learning. We also show that by training on multiple animals, we can improve the generalization ability of the model to unseen animals, paving the way for a foundation model of the brain at single-cell, single-spike resolution. Project page and code: https://ibl-mtm.github.io/

Project page Paper Code Model arXiv

BibTex

@InProceedings{Zhang_2024_arXiv,
author    = {Zhang, Yizi and Wang, Yanchen and Benetó, Donato Jiménez and Wang, Zixuan and Azabou, Mehdi and Richards, Blake and Winter, Olivier and The International Brain Laboratory and Dyer, Eva and Paninski, Liam and Hurwitz, Cole},
title     = {Towards a “universal translator” for neural dynamics at single-cell, single-spike resolution},
booktitle = {arXiv},
month     = {July},
year      = {2024},
url       = {http://arxiv.org/abs/2407.14668}
}

APA

Zhang, Y., Wang, Y., Jimenez-Beneto, D., Wang, Z., Azabou, M., Richards, B., Winter, O., The International Brain Laboratory, Dyer, E., Paninski, L. & Hurwitz, C. (2024). Towards a “universal translator” for neural dynamics at single-cell, single-spike resolution ArXiv 2407.14668

Large-scale pretraining on neural data allows for transfer across subjects, tasks and species

Abstract

Neuroscience

Transformers

Mehdi Azabou, Vinam Arora, Patrick Mineault, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, Eva L. Dyer

Computational and Systems Neuroscience (COSYNE), 2024

Abstract: Current population-level models of neural data are built to operate on a single dataset, condition, or task at a time. This both limits the amount of training data we can build models on, and also makes it difficult to build unified models where it may be possible to compare across individuals and over distinct tasks. To address this gap, we introduce a novel approach for large-scale training over many diverse datasets spanning many sessions and tasks. Much like words or groups of letters are passed into transformers in natural language processing, our approach centers around a novel way to breakdown neural population activity into parts or “tokens” that can be processed by a large transformer model. Rather than binning data, we represent each individual spike as a token and associate these spike events with a learned unit-level embedding which can capture information about the identity of a neural unit or channel (in the case of multiunits). We combine this input tokenization method with an encoder- decoder architecture that compresses the input spikes into a latent space and learns interactions across spikes in time and across neurons. To demonstrate the flexibility and scalability of our multi-session training framework, we apply it to a wide range of motor, visual, and decision making tasks from electrophysiology datasets collected in monkeys, humans, and mice. In particular, we train three large models, the first being from 7 nonhuman primates (27,373 units) engaging in four different motor tasks across three laboratories, the second combining human and monkey motor cortical datasets (9,789 monkey units + 1,920 human units) in diverse tasks, and the third from 58 mice (99,180 units) engaged in visual tasks during recordings to multiple regions in the cortex, deep, and midbrain structures. In these cases, we demonstrate that our pretrained models can be rapidly adapted to new, unseen sessions, enabling few-shot performance across animals and species with minimal labels. Our results highlight the power of scale and demonstrate how pretraining on large datasets can benefit neural data analysis.

Project page

2023

A Unified, Scalable Framework for Neural Population Decoding

Neuroscience

Transformers

We present a novel transformer architecture based the PerceiverIO and trained on a large-scale dataset of neural recordings from nonhuman primates.

Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, Eva L. Dyer

Neural Information Processing Systems (NeurIPS), 2023

Abstract: Our ability to use deep learning approaches to decipher neural activity would likely benefit from greater scale, in terms of both the model size and the datasets. However, the integration of many neural recordings into one unified model is challenging, as each recording contains the activity of different neurons from different individual animals. In this paper, we introduce a training framework and architecture designed to model the population dynamics of neural activity across diverse, large-scale neural recordings. Our method first tokenizes individual spikes within the dataset to build an efficient representation of neural events that captures the fine temporal structure of neural activity. We then employ cross-attention and a PerceiverIO backbone to further construct a latent tokenization of neural population activities. Utilizing this architecture and training framework, we construct a large- scale multi-session model trained on large datasets from seven nonhuman primates, spanning over 158 different sessions of recording from over 27,373 neural units and over 100 hours of recordings. In a number of different tasks, we demonstrate that our pretrained model can be rapidly adapted to new, unseen sessions with unspecified neuron correspondence, enabling few-shot performance with minimal labels. This work presents a powerful new approach for building deep learning tools to analyze neural data and stakes out a clear path to training at scale for neural decoding models.

Project page Paper Poster arXiv OpenReview

BibTex

@inproceedings{azabou2023a,
    title={A Unified, Scalable Framework for Neural Population Decoding},
    author={Mehdi Azabou and Vinam Arora and Venkataramana Ganesh and Ximeng Mao and Santosh B Nachimuthu and Michael Jacob Mendelson and Blake Aaron Richards and Matthew G Perich and Guillaume Lajoie and Eva L Dyer},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
    year={2023},
    url={https://openreview.net/forum?id=sw2Y0sirtM}
}

Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis

Behavior

SSL

Optimal Transport

Mehdi Azabou, Michael Mendelson, Nauman Ahad, Maks Sorokin, Shantanu Thakoor, Carolina Urzay, Eva L. Dyer

Neural Information Processing Systems (NeurIPS), accepted as Spotlight (3% submissions), 2023

Project page Paper Poster arXiv OpenReview

Half-Hop: A graph upsampling approach for slowing down message passing

Geometric Deep Learning

Half-Hop is a plug-and-play augmentation that works with any message passing neural network. We show that it leads to improved performance on various supervised and self-supervised benchmarks.

SSL

Self-Supervised Learning

Half-Hop is used as an augmentation to generate more diverse views for methods like BGRL and GRACE.

Mehdi Azabou, Venkataramana Ganesh, Shantanu Thakoor, Chi-Heng Lin, Lakshmi Sathidevi, Ran Liu, Michal Valko, Petar Veličković, Eva L. Dyer

International Conference on Machine Learning (ICML), 2023

Paper Code Article PyTorch Geometric Talk Poster OpenReview

BibTex

@InProceedings{pmlr-v202-azabou23a,
  title = 	 {Half-Hop: A graph upsampling approach for slowing down message passing},
  author =       {Azabou, Mehdi and Ganesh, Venkataramana and Thakoor, Shantanu and Lin, Chi-Heng and Sathidevi, Lakshmi and Liu, Ran and Valko, Michal and Veli\v{c}kovi\'{c}, Petar and Dyer, Eva L},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {1341--1360},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v202/azabou23a/azabou23a.pdf},
  url = 	 {https://proceedings.mlr.press/v202/azabou23a.html},
}

APA

Azabou, M., Ganesh, V., Thakoor, S., Lin, C. H., Sathidevi, L., Liu, R., Valko, M., Veličković, P. & Dyer, E. L. (2023). Half-Hop: A graph upsampling approach for slowing down message passing.  Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research

Transcriptomic cell type structures in vivo neuronal activity across multiple time scales

Neuroscience

We reveal the existence of characteristic computational fingerprints of transcriptomic cell types, during neuronal activity across diverse contexts: drifting gratings, naturalistic movies.

Attention

LOLCAT uses soft-attention, which enables the model to aggregate trial information across different scales.

Aidan Schneider⁺, Mehdi Azabou⁺, Louis McDougall-Vigier, David Parks, Sahara Ensley, Kiran Bhaskaran-Nair, Tom Nowakowski, Eva L. Dyer^*, Keith B. Hengen^*

⁺Contributed equally as co-first authors

Cell Reports, Volume 42, Issue 4, April 2023

Abstract: Cell type is hypothesized to be a key determinant of the role of a neuron within a circuit. However, it is unknown whether a neuron’s transcriptomic type influences the timing of its activity in the intact brain. In other words, can transcriptomic cell type be extracted from the time series of a neuron’s activity? To address this question, we developed a new deep learning architecture that learns features of interevent intervals across multiple timescales (milliseconds to >30 min). We show that transcriptomic cell class information is robustly embedded in the timing of single neuron activity recorded in the intact brain of behaving animals (calcium imaging and extracellular electrophysiology), as well as in a bio-realistic model of visual cortex. In contrast, we were unable to reliably extract cell identity from summary measures of rate, variance, and interevent interval statistics. We applied our analyses to the question of whether transcriptomic subtypes of excitatory neurons represent functionally distinct classes. In the calcium imaging dataset, which contains a diverse set of excitatory Cre lines, we found that a subset of excitatory cell types are computationally distinguishable based upon their Cre lines, and that excitatory types can be classified with higher accuracy when considering their cortical layer and projection class. Here we address the fundamental question of whether a neuron, within a complex cortical network, embeds a fingerprint of its transcriptomic identity into its activity. Our results reveal robust computational fingerprints for transcriptomic types and classes across diverse contexts, defined over multiple timescales.

Paper Code Article Cell Reports

BibTex

@article{SCHNEIDER2023112318,
title = {Transcriptomic cell type structures in vivo neuronal activity across multiple timescales},
journal = {Cell Reports},
volume = {42},
number = {4},
pages = {112318},
year = {2023},
issn = {2211-1247},
doi = {https://doi.org/10.1016/j.celrep.2023.112318},
url = {https://www.sciencedirect.com/science/article/pii/S2211124723003297},
author = {Aidan Schneider and Mehdi Azabou and Louis McDougall-Vigier and David F. Parks and Sahara Ensley and Kiran Bhaskaran-Nair and Tomasz Nowakowski and Eva L. Dyer and Keith B. Hengen},
}

APA

Schneider, A., Azabou, M., McDougall-Vigier, L., Parks, D. B., Ensley, S., Bhaskaran-Nair, K., Nowakowski, T., Dyer, E. L. & Hengen, K. B. (2023). Transcriptomic cell type structures in vivo neuronal activity across multiple time scales. Cell Reports, Volume 42, Issue 4, 2023

BibTex

@inproceedings{azabou2023relax,
    title={Relax, it doesn{\textquoteright}t matter how you get there: A new self-supervised approach for multi-timescale behavior analysis},
    author={Mehdi Azabou and Michael Jacob Mendelson and Nauman Ahad and Maks Sorokin and Shantanu Thakoor and Carolina Urzay and Eva L Dyer},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
    year={2023},
    url={https://openreview.net/forum?id=RInTOCEL3l}
}

Learning signatures of decision making from many individuals playing the same game

Behavior

We study a large-scale behavioral dataset from 1,000 humans playing a 3-armed bandit task. By learning to predict future choices, we identify characteristic playing strategies and signatures of differences across individuals.

SSL

Self-Supervised Learning

To simultaneously extract both global and local variables from complex human behavior, our method combines a multi-scale temporal convolutional network with latent prediction tasks, where we encourage embeddings across the entire sequence, as well as subsets of the sequence, to be mapped to similar points in the latent space.

Michael Mendelson⁺, Mehdi Azabou⁺, Suma Jacob, Nicola Grissom, David P. Barrow, Becket Ebitz, Alexander Herman, Eva L. Dyer

⁺Contributed equally as co-first authors

2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, Maryland, April 2023

Abstract: Human behavior is incredibly complex and the factors that drive decision making—from instinct, to strategy, to biases between individuals—often vary over multiple timescales. In this paper, we design a predictive framework that learns representations to encode an individual's `behavioral style', i.e. long-term behavioral trends, while simultaneously predicting future actions and choices. The model explicitly separates representations into three latent spaces: the recent past space, the short-term space, and the long-term space where we hope to capture individual differences. To simultaneously extract both global and local variables from complex human behavior, our method combines a multi-scale temporal convolutional network with latent prediction tasks, where we encourage embeddings across the entire sequence, as well as subsets of the sequence, to be mapped to similar points in the latent space. We develop and apply our method to a large-scale behavioral dataset from 1,000 humans playing a 3-armed bandit task, and analyze what our model's resulting embeddings reveal about the human decision making process. In addition to predicting future choices, we show that our model can learn rich representations of human behavior over multiple timescales and provide signatures of differences in individuals.

Paper IEEE Xplore arXiv

BibTex

@INPROCEEDINGS{10123846,
    author={Mendelson, Michael J. and Azabou, Mehdi and Jacob, Suma and Grissom, Nicola and Darrow, David and Ebitz, Becket and Herman, Alexander and Dyer, Eva L.},
    booktitle={2023 11th International IEEE/EMBS Conference on Neural Engineering (NER)}, 
    title={Learning signatures of decision making from many individuals playing the same game}, 
    year={2023},
    volume={},
    number={},
    pages={1-5},
    doi={10.1109/NER52421.2023.10123846}
}

APA

Mendelson, M., Azabou, M., Jacob, S., Grissom, N., Darrow, D., Ebitz, B., Herman, A. & Dyer, E. L. (2023). Learning signatures of decision making from many individuals playing the same game. 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, MD, USA, 2023, pp. 1-5, doi: 10.1109/NER52421.2023.10123846.

Detecting change points in neural population activity with contrastive metric learning

Neuroscience

We introduce an approach for identifying switches in neural population during free behavior, and find that different groups of neurons are useful for different contexts.

Carolina Urzay, Nauman Ahad, Mehdi Azabou, Aidan Schneider, Geethika Atamkuri, Keith B. Hengen, Eva L. Dyer

2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, Maryland, April 2023

Paper IEEE Xplore

BibTex

@INPROCEEDINGS{10123821,
    author={Urzay, Carolina and Ahad, Nauman and Azabou, Mehdi and Schneider, Aidan and Atamkuri, Geethika and Hengen, Keith B. and Dyer, Eva L.},
    booktitle={2023 11th International IEEE/EMBS Conference on Neural Engineering (NER)}, 
    title={Detecting change points in neural population activity with contrastive metric learning}, 
    year={2023},
    volume={},
    number={},
    pages={1-4},
    doi={10.1109/NER52421.2023.10123821}
}

APA

Urzay, C., Ahad, N., Azabou, M., Schneider, A., Atmakuri, G., Hengen, K. B., & Dyer, E. L., Detecting change points in neural population activity with contrastive metric learning, 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, MD, USA, 2023, pp. 1-4, doi: 10.1109/NER52421.2023.10123821.

2022

Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers

Neuroscience

Our novel transformer model performs competitively on behavior decoding tasks for spiking data, and shows promising potential for across-session and across-subject generalization.

Transformers

We present a novel transformer architecture that first operates on individual time-series and then captures the interactions between them at a later stage; this induces a permutation-invariance property and can be used to transfer across systems of different size and order.

Ran Liu, Mehdi Azabou, Max Dabagia, Jingyun Xiao, Eva L. Dyer

Neural Information Processing Systems (NeurIPS), 2022

Abstract: Complex time-varying systems are often studied by abstracting away from the dynamics of individual components to build a model of the population-level dynamics from the start. However, when building a population-level description, it can be easy to lose sight of each individual and how each contributes to the larger picture. In this paper, we present a novel transformer architecture for learning from time-varying data that builds descriptions of both the individual as well as the collective population dynamics. Rather than combining all of our data into our model at the onset, we develop a separable architecture that operates on individual time-series first before passing them forward; this induces a permutation-invariance property and can be used to transfer across systems of different size and order. After demonstrating that our model can be applied to successfully recover complex interactions and dynamics in many-body systems, we apply our approach to populations of neurons in the nervous system. On neural activity datasets, we show that our multi-scale transformer not only yields robust decoding performance, but also provides impressive performance in transfer. Our results show that it is possible to learn from neurons in one animal's brain and transfer the model on neurons in a different animal's brain, with interpretable neuron correspondence across sets and animals. This finding opens up a new path to decode from and represent large collections of neurons.

Project page Paper Code OpenReview arXiv

BibTex

@inproceedings{NEURIPS2022_1022661f,
    author = {Liu, Ran and Azabou, Mehdi and Dabagia, Max and Xiao, Jingyun and Dyer, Eva},
    booktitle = {Advances in Neural Information Processing Systems},
    editor = {S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh},
    pages = {2377--2391},
    publisher = {Curran Associates, Inc.},
    title = {Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers},
    url = {https://proceedings.neurips.cc/paper_files/paper/2022/file/1022661f3f43406065641f16ce25eafa-Paper-Conference.pdf},
    volume = {35},
    year = {2022}
}

APA

Liu, R., Azabou, M., Dabagia, M., Xiao, J., & Dyer, E. L. (2022). Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers. Advances in Neural Information Processing Systems, 35, 2377--2391.

Large-scale representation learning on graphs via bootstrapping

Geometric Deep Learning

We train BGRL on several established node-level classification benchmarks, and show competitive results compared to supervised and self-supervised methods, while achieving a 2-10x reduction in memory costs compared to GRACE, which enable us to scale to large graphs.

SSL

Self-Supervised Learning

When trained with supervised data alone, GNNs can easily overfit and may fail to generalize. Using DropEdge and DropFeat augmentations, BGRL learns unsupervised graph representations.

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L. Dyer, Remi Munos, Petar Veličković, Michal Valko

International Conference on Learning Representations (ICLR), 2022

Abstract: Self-supervised learning provides a promising path towards eliminating the need for costly label information in representation learning on graphs. However, to achieve state-of-the-art performance, methods often need large numbers of negative examples and rely on complex augmentations. This can be prohibitively expensive, especially for large graphs. To address these challenges, we introduce Bootstrapped Graph Latents (BGRL) - a graph representation learning method that learns by predicting alternative augmentations of the input. BGRL uses only simple augmentations and alleviates the need for contrasting with negative examples, and is thus scalable by design. BGRL outperforms or matches prior methods on several established benchmarks, while achieving a 2-10x reduction in memory costs. Furthermore, we show that BGRL can be scaled up to extremely large graphs with hundreds of millions of nodes in the semi-supervised regime - achieving state-of-the-art performance and improving over supervised baselines where representations are shaped only through label information. In particular, our solution centered on BGRL constituted one of the winning entries to the Open Graph Benchmark - Large Scale Challenge at KDD Cup 2021, on a graph orders of magnitudes larger than all previously available benchmarks, thus demonstrating the scalability and effectiveness of our approach.

Paper Code Slides OpenReview arXiv

BibTex

@inproceedings{
    thakoor2022largescale,
    title={Large-Scale Representation Learning on Graphs via Bootstrapping},
    author={Shantanu Thakoor and Corentin Tallec and Mohammad Gheshlaghi Azar and Mehdi Azabou and Eva L Dyer and Remi Munos and Petar Veli{\v{c}}kovi{\'c} and Michal Valko},
    booktitle={International Conference on Learning Representations},
    year={2022},
    url={https://openreview.net/forum?id=0UXT6PpRpW}
    }

APA

Thakoor, S., Tallec, C., Azar, M. G., Azabou, M., Dyer, E. L., Munos, R., Veli{\v{c}}kovi{\'c}, P., & Valko, M. (2021). Large-scale representation learning on graphs via bootstrapping. International Conference on Learning Representations.

MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction

Neuroimaging

We introduce a multi-task neuroimaging benchmark for micrometer-resolution X-ray microtomography imaging of a large thalamocortical section of mouse brain.

Vision

We benchmark a series of 2D and 3D image classfication and segmentation methods.

Jorge Quesada, Lakshmi Sathidevi, Ran Liu, Nauman Ahad, Joy M Jackson, Mehdi Azabou, Jingyun Xiao, Chris Liding, Carolina Urzay, William Gray-Roncal, Erik Christopher Johnson, Eva L. Dyer

Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022

Abstract: There are multiple scales of abstraction from which we can describe the same image, depending on whether we are focusing on fine-grained details or a more global attribute of the image. In brain mapping, learning to automatically parse images to build representations of both small-scale features (e.g., the presence of cells or blood vessels) and global properties of an image (e.g., source brain region) is a crucial and open challenge. However, most existing datasets and benchmarks for neuroanatomy consider only a single downstream task at a time. We introduce a new dataset, annotations, and multiple downstream tasks that provide diverse ways to readout information about brain structure and architecture from the same image. Our multi-task neuroimaging benchmark (MTNeuro) is built on volumetric, micrometer-resolution X-ray microtomography imaging of a large thalamocortical section of mouse brain, encompassing multiple cortical and subcortical regions, that reveals dense reconstructions of the underlying microstructure (i.e., cell bodies, vasculature, and axons). We generated a number of different prediction challenges and evaluated several supervised and self-supervised models for brain-region prediction and pixel-level semantic segmentation of microstructures. Our experiments not only highlight the rich heterogeneity of this dataset, but also provide insights into how self-supervised approaches can be used to learn representations that capture multiple attributes of a single image and perform well on a variety of downstream tasks. Datasets, code, and pre-trained baseline models are provided at: https://mtneuro.github.io/.

Website Paper Code OpenReview

BibTex

@inproceedings{NEURIPS2022_22fb65e3,
    author = {Quesada, Jorge and Sathidevi, Lakshmi and Liu, Ran and Ahad, Nauman and Jackson, Joy and Azabou, Mehdi and Xiao, Jingyun and Liding, Christopher and Jin, Matthew and Urzay, Carolina and Gray-Roncal, William and Johnson, Erik and Dyer, Eva},
    booktitle = {Advances in Neural Information Processing Systems},
    editor = {S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh},
    pages = {5299--5314},
    publisher = {Curran Associates, Inc.},
    title = {MTNeuro:  A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction},
    url = {https://proceedings.neurips.cc/paper_files/paper/2022/file/22fb65e39d318c4b5b56fbe9cb082e3f-Paper-Datasets_and_Benchmarks.pdf},
    volume = {35},
    year = {2022}
}

APA

Quesada, J., Sathidevi, L., Liu, R., Ahad, N., Jackson, J. M., Azabou, M., Xiao, J. and Liding, C. and Urzay, C. and Gray-Roncal, W. and Johnson, E. C. & Dyer, E. L. (2022). MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction. Advances in Neural Information Processing Systems, 35, 5299--5314.

Learning Behavior Representations Through Multi-Timescale Bootstrapping

Behavior

Natural behavior consists of dynamics that are both unpredictable, can switch suddenly, and unfold over many different timescales. We introduce BAMS which can capture multi-timescale factors that help in behavioral analysis. We apply BAMS on mouse and legged robots dataset.

SSL

Self-Supervised Learning

We introduce a multi-scale representation learning model: we combine a pooling module that aggregates features extracted over encoders with different temporal receptive fields, and design a set of latent objectives to bootstrap the representations in each respective space.

Mehdi Azabou, Michael Mendelson, Maks Sorokin, Shantanu Thakoor, Nauman Ahad, Carolina Urzay, Eva L Dyer

Conference on Vision and Pattern Recognition (CVPR), Workshop on Multi-Agent Behavior (Oral), 2022

Paper Poster Slides arXiv

BibTex

@article{azabou2022learning,
  title={Learning Behavior Representations Through Multi-Timescale Bootstrapping},
  author={Azabou, Mehdi and Mendelson, Michael and Sorokin, Maks and Thakoor, Shantanu and Ahad, Nauman and Urzay, Carolina and Dyer, Eva L},
  journal={arXiv preprint arXiv:2206.07041},
  year={2022}
}

APA

Azabou, M., Mendelson, M., Sorokin, M., Thakoor, S., Ahad, N., Urzay, C., & Dyer, E. L. (2022). Learning Behavior Representations Through Multi-Timescale Bootstrapping. arXiv preprint arXiv:2206.07041.

Detecting change points in neural population activity with contrastive metric learning

Abstract

Neuroscience

We introduce an approach for identifying switches in neural population during free behavior, and find that different groups of neurons are useful for different contexts.

Carolina Urzay, Nauman Ahad, Mehdi Azabou, Aidan Schneider, Geethika Atmakuri, Keith B. Hengen, Eva L. Dyer

Conference on Cognitive Computational Neuroscience (CCN), 2022

Paper CCN Proceedings Poster

BibTex

@misc{Urzay_Ahad_Azabou_Schneider_Atmakuri_Hengen_Dyer_2022,
  title={Detecting change points in neural population activity with contrastive metric learning},
  url={http://dx.doi.org/10.32470/CCN.2022.1261-0},
  DOI={10.32470/ccn.2022.1261-0},
  journal={2022 Conference on Cognitive Computational Neuroscience},
  publisher={Cognitive Computational Neuroscience},
  author={Urzay, Carolina and Ahad, Nauman and Azabou, Mehdi and Schneider, Aidan and Atmakuri, Geethika and Hengen, Keith B. and Dyer, Eva L.},
  year={2022}
  }

APA

Urzay, C., Ahad, N., Azabou, M., Schneider, A., Atmakuri, G., Hengen, K. B., & Dyer, E. L. (2022). Detecting change points in neural population activity with contrastive metric learning. In 2022 Conference on Cognitive Computational Neuroscience. https://doi.org/10.32470/ccn.2022.1261-0

2021

Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity

Neuroscience

Through evaluations on both synthetic data and neural recordings from hundreds of neurons in different primate brains, we show that it is possible to build representations that disentangle neural datasets along relevant latent dimensions linked to behavior.

SSL

Self-Supervised Learning

Views are created by dropping out neurons and jittering samples in time, which intuitively should lead the network to a representation that maintains both temporal consistency and invariance to the specific neurons used to represent the neural state.

Generative Modeling

Our approach combines a generative modeling framework with an instance-specific alignment loss that tries to maximize the representational similarity between transformed views of the input.

Ran Liu, Mehdi Azabou, Max Dabagia, Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith B. Hengen, Michal Valko, Eva L. Dyer

Neural Information Processing Systems (NeurIPS), accepted for Oral (1% submissions), 2021

Project page Paper Code Talk NeurIPS Proceedings OpenReview arXiv

BibTex

@inproceedings{NEURIPS2021_58182b82,
 author = {Liu, Ran and Azabou, Mehdi and Dabagia, Max and Lin, Chi-Heng and Gheshlaghi Azar, Mohammad and Hengen, Keith and Valko, Michal and Dyer, Eva},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {M. Ranzato and A. Beygelzimer and Y. Dauphin and P.S. Liang and J. Wortman Vaughan},
 pages = {10587--10599},
 publisher = {Curran Associates, Inc.},
 title = {Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity},
 url = {https://proceedings.neurips.cc/paper/2021/file/58182b82110146887c02dbd78719e3d5-Paper.pdf},
 volume = {34},
 year = {2021}
}

APA

Liu, R., Azabou, M., Dabagia, M., Lin, C. H., Gheshlaghi Azar, M., Hengen, K., Valko, M. & Dyer, E. (2021). Drop, swap, and generate: A self-supervised approach for generating neural activity. Advances in Neural Information Processing Systems, 34, 10587-10599.

Mine your own view: Self-supervised learning through across-sample prediction

SSL

Self-Supervised Learning

How can we introduce diverse views for learning? We propose the use of the dataset itself to find similar, yet distinct, samples to serve as views for one another.

Vision

We benchmark MYOW against other SOTA SSL methods on CIFAR-10(0) and Tiny Imagenet.

Neuroscience

When tested on multi-unit neural recordings in rodents (V1, CA1) and primates (MC), we find that MYOW outperforms other self-supervised approaches in all examples (in some cases by more than 10%), and often surpasses the supervised baseline.

Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C. Johnson, Kiran Bhaskaran-Nair, Max Dabagia, Bernardo Avila-Pires, Lindsey Kitchell, Keith B. Hengen, William Gray-Roncal, Michal Valko, Eva L. Dyer

Neural Information Processing Systems (NeurIPS), Workshop on Self-supervised Learning: Theory and Practice (Oral), Feb 2021

Abstract: State-of-the-art methods for self-supervised learning (SSL) build representations by maximizing the similarity between different transformed "views" of a sample. Without sufficient diversity in the transformations used to create views, however, it can be difficult to overcome nuisance variables in the data and build rich representations. This motivates the use of the dataset itself to find similar, yet distinct, samples to serve as views for one another. In this paper, we introduce Mine Your Own vieW (MYOW), a new approach for self-supervised learning that looks within the dataset to define diverse targets for prediction. The idea behind our approach is to actively mine views, finding samples that are neighbors in the representation space of the network, and then predict, from one sample's latent representation, the representation of a nearby sample. After showing the promise of MYOW on benchmarks used in computer vision, we highlight the power of this idea in a novel application in neuroscience where SSL has yet to be applied. When tested on multi-unit neural recordings, we find that MYOW outperforms other self-supervised approaches in all examples (in some cases by more than 10%), and often surpasses the supervised baseline. With MYOW, we show that it is possible to harness the diversity of the data to build rich views and leverage self-supervision in new domains where augmentations are limited or unknown.

Project page Paper Code Poster Talk arXiv

BibTex

@article{azabou2021mine,
  title={Mine your own view: Self-supervised learning through across-sample prediction},
  author={Azabou, Mehdi and Azar, Mohammad Gheshlaghi and Liu, Ran and Lin, Chi-Heng and Johnson, Erik C and Bhaskaran-Nair, Kiran and Dabagia, Max and Avila-Pires, Bernardo and Kitchell, Lindsey and Hengen, Keith B and Gray-Roncal, William and Valko, Michal and Dyer, Eva L},
  journal={arXiv preprint arXiv:2102.10106},
  year={2021}
  }

APA

Azabou, M., Azar, M. G., Liu, R., Lin, C. H., Johnson, E. C., Bhaskaran-Nair, K., Dabagia, M., Avila-Pires, B., Kitchell, L., Hengen, K. B., Gray-Roncal, W., Valko, M. & Dyer, E. L. (2021). Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprint arXiv:2102.10106.

Using self-supervision and augmentations to build insights into neural coding

Neuroscience

We investigate recent work in the application of self-supervised learning (SSL) for neuroscience, and discuss the types of data augmentations needed to develop the next brain-machine interfaces.

SSL

Self-Supervised Learning

Many self-supervised methods are based on learning invariances across different views of the data. The augmentations that are used help define what these invariances are. We ask what is required to apply SSL in the neuroscience domain.

Mehdi Azabou⁺, Max Dabagia⁺, Ran Liu⁺, Chi-Heng Lin, Keith B. Hengen, Eva L. Dyer

⁺Contributed equally as co-first authors

Neural Information Processing Systems (NeurIPS), Workshop on Self-supervised Learning: Theory and Practice, Feb 2021

Paper Poster

BibTex

@article{azabouusing,
  title={Using self-supervision and augmentations to build insights into neural coding},
  author={Azabou, Mehdi and Dabagia, Max and Liu, Ran and Lin, Chi-Heng and Hengen, Keith B and Dyer, Eva L},
  journal={NeurIPS 2021 Workshop on Self-supervised Learning: Theory and Practice},
  year = {2021}
}

APA

Azabou, M., Dabagia, M., Liu, R., Lin, C. H., Hengen, K. B. & Dyer, E. L. Using self-supervision and augmentations to build insights into neural coding.

Making transport more robust and interpretable by moving data through a small number of anchor points

Optimal Transport

By imposing a low rank structure to the transport plan, we show that our approach is more robust to outliers and in high-dimensional spaces compared to vanilla OT and other approaches.

Domain Adaptation

We apply LOT to different domain adaptation problems, and show how our approach is both effective and interpretable.

Chi-Heng Lin, Mehdi Azabou, Eva L. Dyer

International Conference on Machine Learning (ICML), 2021

Project page Paper Code Poster ICML Proceedings arXiv

BibTex

@InProceedings{lin2021,
  title = 	 {Making transport more robust and interpretable by moving data through a small number of anchor points},
  author =       {Lin, Chi-Heng and Azabou, Mehdi and Dyer, Eva},
  booktitle = 	 {Proceedings of the 38th International Conference on Machine Learning},
  pages = 	 {6631--6641},
  year = 	 {2021},
  editor = 	 {Meila, Marina and Zhang, Tong},
  volume = 	 {139},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {18--24 Jul},
  publisher =    {PMLR},
}

APA

Lin, C., Azabou, M. & Dyer, E.. (2021). Making transport more robust and interpretable by moving data through a small number of anchor points. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:6631-6641 Available from https://proceedings.mlr.press/v139/lin21a.html.

Multi-session, multi-task neural decoding from distinct cell-types and brain regions Neuroscience Transformers

GraphFM: A Scalable Framework For Multi-Graph Pretraining Geometric Deep Learning Transformers

Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution Neuroscience Transformers

Large-scale pretraining on neural data allows for transfer across subjects, tasks and species Abstract Neuroscience Transformers

2023

A Unified, Scalable Framework for Neural Population Decoding Neuroscience Transformers We present a novel transformer architecture based the PerceiverIO and trained on a large-scale dataset of neural recordings from nonhuman primates.

Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis Behavior SSL Optimal Transport

Detecting change points in neural population activity with contrastive metric learning Neuroscience We introduce an approach for identifying switches in neural population during free behavior, and find that different groups of neurons are useful for different contexts.

2022

Detecting change points in neural population activity with contrastive metric learning Abstract Neuroscience We introduce an approach for identifying switches in neural population during free behavior, and find that different groups of neurons are useful for different contexts.

2021

Multi-session, multi-task neural decoding from distinct cell-types and brain regions

Neuroscience

Transformers

GraphFM: A Scalable Framework For Multi-Graph Pretraining

Geometric Deep Learning

Transformers

Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution

Neuroscience

Transformers

Large-scale pretraining on neural data allows for transfer across subjects, tasks and species

Abstract

Neuroscience

Transformers

A Unified, Scalable Framework for Neural Population Decoding

Neuroscience

Transformers

We present a novel transformer architecture based the PerceiverIO and trained on a large-scale dataset of neural recordings from nonhuman primates.

Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis

Behavior

SSL

Optimal Transport

Detecting change points in neural population activity with contrastive metric learning

Neuroscience

We introduce an approach for identifying switches in neural population during free behavior, and find that different groups of neurons are useful for different contexts.

Detecting change points in neural population activity with contrastive metric learning

Abstract

Neuroscience

We introduce an approach for identifying switches in neural population during free behavior, and find that different groups of neurons are useful for different contexts.