San Diego – The results from Arc Institute’s inaugural Virtual Cell Challenge are in. The public competition, sponsored by Nvidia, 10x Genomics, and Ultima Genomics, declared not one, but two grand prizes, worth $100,000 each, to the machine learning models that “best” predicted how cells responded to genetic perturbations. The announcement closed out Neural Information Processing Systems (NeurIPS) conference week.
A newly established Generalist Prize was awarded to Altos Labs, the for-profit biotech company launched in January 2022 with $3 billion funding with the mission of restoring cell health and resilience through cell rejuvenation. First place went to BioMap Research, a company applying foundation models to an array of applications from therapeutic antibodies to industrial enzymes.
Over 5,000 people registered for the competition across 114 countries with more than 1,200 teams submitting results, and over 300 teams making final submissions.
Dave Burke, PhD, chief technology officer at Arc told GEN that the inaugural Challenge exceeded expectations across dimensions.
“What really stood out was the quality of scientific discourse and the enthusiasm for tackling the problem,” said Burke. “We observed machine learning (ML) specialists and computational biologists engage with the problem in complementary ways. ML teams excelled at systematic optimization while computational biologists brought deeper understanding of functional genomics.”
Most well-rounded model
The Challenge aims to establish standardized benchmarks to accelerate the development of virtual cell models that guide the development of drugs that shift cell states from “diseased” to “healthy” with fewer off-target effects.
The initiative follows in the footsteps of the Critical Assessment of protein Structure Prediction (CASP) competition. This was the biannual experiment founded in 1994, which established Nobel Prize winning algorithm, AlphaFold, as the state-of-the-art model for the structural biology field.
While protein structure prediction is a defined problem, virtual cell models must account for biological complexity, such as the cell type, genetic background, and cell context, posing a challenging feat for the establishment of benchmarks that can distinguish between technical proficiency and biological relevance.
The Challenge’s original scoring system was based on three metrics: 1) PDS (Perturbation Discrimination Score), which measures whether predicted perturbation effects remain distinguishable from each other 2) DES (Differential Expression Score), which evaluates whether models identify the correct set of upregulated and downregulated genes, and 3) MAE (Mean Absolute Error), which quantifies gene-level prediction accuracy across the entire transcriptome.
Hani Goodarzi, PhD, core investigator at Arc explained that the Generalist Prize emerged mid-competition when competition organizers noted that MAE was no longer influencing optimization. As a result, many top performers were strategically focused their efforts on PDS and DES to raise their score without accounting for all metrics.
Rather than alter the scoring framework, the Challenge introduced an additional category using seven metrics total: the three challenge metrics plus four from Arc’s STATE model. The Generalist Prize winner achieved the highest average ranking across metrics.
“This means that the winners from Team Altos Labs presented the most well-rounded model rather than just the highest leaderboard score,” Goodarzi told GEN.
Biology mission
The Altos team developed a flow matching generative model to simulate distributions of perturbation cell response, which contrasts with traditional approaches which rely on response averages. The winning model was ranked using Altos’ PerturBench benchmark and enabled the team to capture fine-grained gene expression details, such as complex gene-gene interactions. The model was also pre-trained on approximately seven million high-quality single cells combined from various public data sources, including the Challenge training data, and Altos Labs’ internal perturbation screens.
Rory Stark, PhD, senior director of AI & ML at Altos, said the team was motivated to enter the competition to get empirical feedback on how their modeling efforts compared to the state-of-the-art from the community. The primary goal was not to win, but rather advance Altos’ mission of restoring cell resilience and health.
“We wanted to do well across all metrics because it was more important to have a model that reflects all aspects of biology,” said Stark in an interview with GEN. “We’re pleased to be rewarded for making a hard decision that we thought would cost us the competition.”
Marcel Nassar, PhD, senior machine learning scientist at Altos and research lead for the company’s winning model, says the Generalist Prize highlights one of the most challenging problems of the field: how to design and quantify metrics that capture biological relevance.
“In addition to training great models, evaluation is at the forefront of perturbation modeling and the virtual cell,” Nassar told GEN. “Arc did a great service to this field with the Challenge. Not seeing the data is as fair of an evaluation as it gets.”
Goodarzi adds that it was interesting to see that pure end-to-end neural networks were yet to outperform hybrid models. “This tells us we’re not yet at a point where pure scaling in size can reliably lead to improved outcomes. We need more thoughtful integration of biological priors,” he said.
He also highlighted that top teams successfully incorporated multi-modal features, such as protein embeddings alongside gene expression data, with strategic loss function design proved to be as important as architecture choices.
Arc plans to run the Challenge as an annual competition. Additionally, teams have published analyses examining the competition’s evaluation metrics, which will directly inform next year’s Challenge.
The field looks forward to the next virtual cell leaderboard.