21 articles tagged with #benchmarks.
An evidence-first review of genomic foundation models across clinical molecular diagnostic workflows. Compares DNABERT-2, Nucleotide Transformer, HyenaDNA, Caduceus, Evo 2, and AlphaGenome against established clinical tools across eight diagnostic tasks and post-calling variant effect prediction.
In its developers' comparisons, SpliceAI2 scored above SpliceAI, Pangolin and AlphaGenome on every benchmark printed, including cryptic splicing in Genomics England blood RNA-seq. Every value is developer-reported, most test variants lie in genes the model was trained on, no uncertainty is printed, and motif scoring, the baseline for both use cases, was not compared.
The best-supported RNA metagenomic result is 93.6% sensitivity for one UCSF assay, reported by its developers. A new Oxford nanopore validation found 51%. No study compares workflows prospectively against an adjudicated reference.
Eight published somatic SNV and indel caller comparisons, read side by side: rankings change between studies, indel accuracy is lower, and no study covers FFPE, ctDNA or panels.
Fragment-feature classifiers reach AUCs near 0.9 on low-pass plasma sequencing and, in one developer study, beat ichorCNA tumour fraction when ctDNA is scarce. Rankings change between cohorts, and no fragmentomic method has been tested in an intended-use population.
GPU and DRAGEN pipelines cut a whole-genome run from 17 to 39 hours on CPUs to between about 25 minutes and two and a half hours, and the one study that compared their calls with the CPU pipeline found close agreement. No study reruns a pipeline to test reproducibility, and FP8 gave a genomic foundation model at most 19.3% more throughput.
SpliceAI and Pangolin lead the DNA-based predictors on variants tested in patient RNA, and SpliceAI beats motif scoring on the same data, but no predictor replaces the RNA test. RNA-seq outlier callers recover between none and most known events and flag outliers in nearly every negative sample.
Seven published comparisons of DNA sequencing workflows for pathogen identification, from mock communities to patient specimens. None is an independent clinical validation of competing workflows.
In the one comparison of cfDNA features on the same participants, whole-genome methylation detected 34% of cancers at 98% specificity and located the tissue better than copy number or mutations. Higher figures come from smaller case-control cohorts. No machine-learning method has beaten the established baseline on the same data.
Short-read genome callers recover most deletions above 10 kb. Small deletions, duplications and exome data are where they fail, and read depth alone misses most 1 to 5 kb deletions. There is no reliable evidence that a machine-learning caller beats the read-depth baseline on the same data.
Short-read callers recover most validated driver fusions in haematologic cell lines but differ by orders of magnitude in review burden. Long-read callers score far lower in an independent benchmark. No comparison covers solid tumours or FFPE tissue.
Labelling cells in a new single-cell study often means training a method on cells that already carry labels. A recorded benchmark, not peer reviewed, trained six methods on one reference, ran two released models, and tested them on four blood studies kept out of training (32 donors). The benchmark could not establish a clear difference among four learned methods in agreement with author labels. At the threshold chosen on other studies to label 90% of cells, no method labelled 90% of the pooled test cells.
You have a spectrum, 250 possible molecules and a budget to check five. Trained alignment models found the true structure about six times in ten in this benchmark, but even a high score could not reliably tell when the answer was missing.
A model can put proteins in a useful order while barely improving its predictions. Six rhodopsin models show why error in nanometres matters alongside rank correlation.
If you can test 100 variants in a splicing reporter, which score should rank them? On 8,297 held-out MFASS variants, a simple trained baseline put 61 splice-disrupting variants in its first 100, and SpliceAI and Pangolin put 63 to 66. The difference was not established. Pangolin ranked the whole list better in exploratory analyses.
A small CNN cut prediction error by 28% against cheap features on unseen sequence families. For choosing the top 1% of sequences, the two were indistinguishable, and frozen UTR-LM embeddings added nothing.
In this exploratory study, with annotation, reference genome and variants matched, SpliceAI and Pangolin put 63 to 66 assay-confirmed splice disruptions in their top 100 of the same 8,297 held-out MFASS variants. Masking raised average precision slightly for both tools; no top-100 difference was established.
A recall reversal and two constant-prediction controls show why benchmark comparisons need populations, selection rules and reference scores.
The baseline, SpliceAI and Pangolin found 61, 64 and 65 disruptions in their top 100 MFASS variants. The differences do not establish a reliable winner.
Vision-language models can describe a scene in paragraph-length detail and still fail to tell you whether a red cube is in front of or behind a blue cylinder.
Understanding when you're working with an environment versus a benchmark changes how you design experiments, interpret results, and communicate findings. This guide covers the practical differences every RL practitioner should know.
