InstaDeep Launches Nucleotide Transformer v3: A Multi-Species AI Model for Long-Range Genomic Analysis

Can artificial intelligence models trained on trillions of DNA base pairs unlock new insights into genomic regulation across diverse species, from humans to plants?

Revolutionizing Genomics Through AI-Driven Foundation Models

The introduction of Nucleotide Transformer v3 (NTv3) by InstaDeep represents a significant step in applying transformer-based architectures to genomics. This foundation model processes up to 1 million base pair (Mb) contexts at single-nucleotide resolution, enabling unified tasks such as representation learning, functional track prediction, genome annotation, and controllable sequence generation. By integrating self-supervised pretraining with supervised fine-tuning, NTv3 addresses the challenge of connecting local DNA motifs with broader regulatory contexts, potentially accelerating research in molecular biology and personalized medicine.

Model Architecture and Technical Specifications

NTv3 employs a U-Net-inspired architecture tailored for extended genomic sequences. It features a convolutional downsampling tower to compress input data, a central transformer stack to capture long-range dependencies, and a deconvolutional upsampling tower to restore base-level resolution for outputs. This design allows the model to handle sequences that are multiples of 128 tokens, using character-level tokenization over the nucleotides A, T, C, G, N, along with special tokens like , , , , , and . The model family spans a range of sizes to balance computational efficiency and performance:

• The smallest variant, NTv3 8M, contains approximately 7.69 million parameters, with a hidden dimension of 256, feed-forward network (FFN) dimension of 1,024, 2 transformer layers, 8 attention heads, and 7 downsampling stages.
• Larger models, such as NTv3 650M, scale up to 650 million parameters, featuring a hidden dimension of 1,536, FFN dimension of 6,144, 12 transformer layers, 24 attention heads, and species-specific conditioning layers for targeted predictions.

This architecture contrasts with prior models by maintaining single-nucleotide fidelity over megabase scales, which could reduce the need for specialized hardware in downstream applications. However, fixed context lengths in released checkpoints may limit flexibility for varying sequence sizes without additional padding.

Training Data, Performance Benchmarks, and Generative Applications

NTv3’s pretraining phase utilized 9 trillion base pairs from the OpenGenome2 dataset, covering over 128,000 species, through masked language modeling at base resolution. This was followed by post-training on approximately 16,000 functional tracks and annotation labels from 24 animal and plant species, incorporating a joint objective that blends continued self-supervision with supervised signals across about 10 assay types and 2,700 tissues. Performance evaluations demonstrate NTv3’s superiority on the newly introduced Ntv3 Benchmark, comprising 106 long-range, single-nucleotide resolution tasks across species and assays, using standardized 32 kilobase (kb) input windows. The model achieves state-of-the-art accuracy in functional track prediction and genome annotation, outperforming baselines like sequence-to-function models and earlier genomic foundation models on both public benchmarks and this suite. For instance, post-training enables coherent inference of regulatory grammar transferable between organisms, highlighting the value of multi-species exposure during training. Beyond prediction, NTv3 supports fine-tuning as a controllable generative model via masked diffusion language modeling. Conditioning on signals for desired enhancer activity and promoter selectivity allows it to infill masked DNA spans. In validation experiments, the model generated 1,000 enhancer sequences, which were tested in vitro using STARR-seq assays in collaboration with external labs. Results indicated recovery of intended activity level orderings and over twofold improvement in promoter specificity compared to baselines, suggesting practical utility in designing regulatory elements. | Dimension | NTv3 Key Features | Comparison to GENA-LM | |————————|———————————————————————————–|—————————————————————————————| | Primary Goal | Unified representation, prediction, and generation across species | Transfer learning for supervised genomic tasks | | Architecture | U-Net with convolutional towers and transformer; single-base tokenization | BERT/BigBird encoders with sparse attention and recurrent memory | | Parameter Scale | 8M to 650M parameters | 110M to 336M parameters | | Native Context Length | Up to 1 Mb at nucleotide resolution | Up to 45 kb (BERT) or 36 kb (BigBird) with BPE tokenization | | Pretraining Corpus | 9 trillion base pairs from >128,000 species | ~480 billion (human) to 1.07 trillion base pairs (multi-species) | | Supervised Signals | 16,000 tracks from 24 species, 10 assays, 2,700 tissues | Task-specific fine-tuning on promoters, enhancers, chromatin profiles | | Generative Capabilities| Masked diffusion for controllable design; >2x specificity in STARR-seq validation | Primarily predictive; supports masked infilling but not explicit design |

Broader Implications for AI in Genomics

The development of NTv3 underscores a trend toward scalable, multi-modal foundation models in bioinformatics, where pretraining on vast, diverse datasets enhances zero-shot transferability. With genomics data volumes projected to exceed exabytes by 2025, models like NTv3 could streamline annotation pipelines and reduce experimental costs, potentially impacting fields from crop engineering to rare disease diagnostics. The emphasis on controllable generation opens avenues for hypothesis-driven sequence design, though challenges remain in generalizing beyond the 24 supervised species and ensuring ethical handling of multi-species data.

• Statistical Edge: NTv3’s post-training yields SOTA results on 106 benchmark tasks, with cross-species transfer reducing the need for organism-specific retraining.
• Resource Efficiency: Smaller variants (e.g., 8M parameters) enable deployment on standard hardware, democratizing access for academic researchers.
• Validation Metrics: In enhancer design, generated sequences achieved >2x promoter specificity, validated through in vitro assays, indicating reliable predictive power.

This model’s open availability for checkpoints and inference tools may foster collaborative advancements, though uncertainties persist regarding scalability to even longer contexts or integration with emerging multi-omics data. How do you see foundation models like NTv3 shaping the future of genomic research and its applications in healthcare or agriculture?