·
Researchers & Thinkers
Tri Dao
Assistant Professor & Chief Scientist |Princeton University / Together AI
Creator of FlashAttention and co-creator of Mamba. Stanford PhD in hardware-aware ML algorithms. His IO-aware attention kernel is now used in virtually every Transformer and enabled context lengths to scale from 4K to 1M tokens.
6papers
GitHubtridaoResearch Papers
6 papersarXiv
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
2205.14135·2022-05-27
DOI
Monarch: Expressive Structured Matrices for Efficient and Accurate Training
10.48550/arXiv.2204.00595·2022-07-17
arXiv
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
2307.08691·2023-07-17
arXiv
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
2312.00752·2023-12-01
arXiv
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
2405.21060·2024-05-31
arXiv
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
2407.08608·2024-07-11
Biography
Deep Research
Tri Dao is an Assistant Professor of Computer Science at Princeton University and Co-founder & Chief Scientist of Together AI. He earned his PhD in Computer Science from Stanford University, co-advised by Christopher Re and Stefano Ermon, with a dissertation on hardware-aware algorithms for efficient machine learning. His research sits at the intersection of machine learning and systems, with a focus on hardware-aware algorithms and sequence models with long-range memory. He is the creator of FlashAttention, the IO-aware exact attention algorithm now used in virtually every Transformer-based model, and co-creator (with Albert Gu) of Mamba, the selective state space model that achieves linear-time sequence modeling with 5x higher throughput than Transformers. He joined Together AI as Chief Scientist in July 2023 and Princeton as Assistant Professor in September 2024. His work has received the ICML 2022 Outstanding Paper runner-up (Monarch), the COLM 2024 Outstanding Paper (Mamba), the MLSys 2025 Outstanding Paper Honorable Mention (Marconi), and the Schmidt Sciences AI2050 Early Career Fellowship.
Hardware-Aware AlgorithmsFlashAttentionState Space ModelsEfficient Training & InferenceIO-Aware ComputingSequence ModelingStructured MatricesGPU Kernel OptimizationLong-Context LLMsOpen Source AI
Timeline
12 Research12 total
2025
2024
2024-02Research
Appeared on Generally Intelligent podcast: FlashAttention, sparsity, quantization & efficient inference2024-05Research
Published Mamba-2: Structured State Space Duality — faster than Mamba-1 with tensor core support (ICML 2024)Key Contributions
FlashAttention
IO-aware exact attention algorithm that reduces memory from quadratic to linear in sequence length and runs 2-4x faster than standard attention. Now used in virtually every Transformer-based model and enabled context lengths to grow from 2-4K to 128K-1M tokens.
Mamba (Selective State Space Model)
Linear-time sequence model co-created with Albert Gu that achieves 5x higher throughput than Transformers by letting SSM parameters be functions of the input, with a hardware-aware parallel scan algorithm. 17.5K+ GitHub stars.
Mamba-2 (Structured State Space Duality)
Theoretical framework proving SSMs and attention are dual views of the same model, enabling tensor-core-optimized SSM computation that is 2-8x faster than Mamba-1's parallel scan.
FlashAttention-3
Third-generation attention kernel exploiting H100 asynchrony and FP8 low-precision to achieve up to 740 TFLOPS, 1.5-2x faster than FlashAttention-2 on Hopper GPUs.
Monarch Matrices
Class of hardware-efficient structured matrices (products of block-diagonal matrices) enabling 2x training speedup for ViT and GPT-2 with comparable quality. ICML 2022 Outstanding Paper runner-up.
Dao-AILab Open Source Ecosystem
GitHub organization hosting flash-attention (22.8K stars), causal-conv1d, fast-hadamard-transform, and other high-performance CUDA kernels widely adopted across the ML community.
Notable Quotes
“
Try to understand both the algorithm and the systems that these algorithms run on.
“
I'm excited to announce that I'm joining Together AI as Chief Scientist, with the goal of making open source AI more accessible and cost-competitive.
“
The memory is linear in sequence length. In terms of computation, it's still quadratic, but we managed to make it much more hardware friendly.
“
I think my prior is that as long as your model architecture is reasonable and is hardware efficient, and you have lots of compute, and you have lots of data, the model would just do well.
14 sources(click to expand)
Tri Dao — personal websiteTri Dao — Princeton CS profileAI2050 Fellow: Tri Dao — Schmidt SciencesTogether AI: Introducing Chief Scientist Tri DaoFlashAttention: Fast and Memory-Efficient Exact Attention (arXiv)FlashAttention-2: Faster Attention with Better Parallelism (arXiv)FlashAttention-3: Fast and Accurate Attention (arXiv)Mamba: Linear-Time Sequence Modeling (arXiv)Mamba-2 Blog Series — Structured State Space DualityMonarch: Expressive Structured Matrices (ICML 2022)Latent Space: FlashAttention 2 — 800% faster TransformersGenerally Intelligent: Tri Dao on FlashAttention & efficient inferenceTED Talk: How AI is Transforming Work and Everyday LifeDao-AILab — GitHub organization
Research generated March 19, 2026
Researchers & Thinkers/Tri Dao
All Profiles