·
Researchers & Thinkers
Nicholas Carlini
Research Scientist |Anthropic
Adversarial ML and LLM security researcher. Led the 16-Claude-agent C compiler experiment. Best paper awards at IEEE S&P, USENIX Security (2x), and ICML (3x).
7papers
GitHubcarliniResearch Papers
7 papersarXiv
Towards Evaluating the Robustness of Neural Networks
1608.04644·2017-05-22
arXiv
Obfuscated Gradients Give a False Sense of Security
1802.00420·2018-07-10
arXiv
Extracting Training Data from Large Language Models
2012.07805·2021-06-15
arXiv
Membership Inference Attacks From First Principles
2112.03570·2022-05-22
arXiv
Deduplicating Training Data Makes Language Models Better
2107.06499·2022-05-22
arXiv
Quantifying Memorization Across Neural Language Models
2202.07646·2023-05-01
arXiv
Stealing Part of a Production Language Model
2403.06634·2024-07-21
Biography
Deep Research
Nicholas Carlini is a Research Scientist at Anthropic (since March 2025), previously at Google DeepMind (2023-2025) and Google Brain (2018-2023). He holds a Ph.D. and B.A. in Computer Science and Mathematics from UC Berkeley (advisor: David Wagner). His research sits at the intersection of machine learning and computer security, focusing on adversarial robustness, training-data extraction, membership inference, model stealing, and LLM safety. His papers have received best paper awards at IEEE S&P (1), USENIX Security (2), and ICML (3), and have been covered by the New York Times, BBC, Nature, Science, Wired, and Popular Science. In February 2026 he led the landmark experiment where 16 parallel Claude agents autonomously built a 100,000-line Rust-based C compiler capable of compiling the Linux kernel. On GitHub (carlini, 1.9k followers, 42 public repos) his projects range from foundational adversarial-ML tooling (nn_robust_attacks, obfuscated-gradients) to creative hacks (printf-tac-toe, regex-chess) and LLM evaluation (yet-another-applied-llm-benchmark).
Adversarial Machine LearningLLM Security & SafetyTraining Data Extraction & PrivacyMembership Inference AttacksModel Stealing & ExtractionNeural Network RobustnessData Poisoning AttacksAI Alignment & Responsible AIMulti-Agent Software EngineeringCreative Programming & Code Golf
Timeline
17 Research17 total
2026
2025
2025-01Research
Keynote at CoLM: 'Are Large Language Models Worth It?' -- comprehensive cost-benefit analysis of LLM risks and benefits2025-01Research
Published 'Machines of Ruthless Efficiency' -- essay on near-term AI harms including surveillance, phishing, and job displacement2025-01Research
Appeared on Machine Learning Street Talk (MLST) discussing AI security, emergent LLM capabilities, and model-stealing research2024
2023
2022
2021
2020
2018
Key Contributions
Carlini & Wagner (C&W) Attack
Foundational adversarial robustness evaluation methodology for neural networks. The C&W attack became the standard benchmark for evaluating adversarial defenses. Best Student Paper at IEEE S&P 2017.
Obfuscated Gradients
Landmark ICML 2018 Best Paper showing that most adversarial example defenses at ICLR 2018 relied on gradient masking and could be circumvented. 909 stars on GitHub.
Training Data Extraction from LLMs
Series of papers demonstrating that language models leak individual training examples, including PII. Showed adversaries can extract gigabytes of training data from production models like GPT-2 and ChatGPT.
Stealing Part of a Production Language Model
ICML 2024 Best Paper demonstrating model-stealing attacks on production LLMs, recovering embedding layers from ChatGPT for under $20.
16 Claude Agents C Compiler
Led experiment where 16 parallel Claude agents autonomously wrote a 100,000-line Rust-based C compiler capable of compiling Linux 6.9, achieving 99% pass rate on GCC torture tests. Cost ~$20K over ~2,000 sessions.
yet-another-applied-llm-benchmark
Evaluation benchmark for language models on practical problem-solving tasks Carlini has previously encountered. 1,049 stars on GitHub.
deduplicate-text-datasets
Rust tool for text dataset deduplication, supporting the ACL 2022 paper showing near-duplicate removal reduces memorization by 10x. 1,300 stars (archived).
nn_robust_attacks
Reference implementation of robust evasion attacks against neural networks for finding adversarial examples. 860 stars on GitHub.
Notable Quotes
“
The science of security and flaws in machine learning should be open by default.
“
Ensuring that the benefits of LLMs outweigh the risks is literally your job.
“
It's worth being really careful and honest with yourself about whether or not what you're doing is actually net positive.
“
The LLMs that we're developing are already sufficiently advanced that we probably would have called them science fiction just five or ten years ago.
“
We are building new AI systems because they are ruthlessly efficient, and this efficiency directly enables harms previously constrained by human limitations.
“
Agent teams show the possibility of implementing entire, complex projects autonomously.
12 sources(click to expand)
Nicholas Carlini - Personal Websitecarlini (Nicholas Carlini) on GitHub (42 repos, 1.9k followers)Nicholas Carlini on Google ScholarCareer Update: Google DeepMind -> Anthropic (March 2025)Are Large Language Models Worth It? (CoLM keynote, 2025)Machines of Ruthless Efficiency (2025 essay)Building a C Compiler with a Team of Parallel Claudes (Anthropic, Feb 2026)Papers by Nicholas CarliniMLST Podcast: Nicholas Carlini (Google DeepMind) - Jan 2025Security, Cryptography, Whatever: Cryptanalyzing LLMs with Nicholas CarliniNicholas Carlini on dblp (full publication list)Future of Life Institute: Defeating AI Defenses with Nicholas Carlini
Research generated March 19, 2026
Researchers & Thinkers/Nicholas Carlini
All Profiles