Research
My work at Princeton University's Language and Intelligence initiative focuses on the use of AI for software engineering. The SWE-agent project has been trailblazing in being the first open source system to demonstrate how modern language models can effectively utilize tools to fix complex repository-level tasks (as measured on the SWE-bench benchmark). SWE-agent EnIGMA showed that the same system with different tools can set state of the art performance for red-teaming cybersecurity application.
Our most recent project, SWE-smith, is a novel pipeline for generating software engineering training data at scale and allowed us to set state of the art performance for open source models on the SWE-bench Verified benchmark.
I also continue to support research into Graph Neural Networks for High Energy Physics, research I started as a Postdoc at Princeton University.
During my PhD and studies, I focused on various aspects of data analysis, software engineering, maths and physics, including calibrating machine learning algorithms, clustering analyses, integration testing, anomaly detection, differential equations, data acquisition simulations, supersymmetry, and elliptic functions.
Recent News
SWE-agent: Autonomous Software Engineering
SWE-agent enables your language model of choice to use tools to fix issues in real GitHub repositories, find cybersecurity vulnerabilities, or perform any custom task.
It was the first open-source system to significantly score on SWE-bench, far outperforming RAG baselines and creating a breakthrough for agentic AI in software engineering.
Released just days after the commercial equivalent project Devin showed its first public demo, we demonstrated that a simple open-source system with optimized agent tooling could perform similarly (if not better) than a well-funded company's demo, democratizing access to AI-powered software engineering capabilities.
The central innovation discussed in our paper is the design and optimization of the agent-computer interface (ACI) that allows the language model to effectively navigate, understand, and modify large codebases. This includes custom shell commands, file editing interfaces, and feedback mechanisms.
Since the initial release in April 2024, development has never stopped, and SWE-agent still regularly ranks at the top of the SWE-bench leaderboard while maintaining a lightweight, modular architecture that makes it easy to extend and customize for different use cases.
SWE-smith: Scaling Data for Software Engineering Agents
Despite recent progress in Language Models for software engineering, collecting training data remains a significant pain point. The procedures to curate such datasets are often complex, necessitating hundreds of hours of human labor; companion execution environments also take up several terabytes of storage, severely limiting their scalability and usability.
To address this pain point, we introduce SWE-smith, a novel pipeline for generating software engineering training data at scale. Given any Python codebase, SWE-smith constructs a corresponding execution environment, then automatically synthesizes 100s to 1,000s of task instances that break existing tests in the codebase. Using SWE-smith, we create a dataset of 50k instances sourced from 128 GitHub repositories, an order of magnitude larger than all previous works.
We train SWE-agent-LM-32B, achieving 40.2% Pass@1 resolve rate on the SWE-bench Verified benchmark, outperforming GPT-4o and setting state of the art among open source models. We open source SWE-smith (collection procedure, task instances, trajectories, models) to lower the barrier of entry for research in LM systems for automated software engineering.
SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?
SWE-bench has become the industry-standard evaluation framework for benchmarking autonomous software engineering agents, utilized extensively by all major language model providers. It presents language models with real-world software engineering tasks drawn from GitHub repositories, challenging them to resolve complex issues requiring deep codebase understanding and reasoning beyond typical code generation.
While I joined the team after SWE-bench had already been published, I was involved with our follow up project, SWE-bench multimodal, in particular working on baselines with SWE-agent. SWE-bench multimodal generalizes the SWE-bench framework to typical frontend engineering tasks, shifting focus from Python to Javascript and requiring visual understanding in addition to reasoning and agentic abilities. Agents evaluated under this new benchmark must effectively interpret images provided within task descriptions and use visual feedback during issue resolution and validation processes. As a result, current state-of-the-art models continue to find this benchmark exceptionally difficult, successfully solving fewer than 25% of the included tasks.
Enigma: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
Proactively identifying and resolving cybersecurity vulnerabilities through red-teaming exercises where systems are tested from an attacker’s perspective is critical to securing modern infrastructure. However, attack vectors are diverse and require a broad set of skills, tools, and knowledge, making them very challenging to execute for AI systems.
We build on the generalist capabilities of SWE-agent to create EnIGMA, an AI agent equipped with various cybersecurity tooling. In particular, we enable the agent to use interactive terminal applications, including debuggers and real-time server interactions. Evaluated across leading Capture The Flag (CTF) cybersecurity benchmarks including NYU CTF, Intercode-CTF, and CyBench, EnIGMA sets new state-of-the-art standards, significantly outperforming existing approaches and marking a notable leap forward in agent-driven cybersecurity.
