Transcript: Live coding session with Boris Cherny and Jarred Sumner Language: en Paragraphs: 69 RTFx: 495.4x (3.9s for 32:00.02 audio) Processing: 28.0s ──────────────────────────────────────────────────────────── ## Thesis The video demonstrates how integrating autonomous AI agents, specifically Claude Code, into a software development workflow can transform engineering productivity by automating the entire lifecycle of issue reproduction, code generation, testing, and review. By treating AI agents as active contributors—exemplified by the "RoboBun" bot in the Bun codebase—developers can shift their focus from manual debugging and coding to high-level verification and architectural decision-making, effectively creating a "closed-loop" system that continuously improves software quality. ## Key Points * **Automated Issue Resolution:** The Bun project utilizes "RoboBun," an AI agent that automatically reproduces GitHub issues, writes tests, and submits pull requests (PRs) without human intervention. * **Verification-First Workflow:** A critical requirement for this automation is that the AI cannot submit a PR unless it includes passing tests that verify the fix, ensuring that the code is functional before a human reviews it. * **Adversarial Code Review:** The team uses a combination of agents, such as CodeRabbit for stylistic checks and Claude Code for deep, context-aware bug detection, to simulate a rigorous peer-review process. * **The Importance of Documentation (ClaudeMD):** To maintain this system, the team relies on "ClaudeMD," a documentation file that instructs the AI on build commands, testing patterns, and common pitfalls, allowing the agents to learn from past mistakes. * **Scaling Autonomous Agents:** The speakers emphasize that modern model capabilities now allow for "auto mode," where agents can run for hours, handling complex tasks like implementing new features or optimizing libraries (e.g., image processing) while the developer focuses on other work. * **Shifting Bottlenecks:** As AI handles coding and testing, the primary engineering bottleneck has shifted from writing code to "deeper verification" and strategic planning—deciding what features to build and whether to merge AI-generated PRs. * **Enhanced CLI Experience:** The presenters highlight the "no-flicker" mode in the Claude Code CLI, which uses virtualized rendering to improve performance and usability, allowing for a more seamless interaction with the agent. * **Future Vision:** The ultimate goal is a fully autonomous loop where AI agents can handle everything from customer support tickets to feature implementation, with humans acting primarily as curators and final approvers of the AI's output. ## Chapters * [1:41](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=101s) — How Bun uses Claude Code to maintain Bun * [2:12](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=132s) — RoboBun reproduces every issue before a human looks at it * [2:47](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=167s) — The hard requirement: no PR without a passing test * [3:52](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=232s) — RoboBun is now a bigger contributor to Bun than Jarred is * [5:01](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=301s) — CodeRabbit for style, Claude Code review for subtle bugs * [8:19](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=499s) — Why CLAUDE.md is the thing that makes it maintainable * [10:23](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=623s) — Reading CI errors and closing the full loop * [12:59](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=779s) — Reviewing the PRs the agents just opened * [20:26](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=1226s) — Hill climbing: give it a metric and a way to verify * [22:21](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=1341s) — Auto mode, and why permission prompts killed the workflow * [25:30](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=1530s) — Where the bottleneck goes next: deeper verification, then planning * [27:33](https://www.youtube.com/watch?v=DlTCu_pNDHE&t=1653s) — When PRs become suggestions you can decline > Speaker labels are machine-assigned: who is talking comes from automatic > diarization and the spellings from Anthropic's published session roster. > They are not verified against the recording. Treat a NAME: as a strong hint, > not a citation. ════════════════════════════════════════════════════════════ TRANSCRIPT ════════════════════════════════════════════════════════════ [1:02] ANNOUNCER: Please welcome to the stage head of Claude Code of Anthropic, Boris Cherny, and creator of Bun at Anthropic, Jarred Sumner. JARRED SUMNER: Hello. All right. So this is a developer conference. We're going to be doing a little bit of talking, but mostly we're just going be coding. [1:34] So this is for the developers in the room. I'm going to start by talking a little bit about how BUN uses Claude Code to build and maintain BUN, and also kind of how our setup works. Because it's kind of a slightly more advanced setup than what's common today. But first, I'm going to get a few agents running to just fix some GitHub issues. BORIS CHERNY: This is classic Jarred doing work during a talk. [2:10] JARRED SUMNER: So in Bun's repo, every time somebody submits an issue, we have a Claude bot automatically run and try to reproduce the issue. So you can see this person has these side effects. And this is like one of the most recent issues. And we can see that Robobun, which is our bot, went and managed to reproduce the issue and submitted a PR automatically. This PR is like, has All these PRs always have tests. [2:47] It's one of the actual hard requirements before it can submit a PR. And so the challenge here is like, does this code look correct? And one of the things we do to check that is does the test fail in the previous version of BUN and pass in this debug branch? And the bot actually can't submit a PR without that being the case. BORIS CHERNY: And so this is, just to make sure I understand, so this is like every single issue that goes up in the BUN issue tracker, you have RoboBun automatically try to reproduce it before anyone looks at it. [3:22] JARRED SUMNER: Yeah. And this saves a lot of time because we have so many open GitHub issues. It really moves the challenge from just fixing and debugging the issue to is this the right thing to merge? Like is this the right fix? BORIS CHERNY: Mhmm. How how how good is it? Is it is this doing like a 100% of yours? Is it like 10%? JARRED SUMNER: We can go to the insights and go to contributors, and then if we go last three months, and this is specifically to main, we can see that RoboBun is now a bigger contributor to Bun than I am. And that's with merging not all of its PRs for sure. [4:02] You can see we have a lot of PRs open right now. The challenge is really how do we know can we merge the PR? And that's the test. And then the other thing that's really interesting about this is we have automatic code review bots that run, and then they're going back and forth. So like, CodeRabbit leaves a comment, and then RoboBun leaves a comment, And then they go back and forth and they CodeRabbit did the I love this. [4:43] And it also marks the comments as resolved when it's done. And you can see they actually went a lot. There's a lot of back and forth here. It was like 30 comments or something. BORIS CHERNY: And so you're using a combination of agents. So this is like code review, this is like Claude code review, and then also CodeRabbit, and you're using them together. JARRED SUMNER: Yeah. I think basically CodeRabbit is, is good for kind of stylistic issues and things that are like, make sure that it follows the CLAUDE.md. And then the Claude code review is really good at here's this really subtle edge case that would have taken me like thirty minutes of reading all the code and having all the context to figure out. So it's really good at surfacing bugs that need the full context to really understand. [5:31] And I think basically it's really hard to actually have all this automation without having code review that is in the loop with Claude there replying or replying is like very performative, but like fixing. And that's also a big part of like what used to take so much time when like why PRs would take so much time to merge is because you'd have to like check out the branch locally, fix a Lint error, then run the linter locally, then push it back up, and there's all this switching cost that's constantly there. So I think this is like an especially good use case for LLMs because otherwise it just takes up so much time to ship. BORIS CHERNY: And I guess especially for the BUN code base because it's systems code, very easy to repro an issue and then see if the issue is fixed because this is kind of back to what we were talking about before with this kind of verification loop. It's all systems code so it's really like a test case on a particular architecture and you can essentially like repo or verify anything. [6:41] JARRED SUMNER: Yeah. That's one thing that makes it easier in Bun's codebase is because it's a CLI tool, because we don't need to run a browser to test things, but you can also just like have something set up to like take a screenshot or record a video or those sorts of things. In Bun's case, don't need that, at least not yet. There's a couple of things we could do that for. Like we have some front end stuff that would be nice. But yeah, I think this is like the direction that I think is really interesting is because it saves so much time. And this is something that like, this is specifically for Bun, but the more generalizable thing when most products are not open source is like instead of an issue, maybe the starting point is like a customer support ticket. So like you could imagine automatically passing customer support tickets to a Claude bot to then go and try to reproduce the issue and then submit a PR and then having code review go back and forth. And that's where I think for a lot of companies it becomes a lot more impactful. Because it just saves so much developer time. [7:53] BORIS CHERNY: We should think of some kind of name for this pattern. It was like adversarial code review or something like that. JARRED SUMNER: Yeah, don't know. But I do think like there's also a few other things about this that's like, if you just do this then it doesn't quite work. The very first step you need is to make sure the development environment is set up. I think this has been talked about before, but like CLAUDE.md is very important because otherwise it's going to just submit PRs that don't quite make sense for you to merge. So like we very much emphasize in Bun's codebase that it runs this special command to do the build, and this both builds and runs the command, so it forwards the arguments. Because that's also one confusing thing is like, because BUN has to be compiled, you want to make sure that it's running the actual changes and not like a debug build that's like stale. We also go into a lot of detail about how to run tests, how to write tests, where to put the test, and a lot of like, here's all the issues that we've run into previously. Like basically the pattern here is like every time that you find yourself repeating something, it should probably go in CLAUDE.md. [9:14] Because the question now is like, how do you make it maintainable to have lots of Claude's running all the time? To do that, it needs to be written down. It needs to be documented. So, a really small detail is like, have it to make sure that Claude sees the error message. We make it print the error message before the less informative conditions. BORIS CHERNY: So this is sort of like you have Claude write a test and then the test is bad or something about it doesn't work. Then you see this repeated once or twice and then you just add it to the CLAUDE.md so that every time in the future when you write a test, you do it correctly the first time. Yeah. So this is like compound engineering. It's kind of this. [10:05] JARRED SUMNER: And then also it's helpful to give an overview of where all the folders are, like how the code is laid out, and like about dependencies. Another thing I think is interesting is making sure that it can read your CI errors and build logs. You want to set up the agent to be able to read the code, to do the full loop of writing the code, testing the code works, checking CI, monitoring CI, and reading all the errors so that way by the time it gets to a person, everything is like set up. Like the ideal is that you read the code and you have very clear indications that you can be high confidence to merge it, and the only way for that to be true is if it is set up for success. BORIS CHERNY: Interesting. [11:01] I remember when first met, you were talking about your vision of everyone being able to run hundreds of agents in parallel and how that would work. And I feel like I didn't really get it at the time, although now every night I'm running hundreds of agents every single night. And I feel like now I'm finally there, but this is a thing that you've been thinking about for a long time. So it feels like this is sort of the setup in order to be able to kind of scale up agents way more. Like you need this self verification so that agents can run autonomously. JARRED SUMNER: Yeah. Like, this has gone through many iterations in Bun's codebase. We previously just had a Discord bot where I could just mention the bot and it would spin up a container. It didn't have the CI stuff, it didn't have the code review stuff, and it's so much better now, especially with Opus 4.7. All this stuff is getting so much better. [11:58] Oh yeah, we can also check on how it's going. It looks like it created a PR, the first one. And it wrote tests. BORIS CHERNY: So maybe while we look at this, I'm curious for like just to get a show of hands for people in the room, like as people think about their development process, raise your hand if it looks something like this where you have like a bunch of terminal windows or desktop tabs and you're kind of pasting in issues. Okay. So, it's like maybe half of people. And then, what if it looks like RoboBun or something more like that where it's like closing the loop a little more? It's like the next level of abstraction. Starting to get there? Yeah. [12:37] I I think it's not surprising because I think model capabilities are just getting there. Like I think 4.7 is the first model where it's really felt like it's able to do this. And in the past, maybe you could do it with a bunch of scaffolding. Like you just throw a bunch of tokens at it and it can kind of work but now it's efficient enough. You can actually do this day to day. JARRED SUMNER: Yeah. Let's see. So the first PR is there. Let's see if it did any others. Okay. It did two PRs. This looks very plausible. BORIS CHERNY: That's cool. And this beforeafter, do you tell it to do that? JARRED SUMNER: Sometimes it does it. [13:22] It's pretty good about knowing when it should do that, like when it's like a string formatting thing. It also kept Bun style of the label, is good because Node's style is slightly different. Let's see, does this change look good? Yeah. Mostly what I'm thinking right now is it did this. [13:53] This is good because like you don't want to write one bite at a time. You want to write in chunks, and then it used saturation. But I don't like that. You shouldn't have to do that. BORIS CHERNY: Does anyone here actually know Zig? Sort of. Yes. That's what I feel looking at. That's good. JARRED SUMNER: And you can see it followed the patterns from the from the CLAUDE.md await using, and then that pattern of resolving all the promises at the same time. [14:28] BORIS CHERNY: And so what's your workflow? When you see something like this, are you usually going in and commenting or are you just going to wait for code review to come in and drop a comment? JARRED SUMNER: Usually it depends on how complicated it is. In this case, usually I'll wait for like this one is actually pretty simple. I feel pretty high confidence that if the tests pass then I would probably merge this. But I still would wait for the code review, at least for the Claude code review one to run just in case. Because what I really like about that is it will find things that aren't in the diff that are from tracing the control flow, which is what you want when it's like a human reviewing it. It's like somebody who has a lot of context who can think what are all the edge cases that this might run into. And the signal to noise ratio is pretty good. It's something maybe like 10% of the time it's wrong. [15:29] And for how that used to be with other code review products that we've tried, it was like basically you had to ignore most of what it said. BORIS CHERNY: That's pretty cool. How long has something like this worked? Is it like a latest model thing or have you had Robobun or this kind of automated repo, automated fixing, like this whole pipeline? How long has that been actually possible? JARRED SUMNER: We can probably see this in a chart somewhere. That's kind of a lot of commits, but I think that might not be on main. That might be the Rust thing. BORIS CHERNY: Heard Bun is going to be written in Ruststone. Is that JARRED SUMNER: I don't know. [16:13] I just have a Claude running and we'll see what happens. But you can see the volume of commits there is like kind of lower, and then it's definitely gotten up a lot, and then it's gone up a lot. Now really the bottleneck is like, do I feel good about merging this? Am I confident that its changes are correct? And that's new because it used to be like the code wasn't good enough. BORIS CHERNY: What do you think is left? What's it going to take before, like, there a missing tool or a missing model capability or model version or something for you to kind of feel RoboBun can fully close the loop? Issue comes in and then Fix goes out automatically. JARRED SUMNER: I think it needs a little bit more it takes a lot of time to verify the changes are correct. This was kind of already true, like when is a person pushing up a PR? [17:13] But I think the challenge is like how do we make sure to communicate sufficient proof that the changes are correct? Or making it easier to roll back things. I think those are kind of the two directions. But I think for the majority of simple issues, we should probably be pressing merge a lot more, and the bottleneck now is actually CI and making sure that having, fully running the code, making sure all the test stuff works. But I think it's basically there. [17:58] The large projects are still non trivial, but also I've been doing some pretty large PRs lately with Claude, mostly with not as much with RoboBun but with like Claude Code. Like, we recently added support for a built in image processing library to Bun. I could probably pull up the PR. And that was Claude. And also we did a bunch of follow-up PRs too. BORIS CHERNY: Yeah. It's interesting because I think when I look at different people using Claude code, everyone is at a different level of sophistication or adoption of this. And I think for me the hardest thing is the model changes very often so I have to constantly retune and recalibrate to what it can do. And as an engineer, it's hard because it's a very weird technology. It's the first technology I've used that's like that. [18:48] And I sort of feel like this is actually the way that you do it is ahead of how the Claude code team does it for Claude code itself. And to me, way the Claude code team does it is actually very automated, but this is even further ahead. This is almost full lift off, fully closed loop. JARRED SUMNER: In the last two weeks, we've added an HTTPthree server to Bun. There's a PR for an HTTP two server. There's fetch support for HTTP three and HTTP two. There's this image processing API. There's the ongoing Rust rewrite, which may not ship. That's like the most ambitious one I've done so far. And done is too strong of a word because it's very much not done. [19:32] BORIS CHERNY: And so even something like this, this is like a benchmark. So Claude ran this benchmark for you? JARRED SUMNER: Yeah, Claude ran this benchmark. I gave it like a this ran in like a separate, on like a Linux box. Yeah, I was like, make it faster than Sharp. And that's basically what I did. I gave it like a few ideas like, oh you could try like, read this code in Javascript core to like figure out how to avoid cloning the typed array when it's not strictly necessary. But like, it then went and did it and figured it out. And like, yeah, it's pretty crazy honestly because none of this would have worked several months ago. BORIS CHERNY: Yeah. [20:23] I feel like within Anthropic, within an AI lab, call this kind of thing hill climbing and this is this idea that if you give the model some sort of metric and then you give it a way to verify its result, you can just make it iterate and keep going and keep going until it hits that metric. And this is something like FourSeven I think is uniquely good at and I think it's something really underutilized because I think it's the first model that's actually very good at that. And if you just give it a target, you give it a way to improve the performance, and you give it a way to measure, it will just keep going until it's done if you let it go in auto mode. JARRED SUMNER: Yeah. And we can also see this is another case where the code review comments was really, really helpful because like in this PR there was like 100 comments or something, and it's just going and fixing everything. [21:17] Like it goes on for a while. BORIS CHERNY: In the meantime, you were just working on something else. JARRED SUMNER: Yeah. This was not like the thing I was 100% focused on. I was maybe like 10% focused on this. I was doing like five things at once. And this definitely wasn't possible like six months ago, three months ago. This is like very recent that this is doable. BORIS CHERNY: Okay. So how are sessions doing? JARRED SUMNER: Yeah, so we have one PR there. There's almost another PR coming up, looks like pretty soon. This one should be the trickiest one. It mostly looks good though, or like it looks plausible based on these changes. I wouldn't exactly do it this way, but I think we need a better, more optimized way to do this because that's a lot of checks. [22:14] BORIS CHERNY: And looking at your setup here, so using, you mostly use CLI? Yeah. And do you always use auto mode for permissions? JARRED SUMNER: Yeah, and then before that I used dangerously skip permissions. BORIS CHERNY: No, you guys, I can delete stuff if you do that. JARRED SUMNER: Think I'm not supposed to recommend that. But I think it's just not fun to wait for Claude to like press approve because then you just like go off and do something else and then it's just been sitting there. So that's why auto mode is really good because it's actually like a real way to fix that instead of just like trusting. BORIS CHERNY: And I also noticed like the input, like the little composer, it's stuck to the bottom of the screen so you're using no flicker mode. JARRED SUMNER: Yeah. [23:01] I'm using no flicker. Honestly, think we should just like make that the default because it's so much better. Like you can see I can scroll really fast and like you could scroll fast before but like sometimes there would be a flicker and now there's not. BORIS CHERNY: Have folks tried no flicker mode for CLI? Yeah. A few people? Yeah. So it's like we launched it on April Fools so you can think in hindsight it came across as a joke a little bit. But if you actually do Claude code no flicker equals one, Claude, so just like set that environment variable, We totally rewrote the renderer that's running in the CLI. So it's using virtualized scrolling, virtualized selection. [23:43] And so what this means is constant memory usage, constant CPU usage, and also some nice stuff. Like if Jarred types, he can actually click around the composer So you can actually click and mouse events work, which is pretty crazy for a terminal. JARRED SUMNER: So I'm just also having it monitor the PR. And you can see it ran some commands and then it's going to go to sleep for twenty minutes and wake back up. Twenty minutes is probably a little bit too long, but it's okay. [24:14] BORIS CHERNY: Is that like using a loop or something? JARRED SUMNER: I think so. Yeah, and then it's Let's see, how else is it doing? And then the other ones are still Apparently it fixed an extra bug as well. BORIS CHERNY: Okay. So, it's been like twenty minutes or so, twenty five minutes and we got one how many PRs have we gotten? Three PRs. Three PRs? That's not bad. JARRED SUMNER: Yeah. [24:48] And I think we'll get a fourth one once it finishes running the tests. BORIS CHERNY: And then in the meantime, RoboBun is still running and kind of like generating even more PRs. JARRED SUMNER: Yeah. Every time somebody submits an issue, it tries to reproduce it. BORIS CHERNY: Yeah. I kind of feel like every like the way Claude code makes you think is every time there's a new bottleneck, you have to kind of automate that bottleneck and then there's always some other bottleneck and you kind of move on to that. It started like writing code was the bottleneck and now it's no longer the bottleneck. Then verification and running tests that was the bottleneck and it's no longer the bottleneck. And now there's a deeper layer of verification. Maybe that's it. [25:30] What do you think are the bottlenecks remaining? JARRED SUMNER: It's definitely this deeper layer of verification. I feel like the bottleneck after that is going to be planning. Like, what should we do and what should we not do? And what is the right way to fix this? Ideally, Claude would be smart enough, or we could trust Claude enough to merge the PRs by itself. And I think in certain projects, you could probably do that and just have that be automatic completely. I think BUN is not yet there for Claude, or sorry, for BUN. But I think it would be really cool if we had the tooling for us to feel confident enough to do that. BORIS CHERNY: So, like right now Robobun, it doesn't build features. [26:18] Doesn't do feature requests yet. JARRED SUMNER: That's true, yeah. It doesn't do feature requests but we do also use it sometimes. So, we can also mention it in either Discord or Slack and it will try to implement the feature. So, sometimes when people are like, hey Bun is missing this thing, then I just mention the bot and maybe like an hour later there's a PR. A bunch of times somebody has like tweeted at me something like, can you fix this bug or whatever? And that's basically what I do. And then I reply with a link to the PR. BORIS CHERNY: Should we add a RoboBun account on Twitter? JARRED SUMNER: And I think like so it can do feature requests, but I'm hesitant for it to implement literally everything anybody asks for in a GitHub issue because that's kind of a lot. [27:13] BORIS CHERNY: Because in some ways it's kind of crazy to put something like an image processing library inside of Bun. but like we talk about engineering tastes and there's like an element of taste that goes into that. Like you felt like that's a good idea. Like, we're not sure yet if Claude is at the point where you would also think this is a good idea. But, you know, at some point in the future it will get there. Maybe it's starting to get there. JARRED SUMNER: And I do think that PRs become suggestions. Like, not merging PRs used to be you feel bad if you don't merge a coworker's PR because they put work into that. But you don't have to feel bad when it's Claude. So like, if the PR is wrong or for whatever reason, then you can just not merge it. But it does mean that bar for merge is like, should it be there? [28:12] Because I think there's also a difference when it's like people. Because with people you don't want people to feel bad about their lost work. So, sort of in some ways it does actually end up raising the bar for what you decide to merge. BORIS CHERNY: Yeah. It's interesting as the bottlenecks move, the dynamics change a little bit. It's sort of like having to trust each other, having to trust people on the team, this kind of changes a little bit. Now it's a little bit more about like do we have the right automation and do we trust automation as a group. Yeah. So I think we're almost at time. Is there any like maybe one last thing we want to show people where we can kind of check-in on kind of like the progress that we've made? [28:56] JARRED SUMNER: I don't think so. Yeah, it's still going one last onto that fourth PR and it's going back and forth like found a bug and then fixed a bug. It looks like it's about to submit the PR now. BORIS CHERNY: Okay. Let's wait for this one. Just love to have one more. This is like the cool thing about auto mode. It's like in auto mode, I can let like, Claude runs for hours and hours at a time. Like, run it almost every night. I'll just have a bunch of Claudes running in auto mode. [29:27] And I was like, before this, it just didn't work because it always got stuck at some kind of permission request. And that was crazy. Like, this entire thing was one prompt and that just ran for thirty minutes. JARRED SUMNER: Yeah. This is all I said. BORIS CHERNY: Okay. So it's pushing it. JARRED SUMNER: It's about to submit the PR. That sounds like the right fix too. This has been an issue that's been open for a long time. [30:04] BORIS CHERNY: Okay. And we got a PR. JARRED SUMNER: Yeah, we can go into this issue and we can see how many uploads it has. 20. Yeah. It's kind of a lot. BORIS CHERNY: Cool. Maybe we can pause there. But to me, this is just like such a cool vision of where engineering is going, I think, for everyone in this room. And, you know, we're gonna see this first. [30:36] We're gonna have to figure it out first and then everyone else is gonna have to figure this out. So, you know, like we were talking about this morning, just excited to be on this journey together. And like you can see, we haven't figured everything out yet but I think like the mode that we're in is just constantly experimenting, constantly trying to to see what the next bottleneck is so we can solve it. JARRED SUMNER: It's very exciting. This is so cool. [31:06] Cool. Alright. Should I like BORIS CHERNY: yeah. You can weave it.