Transcript: Caching, harnesses, and advisors: Building on Claude at GitHub scale
Language: en
Paragraphs: 56
RTFx: 271.1x (5.8s for 26:15.04 audio)
Processing: 27.1s
────────────────────────────────────────────────────────────
## Thesis
Mario Rodriguez, Chief Product Officer of GitHub, outlines the engineering strategies GitHub uses to maintain high performance and developer productivity while scaling Copilot on Anthropic’s platform. He argues that achieving efficiency and intelligence at scale requires a rigorous, data-driven approach centered on three core pillars: optimizing prompt caching to minimize costs, implementing "adviser" and "critique" model architectures to balance performance with affordability, and establishing a methodical, outcome-focused lifecycle for integrating and testing new AI models.

## Key Points
* **Prioritizing Developer Flow:** GitHub’s product decisions are grounded in the mission to keep developers in "flow" while ensuring team velocity, efficiency, and trust as they scale their AI services.
* **The Importance of Prompt Caching:** Prompt caching is essential for cost-efficiency; GitHub targets cache hit rates above 94% by keeping system prompts static, avoiding dynamic content in prefixes, and ensuring cache affinity across multimodal interactions.
* **Data-Driven Decision Making:** Using dashboards to monitor cache hit rates and performance deltas between model versions (e.g., comparing Opus 4.6 and Opus 4.7) is critical for making informed engineering decisions before and after model launches.
* **The Adviser Strategy:** GitHub and Anthropic are collaborating on an "adviser" model, where a smaller, faster model (Haiku) acts as an executor and consults a more powerful model (Opus) only for complex tasks, providing high-level intelligence at a lower cost.
* **The "Rubber Duck" Critique:** The "Rubber Duck" experiment introduces a critique model at key stages—such as after drafting a plan, following complex implementation, and before running tests—to improve code quality and efficiency before final deployment.
* **Rigorous Model Integration:** Integrating new models involves a structured lifecycle that includes updating the multimodal harness, running offline benchmarks, and conducting extensive internal "dogfooding" to gather real-world online evaluation data.
* **Outcomes Over Activity:** When measuring success, teams should prioritize meaningful outcomes, such as code "survival rate," over simple activity metrics like "acceptance rate," to ensure the AI is providing genuine value rather than just generating code that is later deleted.

## Chapters

* [2:02](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=122s) — The outcomes GitHub optimises for: flow, velocity, trust
* [3:30](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=210s) — Three things: prompt caching, advisors, and taking new models
* [4:53](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=293s) — Reading the cache dashboard, and the deltas between models
* [6:24](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=384s) — Why GitHub runs above a 94% cache hit rate
* [8:19](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=499s) — No dynamic content in the prefix, and the UUID mistake
* [9:27](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=567s) — Cache affinity across a multimodal harness
* [10:17](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=617s) — Debunking the myth that long context costs more
* [14:02](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=842s) — Brad Abrams on the advisor strategy: Haiku executes, Opus advises
* [15:52](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=952s) — Live: Copilot with Haiku, with and without the advisor
* [17:58](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=1078s) — Rubber duck: inserting a critique after the plan, after implementation, before tests
* [20:22](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=1222s) — How GitHub takes a new model: harness, offline benchmarks, dogfooding
* [24:36](https://www.youtube.com/watch?v=y5TmF_6o6xk&t=1476s) — Measure outcomes, not activity: survival rate over acceptance rate

> Speaker labels are machine-assigned: who is talking comes from automatic
> diarization and the spellings from Anthropic's published session roster.
> They are not verified against the recording. Treat a NAME: as a strong hint,
> not a citation.

════════════════════════════════════════════════════════════
TRANSCRIPT
════════════════════════════════════════════════════════════

[0:58] ANNOUNCER: Please welcome to the stage chief product officer of GitHub, Mario Rodriguez. MARIO RODRIGUEZ: Hello. Hello, It's great to be back at Code with Claude. I love developer conferences. You know, they're a little bit messy.

They're vibrant. They're still dev first, not agent first, and this is where we took half fun. We're here to have fun and to learn together. Now you're here because you want to learn more about the platform, and I'm here to kind of share with you some of the top things that we end up doing to running Copilot and all of our inference on top of this platform. Now I wanna get started with the why.

[1:43] Like, at GitHub, we're very mission driven. You know, our vision is to kind of empower developers to provide the best tools out there to advance human progress. But when I talk to customers overall, I'm telling them something that they want to achieve. They want to achieve a set of outcomes. And for them, they wanna keep their people in flow.

They wanna keep developers in flow. They want their teams to gain velocity. They want teams to actually achieve more with the people that they have, and they wanna do this at scale. And to be able to do it at scale, you have to have efficiency and you also have to have trust. And I start with this because almost every single product decision that I make kind of is grounded on these pillars.

[2:32] Even when we actually talk about the platform and integrating with our cloud platform, I wanna make sure that I keep developers in flow. I wanna make sure that teams gain that velocity, and I wanna make sure that companies can do that with intelligence and trust. Now as I think about it forward, I was thinking really hard. Okay. Like, what do I wanna say here?

And if I was sitting there, what I would like to understand is, okay, how do you operate? You know, we do billions and billions, probably over a big enough time sense, trillions of messages into against the platform. So I wanted to kind of give you the best practices or maybe not even best practices because things change so often almost on a weekly basis, but kind of like the key learnings, the things that are at the bottom of almost all the decisions that we make to be able to integrate with the platform and achieve that scale and that efficiency that is necessary. So I'm gonna divide it into three things. Number one is prompt caching.

[3:35] Without that, you know, we're not dead, but oh my god. Like the amount of money that we will spend compared to what we do is incredible. Right? So just 1% efficiency on this means a lot to us. It's kind of like high frequency trading.

Just 1% efficiency means millions overall. So I just want to spend a little bit more into how we're doing that. The second thing is we're collaborating with Anthropic on capabilities to make sure that our customers are using the right inference, the right amount of intelligence at the right time. And then one of those is this adviser model, and we have two ways of looking at it, both through an adviser and a critic. So Brad and I will share a little bit more about that too.

[4:22] And then the third one is, okay, every time a new model drops, and if you're integrating with the platform, you know that this happens constantly, and you get a call at five, you know, AM on a Saturday. It's like, we're launching on Tuesday. So how do we actually go through that the right way? How do we make decisions like what is the default model to keep developers in flow? And how do we make then decisions to actually route to the right intelligence at the right time overall?

So I'll start with this, and this is a dashboard. This is not our dashboard, but if you are on top of the platform, I believe Anthropic released this last week. We got a little bit preview of it. And what allows you to understand is how are you doing, you know, like your cache hit ratios, how many messages you're sending against Messages API. Right?

[5:10] And I think this is great. This is the first step because without data, it's really, really hard to make decisions. Really, really hard. So if you haven't checked it out, please do so. Now the one that I wanna spend a little bit more time on, it's ours.

And this is just one is not the entirety of all of our dashboards that we have, but it's one where we take a look at deltas between models. In this case, I want you to look mainly at the left hand side. That's Opus 4.6, and then on the right is 4.7, and then the next column is the delta between them. So when a new model drops, and sometimes we get it in EAP, we go ahead and run a set of benchmarks against it. So think about terminal benchmark two and then or or one of the suite.

[5:59] We have our own ones as well. And then we try to decide, okay, how is the model performing? And then we ship it, and then we pay attention to that data again. And then after a period probably of thirty days, we're probably done or maybe sometimes even sooner, we're done with all of the optimizations for that model. Now you can see over here kind of the median cash tokens and at the end the cash rate.

For us to operate the service at scale, we need to run above 94 usually, 94, 95, 96%. If we operate at 70%, that means usually that we have a bug, believe it or not. Like we're doing something not right, and then we need to change the approach on how we're calling the model, how we're assembling the prompt, how we're doing the end to end to keep that developer inflow. So we pay attention a lot to that. The same thing if I wanna actually change the cache rate from I'm sorry.

[6:54] Change the default model from 4.6 to 4.7, I need to understand what's the cache rate difference and what it actually means to me. Again, we make billions and billions of calls overall. So just 1% in that, in this case, it's 1.3% means a lot to us. Now you also have to take into account that from an input perspective, your cash rate is only 10% of the cost. Right?

So it's a 10 x difference in there. So you're constantly invalidating that cash. That is not a very that's not a very good thing overall because you're gonna be paying 10 x more. And then we kind of go across it. In this case, we did a a baseline against 4.6 Sonnet and then also on Haiku.

[7:37] Now imagine it stays like this for a second, and there's a bunch of red as you can see in this screen overall. Then we have a decision to make. We have decision to make to try to figure out how to get all of those reds into green. And it's not luck. It's a lot of hard work.

And I wanna make sure you understand that. It's a lot of hard work to get it, you know, maybe from 50 to 70% is not because you use kind of easy hanging fruit, but from 70 to 80, from 80 to ninety, and ninety and above, there's a lot of hard work that goes into this and engineering. And there's three things that we take a look at. Number one is we make sure that there's no and these are lessons learned. Like, we have made these mistakes before, and that's why I wanted to share it with you.

[8:19] We put no dynamic content in the prefix. Right? Like you need to keep that prefix as static as possible. And as an example, at one moment we had UUIDs in the actual system prompt, and then we're getting constantly reset and then that was invalidating the entirety of the cash rate. So remember, system, then there's tools, then there's the conversation, and then there's the last message that you're sending.

That's kind of the hierarchy. So you wanna keep that system front as stable as humanly possible. No dynamic content on it. Then goes tools. We made a lot of mistakes in tools at times.

[8:59] If you're actually loading tools dynamically and you're changing that tools prefix, then all of a sudden everything, the entire conversation gets invalidated again. So you have to do work and you have to make sure you have a lot of regression tests because you're gonna be experimenting a lot with what? Skills and tools overall in your end to end. You have to have regression test to make sure that you're not affecting your tools. And then the third thing then overall is that we need to have that cash affinity.

And let me tell you, it's really hard when you're doing a multi hard a multimodal harness. So take for example Copilot. I you know, the customer can be calling, let's say, Opus, then calling a GPT model, and then going to an OSS model, and then coming back into Opus again. And then through all of that, I have to make sure that the next Opus call, that last one that happened, actually has the right cache affinity too. So we do a lot of work when it comes to our multimodal harness to make sure that that is guaranteed.

[10:01] So the next slide. So here's an example on something that we ran in also to debunk a myth. So one of the key things that I heard many times, actually, when I go to customers and they ask me about integrating with the platform is, hey. Is long context mean that it's more expensive? And the answer to us is no.

And this is a quick test that we ended up running. So there's a smaller context window, and you could see in there that my average compaction went up three times compared to a largest context window. This was something that we simulated. We kept the same model with the same context window. We just ended up doing more compaction on it and filling it at a rate.

[10:43] Now the key thing over here, if you remember the math of how many input and output tokens ends up happening is is five x. Right? So for Opus it would be $5.25 dollars as an example. In in that case, whenever you do compaction, you get 4,000 tokens of output, and then you have because you have to summarize the message, etcetera, etcetera. So whenever that happens, then you end up usually paying more because you end up com doing a lot more compaction, which means that your output tokens go through the roof, and then your cash rate also gets a little bit more invalidated because of that too.

[11:18] Well, significantly invalidated because of that too. So longer context windows does not mean you're spending more. In fact, what you have to understand is how compaction is being done and depending on the scenario, want to manage that for the user appropriately. So and that's good. Alright.

So at a high level then, instrument your cash risk rate, have a dashboard that actually shows it to you, invest the time to do it. They're already shipping one for you so at the very least use that one, but go and invest in deltas, go and invest in pre the model launch and then post the model launch as well. Now no other efficiency matters until you see that one again. Like, you wanna be driving that from 5070, nineties overall, and it will take a lot of hard work, a lot of engineering time. And then you wanna measure per surface.

[12:08] We have Versus Code, which is one of the the dashboards that we show you there, but we also have the Copilot CLI. We also have our coding agent in the cloud. We also have IntelliJ, and we have mobile too. So it's a lot of surfaces that you're gonna have to go and understand, either share the harness across them or then tweak individually. So you wanna have a very good kind of sense of we shipped something, we regressed or not.

We shipped something, we improved or not. Now another thing that I was telling you about is, okay, if you got prompt caching already done, what about making sure that the right intelligence hits the user at the right time? And for that, Anthropic and I Anthropic and us have been partnering on this adviser model. And what I want to do is bring Brad Adams from the Anthropic team to talk more about it. Go ahead.

[12:58] Brad. BRAD ABRAMS: Thank you, Mario. Oh. Thank you, Mario, for being here. It's so great to have a partner like GitHub Copilot.

We get the Copilot team gives such great feedback. We give them very little time to evaluate models before we launch and they give great, insightful feedback and it's true on our API features as well. In fact, if you are using the Claude platform, a lot of what you are seeing is thanks to the things Mario and his team are doing on Copilot. And it's actually one of those pieces of feedback that I want to talk about today. One of the pieces of feedback we got from the Copilot team is they really wanted Opus level intelligence but it's at Haiku-level prices.

[13:48] That sounds like a good deal, right? Opus intelligence. So I I don't get to write the deals but what I do get to do is build fun API features. So I want to talk about the adviser strategy. And really the insight on the adviser strategy comes from software development teams.

We all know if you take a junior engineer and you give them a mentor who's a senior engineer, that junior engineer gets a lot better because the senior engineer looks over their shoulders, they review code, they look at design docs together, the senior engineer can make the junior engineer a lot more productive without taking too much time from the senior engineer. And it turns out the same thing is true for models. You can take a junior model like Haiku, who we're listing here as the executor, and give it access to Opus and you can very conservatively use those Opus tokens. So in this beautiful diagram that Claude created for me, what you see is the executor, Haiku, is able to identify every shape as it comes in, no problem, except for one little weird shape. That weird shape, it's beyond what Haiku can do, so Haiku has a tool, it calls the advisor, it calls Opus.

[15:09] Opus then, because it's a bigger model, it does more reasoning and it knows that. And what we see from this in our evals is that we get close to Opus level intelligence at much lower prices because we're being very conservative about the tokens that Advisor actually sends. And because of the price difference between the two, it works out really well. And what I am excited about is an integration that we work with, with the Copilot team. So let's switch over to the demo machine.

[15:44] Yeah, let's switch over to the demo machine. And what you're seeing is GitHub Copilot. On the left hand side, it's just GitHub Copilot with Haiku and on the right hand side, you're seeing GitHub Copilot with Haiku plus the new advisor tool hooked up. So I'm gonna hit enter here to give this time to run and there's a little kind of brain teaser y problem here that it has to solve. It has to give them the exact same problem and we see Haiku has taken off whereas on the right hand side is consulting advisor so it's going a little bit slowly so I'm keeping my fingers crossed here that Haiku will, I mean Opus will come back and so on the left hand side, Haiku is just spinning away trying a bunch of things just like you would see a junior engineer who's very eager, trying lots of things but hasn't found it yet.

[16:40] On the right side, the Opus Advisor just returned and because it's Opus, it actually knows this bit of data and so it's able to bring that back into context And now we're done with Opus and everything's still in Haiku but because it got that little bit of hint, the right hand side has finished here. So we see the right hand side finished. The left is still trying to figure out which way is up here. So just that little bit of hint from Opus, very small cost, very small latency hit makes Haiku so much more powerful. So this is an experiment we are doing in the GitHub Copilot CLI that we'll release soon and looking forward to you playing with it.

[17:31] Thank you and welcome back, Mario. MARIO RODRIGUEZ: Thank you. Thank you. I love some Opus Intelligence at Haiku prices. That's kind of what we gotta do.

Okay. So what we show you is an adviser model. There's another algorithm that you could actually employ as well as you're tinkering more how to get the right call to the right intelligence. And we call you know, we're GitHub. We're a little bit weird on some of the things.

We call it rubber duck. And what rubber duck is really all about, and here's a demo I'm gonna show you in a second, It's really about inserting a critique at the right moments in time. So I have a feature. That feature, you know, it had an issue, and over here, I'm implementing it at the moment. And what I wanna highlight, and it's gonna come really soon, is the model now is asking for a critique, and you can see how it's asking for a critique from four point six and 4.5 Opus.

[18:27] Then it receives that critique, and then it's able to actually change the plan prior to implementation and then continue with that implementation the right way. And then we get that feature and we deploy it. So that that's very, very quick, but really what I wanted to highlight is a different model. It's a different model that acts more less as an adviser and more as a critique on, hey. Here's what I think you should do, and we insert this in three core places.

One of them is after drafting a plan. We have a lot of our users that do a plan first, and then after drafting that plan, they go in through execution. We have another one after a complex implementation. So think about it. I just finished all of this.

[19:11] Go and critique. It's kinda like a pre code review to to a degree, and we do it sometimes there because it ends up saving tokens then to wait all the way into an official code review as well. And then the other one is after writing test but before running them. And in certain places where your CI suite with test takes a significant amount of time, then you could see how that at the end gets you to that flow faster, and it keeps that developer in flow. So those are the three places that we're doing it right now.

[19:42] And from my end, what I really have seen work well is kind of at the plan phase. Like, a lot of these systems are getting really, really good at planning. And then if you catch it there, then you end up with the most gains post as well. Now Rubber Duck is already an experimental. So if you download the Copilot CLI and you enable that experiments, you're gonna see it there.

And you could in invoke it whenever you want. So you could just say, hey. Create a plan on this and consult with Rubber Duck." And then you get a quick critique across model families as well. Now let's continue moving. The third thing that we wanted to talk about is, okay, how do we, at scale, take new models?

[20:25] Like, what is the process that we do? You know, at the very beginning, it was messy for us. But since over the last kind of two and a half years, we've done a lot of work to make sure we're very methodical about how we take this model. So you can see Anthropic has a model to try. We onboard that model into what we call CAPI, which is our Copilot API.

And then we have an endpoint. And then from that endpoint, three things really need to happen. Number one, as you all know, you have to go ahead and work in the harness and work in your prompts. So for us, we have a multimodal harness. Right?

[20:59] So it's not just one model family. So we have to go over there and make sure that we update the system prompts. We partnered very, very closely with Anthropic on that. We have to then make sure the tool interfaces are right and optimized. We then have to and we don't do this anymore as much, but we have to tweak the agent loop a little bit, and then a lot more work goes into context management, into compaction, into making sure that we're hitting the right cache hit rates overall.

Then from there, we do two things. We run offline benchmarks, and then we also do internal dogfooding. Think of that as online benchmarks overall. So a lot of the Microsoft developers use it, lot of GitHub developers use it, and we have a set of other users that we kind of partner very closely with as well to give us feedback. So we have that offline, and then we have that internal dogfooding.

[21:51] Then from there, we share findings with Anthropic, And this is a presentation, you know, before we used to write a document. Now we write a document and kind of expand it in detail where we're like, this is everything that we're seeing. And then we go on a loop with them on changes that we need either at the API or even changes on some of those models as well on the checkpoints that we end up getting. And then from there, we kind of do another loop all the way onto the release of the model.

Now it's very important, two things. Like, yes, we do offline, and offline gives you an indication, but I would say it's not going to be at times reality. You learn a lot more from your online evals overall and the online experiments after launch than really the offline. Offline just kind of sets a base, and if that base is consistent, you kind of know what to expect out there, and it gives you a pretty good indication on that of what to expect. But all of the details do not get done through offline.

[22:50] So it takes us usually days or sometimes even weeks to tune everything of that model out through those online experiments. We do a lot of AB testing, and we're very, you know, methodical about that. And we do weekly reporting that we report up to Anthropic and also across the entirety of the teams that we have to make sure that we are tuning up the right model at the right time. So when it comes to optimizing the harness for us, there are kind of four things. One is there's the build prompt and context.

[23:21] There's calling the model as it's there, executing the tool, then appending the results and pull pull back and loop back. And what I was saying here, the places where I spend the most time with the team is usually in the execute tool and that that build from context. Tools are very important to us. The more tools you have, then the more confusion ends up happening, the more you have to tune. So hundreds and hundreds of tools is not a good thing.

So you wanna make sure you're tuning by surface and you're tuning for the exact scenarios with the right tools in that package. So we spend a lot of time in there and optimizing that model. And then the check permissions and then run. We end up, again, in that execution of the tool, you have to do the right things to pass the right context on it and then to read the right results overall. So whenever we wanna introduce a new tool as an example, we spend a lot of time making sure that we're optimizing the harness both at the model level and at the tool execution level.

[24:16] Okay. So what does this mean at a high level? There's two things. You have to go in and have both published benchmark, internal benchmark, but more importantly, dogfood with a and b and make sure that you have online eval setup. And make sure you have to you could trust them with Statsig as well.

Then the second thing is you wanna measure outcomes, not activity. When I talked about the tools like, you know, as an example, acceptance rate on a code line, that's okay. Survival rate is actually an even better metric overall because if you ended up accepting it and then you ended up deleting it after, well, that actually did not accomplish the outcome overall. So even though I could have an amazing acceptance rate, if the survival rate ends up being very low, that means that we did not do the right work. So it's very important if you're in product, and product really changes in the era of AI, that you're optimizing for the outcomes and not just the individual rows.

[25:12] Like I said, you end up then optimizing for a metric that then tags tanks the measurement of your entire product. Okay. So then with that, again, we walked through prompt caching, the adviser strategy, and then we talked a little bit about measurement and how we improve the harness. And then for us, I just wanna leave you and say thank you. Really excited again to be with you, and hopefully, you learned a couple of things today.