Garrett Berntsen was last seen on ChinaTalk in his role as Deputy CDAO at the State Department. He then went to the Department of Defense for a little bit, and now makes a return in his new role as Chief AI Officer at Accenture Federal Services.
We’re going to talk about what the national security community needs — beyond just API access to wonderful models — in order to really change how it does its mission.
Our conversation covers:
Why the U-2 spy plane offers a surprisingly useful lesson for AI: The hardest part of adopting transformative technology isn’t building it, but redesigning the institutions, workflows, incentives, and decision-making systems around it.
The government’s need to manufacture its own “forcing functions”: How carrots, sticks, budget cuts, and a willingness to create winners and losers could help bureaucracies move faster.
Giving the troops the tools and getting out of the way: Why Berntsen thinks the U.S. military’s greatest AI advantage may come from empowering service members to experiment, improvise, and “figure it out.”
Will AI eventually negotiate with AI? We debate whether models can move beyond advising diplomats and executives to actually conducting negotiations.
Plus, cyber challenges, a chat app for government, and book recs.
Listen now on your favorite podcast app.
Thank you to Accenture Federal Services for bringing us this episode.
The U-2 and the Technology Behind the Photo
Jordan Schneider: All right — a U-2 flies over Cuba in 1962. Why is that a parable we should recall as we think about how to deploy AI in government?
Garrett Berntsen: Let me tell the CliffsNotes version of the U-2 story. It was built in a classified program with government buyers and government money, giving money to industry and unleashing a whole bunch of innovation around sensors and high-altitude flight. They built everything new. Putting all of the pieces of that together was incredibly innovative, and it was done in a secret lab, very quickly and famously.
Fast forward to the Cuban Missile Crisis. Suddenly we have overflight imagery of missile sites in Cuba. We’re getting really good raw intelligence in a way that the Russians didn’t know we were getting, and it was kind of miraculous to US decision-makers. They could actually see changes on the ground over time.
You hear about the technology, the plane — which, by the way, I was looking this up recently, is still in service. That’s incredible. It was that revolutionary, and it remains really valuable. They keep talking about decommissioning it and then decide not to. So the technology is of course incredibly important, as was the innovation required to put it all together.
The reason I like this metaphor, and why I think it’s relevant to AI today, is that it wasn’t just the technology, not just the plane. The way they bought it mattered. The innovations around acquisition reform mattered. The way industry and government worked together mattered. And then the way the information coming out of the plane was actually analyzed and institutionalized through intelligence analysis mattered. They built a whole new intelligence service, the NPIC — the National Photographic Interpretation Center — which eventually became NGA. They built that whole chain of judgment, and then, probably most importantly, put it in front of the president and other decision-makers. That’s the miraculous piece of the story.
Pivoting to the AI moment, we’re in a situation where the technology has moved much faster, because so much of it comes from the commercial world rather than being managed in a government program. It has moved so far ahead of our ability to integrate it into actual systems, workflows, and decision-making processes that we’re now flat-footed in the face of this technology. That’s a lot of what I work on at my company, and what I worked on in government — trying to think through the other pieces that need to change, not just the AI itself. That’s why APIs alone are not enough.
Jordan Schneider: Let’s spend a little more time on the U-2. What were the technological unlocks needed to take that photo and realize the missiles were arriving in Cuba? You needed new lenses, a new canister, what else?
Garrett Berntsen: High-altitude flight. At the time, no one had ever flown that high. Even the equipment on the pilots had to be developed.
Jordan Schneider: Could you even see it from the ground?
Garrett Berntsen: No.
Jordan Schneider: Right, because Eisenhower was flying these over the Soviet Union for a while before they even realized it.
Garrett Berntsen: All of those pieces mattered. The other incredible thing — and we have to remember this — is that doing things at pace and speed in an engineering way is actually very hard. We’re much more used to running a first test, then a second test, collecting data, and going back to refine it. I’m not saying that’s the wrong way to do it. That’s how you do engineering, especially if you’re going to put pilots into life-threatening environments. But the ability to move more quickly through that cycle and to take more risk is part of the magic of that technology.
Coming back to today, that’s why we’re seeing such a focus on experimentation — pushing experimentation as far as we can, with senior leaders saying, “I’m willing to accept some risk here because the mission is so important, and we need to take a little more risk to try these technologies out in operational environments.”
Jordan Schneider: Okay, so we have this photo, which is an incredible thing in itself. What does the intelligence apparatus do with it?
Garrett Berntsen: They take the canisters out of the cameras and blow the images up into huge printouts on blotter paper, then lay them out on the floor at NPIC in a classified space. People are literally pulling manuals and intelligence reports on Soviet weapons and looking through magnifying glasses, asking, “Is that the right paint on the side of this truck to make it a missile launcher? Is the shape of that missile consistent with what our intelligence reports say?”
You’re taking the raw intelligence and analyzing it with human judgment, with experts on military equipment, and blending it all together into an assessment. And you’re doing it fast. If you follow the Cuban Missile Crisis day by day, those were really difficult days.
Garrett Berntsen: These were NSC situation room decisions. What do we do? Do we attack, do we put up a blockade, what do we message back to the Soviets? Those decisions were happening very quickly, and you didn’t have a lot of time to sit back and do a deeper memo analysis.
Jordan Schneider: In that era, it was very clear — you had a few new pieces of data, these new photos, and you were going to do as much as you could with them. Today we’re in a completely different situation.
Garrett Berntsen: It’s almost the opposite. You almost have too much information.
Jordan Schneider: You can’t build the NPIC at the same time as designing the exact spy plane you want the NPIC to analyze. You just have this giant technological overhang, and it’s growing in directions you aren’t dictating.
Garrett Berntsen: Yes. You’re not in control, effectively.
Jordan Schneider: How are people processing this?
Garrett Berntsen: You want to give people their due for pushing. The current team in the Pentagon and elsewhere is working really hard at it — I’ve spent time at the State Department, the NSC, and the Department of War over the last five years or so. But there’s a lot to do, and we’re seeing pockets of success.
More of that success is in operational spaces. For example, in what they call CJADC2 — Combined Joint All-Domain Command and Control, there’s a lot of progress in giving people situational awareness of the entire battlefield and using AI to distill out and identify objects and intelligence, then bring it live and quickly to decision makers in the field.
Where we’re seeing less progress, candidly, is in what I’d call core business systems.
My personal belief is that we shouldn’t think of systems as mission systems and business systems. Increasingly, everything is a mission system. The department and all the agencies in the federal government are really starting to realize that if they want operational systems out at the edge supporting warfighters or decision makers in whatever mission, they need really, really good data from their core systems — business systems, logistics, personnel, finance. Those are warfighting functions too.
You have to invest in getting the data cleaned, organized, and ready for use at an operational tempo. A lot of these business systems aren’t built to run at operational tempo, because they’ve never had to before.
Here’s a very simple example. In some of these business systems, the rules literally say you only have to update the status on a form once a week. Well, that’s a policy — someone has written a policy that says supply sergeants in the Army, of which there are probably 10,000 or 20,000, update once a week. I was in logistics in the Army, so imagine the challenge of going to your supply sergeant and saying, “Hey, you actually have to input this into the system every day instead of once a week,” because some decision maker wants that data every day instead of once a week. That’s a big boat to turn around, and the value has to be there. A lot of times we’re also giving them systems that are really not easy to use. We’re making it hard for them to put the data in, and now we’re telling them the tempo and the latency of that data need to be a lot faster.
Jordan Schneider: I want to come back to the U-2. If you were sitting there in the 1940s imagining a U-2, the vision of what you wanted it to accomplish was relatively straightforward. Okay, we’re going to put cameras in the sky and take photos of things whenever we want.
But when you’re talking about modernizing the mission stuff, modernizing the logistics stuff, it’s not nearly as obvious what an agentic warfare bureaucracy actually looks like. Do you have a pitch you give people, or is it, we’re just going to feel our way into it?
Garrett Berntsen: It’s a bit of an unfair comparison, I’ll give you that much, because the U-2’s purpose is to do one thing. AI can do almost any piece of knowledge work, within the confines of these models — and the border of what’s possible is changing every day. It’s crazy. It’s a general-purpose technology, so it’s a slightly unfair comparison.
What I’d say is that every command, every office in government knows their problems.
They know the things that drive their people crazy. They know the things that cost them money and time. Half the time I meet with folks in government about these things, they’re asking me, “Hey, what do you think I should do?” And I say, no, no, no, you tell me. You know your problems. I’ll help take those problems out of your brain, but you’re the one who knows them.
You have to have a feel for your organization and what its biggest problems are, and you have to be able to follow the current of what will actually solve people’s problems. What can you get money for, which is always a skill in government? What will deliver wins for your leadership? You try to find the middle of that Venn diagram, and that’s where you start. Once you get some traction, you can move on to other challenges.
Incentives and Forcing Functions
Jordan Schneider: If a company doesn’t do this, it just gets outcompeted. But what are the checkpoints for you? Is the Air Force being outcompeted? Is national security sound? Those tests only come along — hopefully — once or twice a century.
Garrett Berntsen: Hopefully less than that.
Jordan Schneider: The absence of a forcing function just means you can get away with not being good for longer.
Garrett Berntsen: That’s right. But we underestimate our ability to create our own mechanisms to drive that change, and this is where leadership and personnel policy come into play. We need better mechanisms — both carrots and sticks — to encourage risk-taking and to support innovators in and out of uniform who are doing this work.
People follow incentives. People are ambitious, and they genuinely want to do a good job. If you have a system that encourages more of this behavior, it looks like people taking big risks, pursuing unusual career trajectories, and being rewarded for it. Meanwhile, people who aren’t moving fast enough face career stagnation or removal. That can feel crude, but it’s the mechanism you have to use. You have to incentivize people and leaders, and there have to be clear rewards and consequences depending on whether they’re making progress.
Jordan Schneider: The problem with the Garrett 1.0 approach — just solving the problems everyone can identify — is that you’re only identifying problems whose solutions also work for you. It’s rare to find a person who will tell you their own job shouldn’t exist.
Garrett Berntsen: People will say, “I really don’t like the way I have to do my job,” though.
Jordan Schneider: Sure.
Garrett Berntsen: So many people feel stifled by how they’re working. They want better tools and a better process. It won’t be 100% of the workforce — you’re right, there will always be people who say, “I’m used to how I work.” But that’s the adoption curve any technology faces. There are early adopters, and eventually people jump on the bandwagon once it works. There will always be holdouts who preferred the old way. That’s hard to overcome, but you just have to force it.
Jordan Schneider: There’s a Jen Pahlka anecdote about her talking to a general about how to fix the system. She said, “I don’t even know if we should cut your budget — this stuff is really important.” And the general said, “No, 50%. We need to cut 50%, because that’s the only way to get people to change.” We’re never going to hear someone selling software to the government say we need to cut spending by 50%.
Garrett Berntsen: Programs have to have winners and losers. As an industry, we want money spent on the most valuable things for the government, even if that money isn’t going to us. We care about the government running well. What I’d also say is that changing what you’re buying often isn’t a bad thing, because we’re excited to do new things and tackle new problems. Programs that are growing are usually more exciting for our people, and they want to jump on something new. Change always happens in this industry, and it happens in every industry, but from our point of view it isn’t all bad.
Jordan Schneider: There’s a question about AI — this is going to radically remake how organizations can be structured. That radical restructuring is going to start on the software side, and maybe one day we’ll get to the hardware side. But we’re still going to have to build things, even if they’re drones rather than manned fighters.
Garrett Berntsen: You still have to manage the logistics supply chain, the warehouses, all of that. None of that is going anywhere. If anything, we just want it to be more efficient, and you might end up doing more of it.
The Advantages of Moving Second
Jordan Schneider: It feels like there are scenarios where things need to change by 75%, and the system can only change on its own by 5%. With smart, enlightened leadership, maybe you get to 15%. Is the answer Congress? Or is it crazy to say, “We have the most powerful military in the world, so maybe we don’t need to change”?
Garrett Berntsen: The reason we have the most powerful military in the world is that it can adapt and change, and it needs to. That has always been true. We’re just facing a very radical pace of change at the moment, so we need leaders willing to move at that pace.
We talked a little about key areas where we’ll need change, and you hit on software development. That’s the most obvious one, because of how powerful these models are at writing software. You could radically change the way you buy and the way that work gets done.
Contrary to the U-2 example, where the technology started in government, all of this is happening commercially first and then coming into government. We get the upside of second-mover advantage — we get to see what’s working. There are already lessons from the past year. Do you want to unleash everyone with unlimited tokens so they burn through your budget in a quarter, or do you need to understand tokenomics a little better before you deploy that across your workforce?
There are real upsides to moving second. If you do it right, the federal government can be a very big, powerful buyer and negotiate good bulk rates. Going second isn’t all bad — that’s my bottom line.
Jordan Schneider: I want to come back to 21-year-old Garrett, the logistician driving around Afghanistan running his trucks off his own Excel spreadsheet because he couldn’t trust the software he’d been given.
I would like for you to be able to vibe code your own solution. Is that the answer for everyone everywhere? What should be in-house versus just letting enlisted service members — or, sorry, officers I mean — go crazy?
Garrett Berntsen: My enlisted did almost all the work anyway, which is how most officers feel.
There will be a need for these skills, and there already is a need for them out in the field — that was true even 10 or 20 years ago. When I said I was doing a lot of this on spreadsheets, there were people in my unit helping me out, using technology in whatever way they could.
The US service member has been famous since World War I and World War II for being innovative, for just figuring things out. I served in the 82nd Airborne Division, and this is 82nd Airborne lore. Paratroopers hit the ground on D-Day and simply figured things out — merging with units that weren’t their own and fighting. That’s something we’re all very proud of, and it’s going to continue to be true.
The reason I tell that story is that if we get these tools into their hands, you’re going to be shocked at how innovative and clever people are in using them to do their jobs. That may mean getting compute out to the edge. It may mean form-factor GPUs out on service members. Then it’s about giving them what they need to get the job done and capturing the lessons learned as much as possible — what’s happening out at the units — so we can share that knowledge across the whole force.
There will always be some dividing line between the companies sitting back in safer locations and the service members out at the pointy edge of the spear. But as much as humanly possible, get whatever the most modern tools are out to the edge. You will be shocked by the existing capability and skill in these service members. I’ve seen it. They just figure things out. That’s what NCOs and enlisted service members do — and maybe officers, by exception. It’s really about figuring out what they need and then getting it for them, whether that’s hardware or software, and then breaking through all the inevitable policy reasons why someone would say no.
I can already think this through in my head as a logistics officer. Who’s signing the hand receipt for the GPU? These are $500,000 chips. It breaks, and suddenly someone’s in trouble because they broke the GPUs. You have to clear the policy out of the way and say, we’re assuming the risk. If it means some GPUs get broken, and we lose the money, it was worth it, because we’re getting capability out to the edge.
Jordan Schneider: Coming back to the 82nd Airborne in World War II and having enough talent for emergent warfighting — that’s what Ukraine has today. You’re pulling from your entire population. The distribution curve you live with in a volunteer army, and in a civil service that’s underpaid relative to the private sector, is different from the curve you get when everyone is in the fight. Let’s hope we never have to find out.
Garrett Berntsen: I might argue with you a little on that. I actually think you get a more skilled distribution. If you look at the data on who joins the government, the civil service, and the enlisted ranks, they’re on average more educated than the average American. Service members are also largely representative of different ethnic backgrounds — almost to a T, in a lot of ways, of the composition of the country. You would be shocked how many enlisted soldiers you meet who say, “Oh, I have a master’s degree.”
By the way, the difference for Ukraine — and this makes it sound like a positive, which obviously it isn’t — is that they’re facing the brutal reality of having to survive. That’s the honest-to-God situation there. They have to survive, so they’re innovating incredibly fast. We don’t have to face that right now, so it’s up to us to watch, learn, and be smart about what other people are having to face, and to learn those lessons now so we don’t have to learn them the hard way later.
This force over the last 20 to 25 years is uniquely educated, in a good way. It’s a smaller force, but a very skilled one.
Algorithmic Negotiation and AI Advisors
Jordan Schneider: I want to come back to evals. You mentioned JADC2, this idea that you can do targeting in a new way — it’s almost an evolution of the U-2 stuff, right? We’re just going to be able to figure out where things are in real time. That’s the sort of thing where it’s relatively straightforward to imagine how you would test and evaluate it. Obviously you won’t be perfect, and horrible things will happen because of that, but there is a line on a graph that you can chart. We’ve already launched ChinaTalk’s AI evals for the Situation Room.
The idea is to move up from the tactical and operational level to decision-making and strategic calls. What do you make of that?
Garrett Berntsen: It’s possible. I just don’t see a world in which we’d be able to get all the data. You need good data to do that type of modeling, especially if you’re doing multi-party negotiation modeling. People try to do this — economic modeling, trade modeling, questions like what do the parties want? There’s game theory, there’s all this stuff. There are different approaches to the problem.
If you had absolutely enormous access to people’s private notes and emails and all the audio and video from every Situation Room meeting — which obviously would never happen — you’d be surprised. You probably could do some very good modeling on decision-making outcomes. But it’s purposely obfuscated.
That makes it a challenge. That’s the ideal case, where you have all this perfect data. If you step back a little bit from that, betting markets and similar efforts are trying to do this sort of thing — crowdsourcing outcomes of events and negotiations. They’re trying to use the Phil Tetlock, Good Judgment Project approach. I haven’t seen the research or data on how well that works. I don’t know if you have.
Jordan Schneider: It’s getting better, and it’ll keep getting better. Maybe set aside the decisions themselves — what about intelligence analysis? Where do you see the uplift there?
Garrett Berntsen: That’s a more doable project. There are also incentives inside the intelligence community to do it. It would make their analysis better, and they have access to the data.
If you think about the intel workflow, it’s pretty structured — pretty rigorous, actually. Every one of the three-letter agencies has a different discipline, but they have a process, they have an SOP, and they go through training when they start. This is how you write this type of report and how you write that type. If you have decades of data on that, plus some signal on what was good and what was bad, what was high quality and what was low quality, then you can start to net out what looks good and what looks bad in the intel process, and what a benchmark for that would be. That’s definitely a more doable problem.
But it’s upstream from decision-making — it’s an input to decision-making. As we know from history, plenty of decision-makers see the intel and make a completely different decision for whatever reason. It would still be just one input into your broader goal of modeling all decision-making.
Jordan Schneider: The dream is that you’re sitting here and you know exactly what offer you can put on the table to the Iranians — one that best aligns with your national interests, political priorities, and timelines, and that gets them to yes. That’s really hard. Even if you understand your own system perfectly and can model the oil markets perfectly, it’s still hard.
Garrett Berntsen: I don’t want to overindex on the human factors, but they matter. If you look at famous American negotiators in history, people like Holbrooke — the call in the middle of the night, the flight out, spending time with these people, haranguing them — those things aren’t going into a model.
You certainly could give our negotiators better tools, showing them where the lever points are. But really good negotiators in diplomacy are already getting a lot of that intelligence. Maybe they’re synthesizing it all in their brain, but they’re reading the intel. They have experts — economic experts, political experts, intel officers — saying, “Hey, don’t forget about this.” What we’re talking about is taking it out of Holbrooke’s brain and putting it into a model. But then you still have to give it back to a person.
Jordan Schneider: That’s the question. Do you have to give it back to a person?
Garrett Berntsen: You’d just have the model send the counterproposal back over?
Jordan Schneider: I would not be shocked if, five years from now, business negotiations are just two models arriving at a number that works for both parties.
Garrett Berntsen: Maybe for highly transactional things. We already have that in certain business transactions — Facebook Marketplace, or ad auctions. You say what you want and it gives you a calculation. But here I am, the chief AI officer, telling you how AI isn’t going to do this — maybe I just want to keep my job. But people like doing business with people. They like the shtick of it. Humans are still going to prefer humans.
Jordan Schneider: TBD, man. But what if the non-humans can outcompete the humans? Five, ten, fifteen years out, there will still be things humans have a comparative advantage on. The idea that arriving at a number in a negotiation is one of them, though — I’m not so sure.
Garrett Berntsen: That seems unlikely to me. For microtransactions, we already do this largely algorithmically. Think about Uber, or the era before the meter. I lived in Egypt for a while in college. I’d get in a cab, say where I was going, and the driver would say, “That’s 50 pounds.” And I’d say, “It’s not 50 pounds, it’s 20 pounds.” He’d say, “No,” and we’d go into this whole negotiation, because I’m the foreigner and he’s going to charge me more. Understandably.
Then the next upgrade is the meter, a flat fee. And now we have an algorithmic transaction that looks at all these factors, comes up with a market price, and we agree on it. That’s a natural progression, and more transactions will occur that way. You’re right about that. But the big transactions, the big negotiations — the biggest ones are country-to-country, massive political and economic negotiations. Those, I feel, will be the last.
But bottom line, few diplomats in foreign countries are going to say, “Sure, send me over that ChatGPT response.” People want to talk to one another and negotiate.
Jordan Schneider: My thesis is that this happens once you have candidates who win because they follow everything the model tells them to do. It won’t emerge organically — it’ll just be that if there are ten of us running in a presidential primary, and then one of us —
Garrett Berntsen: Uses all the model output. That’s very interesting, because then it really becomes a question of which model you listened to. Did you listen to the guardrailed model that said don’t run negative attack ads? Or did you listen to the totally unhinged model that said, “Yeah, go attack them, make up lies about them”? Then it gets into what model you’re using and how we guardrail these models.
Jordan Schneider: It’s going to be scary. It’s going to be really messy.
Cyber Exploit Abundance
Jordan Schneider: Let’s talk about war a little bit more — cyber. Big week for cyber. We’re recording this on July 23rd, two days after OpenAI’s model jumped its guardrails. Where are we going with all this, Garrett? How are you feeling right now?
Garrett Berntsen: I’ll start by saying I’m more of an AI person than a cyber person, but over the last few months I’ve had to get deep with our team, our cyber team in particular.
What we’re doing is putting lighter fluid on things we’ve already been talking about. These vulnerabilities have been around — it’s not as though these are all new issues. It’s just that we suddenly have a tool, or a series of tools, that can find them really, really fast. The pace is the key difference here.
We’re talking to clients and government leaders about how we go from “oh my gosh, we’re freaked out by this” to “what do we actually do?” Literally, how do we fix these problems? The answer has to be that you use these models to solve the problems they’re identifying. It’s both a weapon that can be used against you and a defense you can use.
Famously, every CISO and CIO has a massive list of potential issues. How do you triage? How do you prioritize? These models are really good at helping you do that. And how do you write the code to harden and strengthen your own systems? They’re good at that too.
You cannot put your head in the sand and say this is just a small change. The first step is accepting the problem. If a model is breaking out of the security protocols put around it by OpenAI itself, your system is at risk too. That’s the reality of the situation. So the first thing is acceptance — like any problem, accept that you have one, and then move as quickly as possible.
You just did a whole podcast about Kimi and these new Chinese models. There’s not a huge lag between the pace of the Chinese open-source models and our frontier-grade models. It’s coming.
You just have to move out quickly. Then you need Congress to budget it, you need people to find the money, and you need leaders willing to take the risk to make changes and patch faster. All these things come back to attitudes like, “Oh, we do patching once a month.” No — you’ve got to patch a lot faster than that. It’s a boring thing, but you have to be able to tell your user base, “Sorry, your computer is going to reboot every day.” I know that’s annoying. It’s going to hurt productivity and people are going to yell at you. But you just have to do it.
Status Quo Bias and Book Recommendations
Jordan Schneider: On Congress, budgeting, and the FAR — a lot of what the government pays for is people and their credentials at a specified hourly rate. But jobs are changing so much. Some things one person can now do alone, and suddenly that person is a hundred times more productive. Why won’t the time-and-materials model die?
Garrett Berntsen: They’re trying to kill it, and I support any effort to move away from that incentive structure. Fundamentally, it’s status quo bias. People feel more comfortable. When you trace the problem back, it’s almost always that change is hard for people. If I had to give one answer, that’s the answer. Change is really hard. Acquisition professionals are used to doing things a certain way. They get a rate card, they compare the rate cards, they compete, and they give out the contract.
Increasingly, that just isn’t going to produce the outcomes we want, and the reason is obvious. If you incentivize a company to bill more hours, and there’s no hard requirement to actually deliver something at the end of those hours, they’re going to bill more hours. It needs to change. There are signs it has been changing a lot in the last 12 to 18 months, but if you actually dig into the spending data — and that data is public — it’s not a massive shift. Most spending is still time and materials, billed hourly.
Jordan Schneider: There’s also the fact that top companies are spending something like 10% of their budget on tokens. I don’t know where the US government is, but I doubt it’s anywhere close to that.
Garrett Berntsen: That’s true too, and it’s another piece of this. There’s still uncertainty about whether you truly get 10x outcomes from tokens, or whether you’re basically paying for the same amount of work in a different form. I’m more of a believer that when you have teams built to do this well, you do get the gains — but a lot of learning still needs to happen, particularly in software development. You’re seeing teams that honestly aren’t very experienced and don’t really know what they’re doing, and their token burn is just wildly out of control, especially if the company is incentivizing token use. Not outcomes, not new features, not customer satisfaction. You have to incentivize the right things.
But with really mature teams, I am seeing large uplift in productivity — smaller teams able to produce more features, achieve technical outcomes, deliver a better user experience, and actually ship an enhanced product. There’s still a learning curve, though, in how to do this more efficiently.
We also have to accept — or I believe, anyway — that with all of this investment in infrastructure, the price of intelligence is going down. When you see these companies competing ferociously with each other, big investments in infrastructure, and improvements in model architecture and performance, intelligence is going to get cheaper. That feels inevitable to me.
Jordan Schneider: And as we discussed earlier, when you’re the second mover, you’re already that much further down the Moore’s Law curve. Plus, we have ten other Moore’s Laws running at the same time to make this intelligence cheaper. Token spend should not be a limitation the government gets blocked on anytime soon.
Garrett Berntsen: No. I work for a large services company, and most of our business is actually commercial — 92% commercial, only 8% federal. That’s a big upside for us in general. We have so much experience doing this stuff on the commercial side, where clients have been adopting really aggressively. We get to have those conversations and ask, how is that commercial company leveraging AI? How is our delivery team in that commercial company doing it? How are you tracking tokens? How are you tracking whether you’re actually getting outcomes from that? Then we try to bring those lessons in. We learn the lesson beforehand as a second mover and don’t make the same mistakes — because inevitably, with something changing this fast, you’re going to make mistakes.
Jordan Schneider: You mentioned earlier that even before ChatGPT, no one in the corporate world was paying for hours of PhDs. That’s been true for decades now.
Garrett Berntsen: Totally. I, in particular, love an outcome that we all agree on. We definitely have to stack hands on the outcome. This is where doing fixed-price work is hard, because you have to agree on what you actually want to achieve. A lot of times it’s just easier to say, “Oh, just keep working.” Telling people to work is not that hard. Telling people, “This is my vision, I want you to achieve this thing” — that takes alignment, human agreement, stakeholders, getting everyone on the same page. If you can do that, then you give us a goal, we agree on a price, and it’s really on us. The risk is on us. If we’ve agreed to that price and we’ve agreed to six months, and you don’t have your thing in six months, we’re just going to keep working to get it for you, because we committed to it contractually and we either did a bad job of pricing it or we didn’t deliver it well.
If you’re just paying me by the hour, you’re going to call me and say, “Hey, I’m mad it’s not done.” And I’ll say, “Yeah, I know, it’s hard to get done. We’ll keep working.”
Jordan Schneider: But in five years, is that scoping conversation something two models couldn’t have?
Garrett Berntsen: I don’t think so. We will see that scoping process change, and we’re already seeing it happen a little bit with AI. What we do is bring some leaders in from government, put a chat app in front of them — recording — and say, “Just say what you want. Describe your problem, or what you want.”
We’ve pre-configured the model a bit to be a question asker, like an agent designed to gather requirements. You can go back and forth pretty quickly, and then it’ll just build it. Of course, our engineers will look at what it built and make some changes and tweaks.
But you have to show people. People in government have never seen that. Honestly, most people in the world haven’t seen that — it’s only been possible for about twelve months. We have a space downtown where we do a lot of these events called The Forge. We bring people in and show them this on a computer, and it blows people away.
When you show customers that kind of thing, they say, “Whoa, okay, can you do this in my environment?” A lot of times we’ll say, “Not yet, the model isn’t authorized in your environment yet, but as soon as it is, we can.” That becomes a reason for them to get it authorized. Showing people the possibility matters — with this technology, you have to see it, you have to watch it happen, to understand it.
Jordan Schneider: Let’s close with some book recommendations.
Garrett Berntsen: All right, three things. I just read a summer novel called Yesteryear, by Carol Clark Burke, I think. It has a crazy plot twist I won’t ruin. It’s a little bit contentious, but it was a fun sit-by-the-lake kind of read.
Then there’s Raising Good Humans. I have two kids, two little boys, so I’ve been trying to read some parenting content. The short of it is that if you want to be a good parent, work on yourself. It’s not actually about your kids — it’s about making yourself a better person. That’s the CliffsNotes version, in case you don’t want to read it.
The third book, which I just picked up at a book sale, is the huge Truman biography — the one by David McCullough. I’ve never read anything about Truman, and it’s super interesting. Here’s a fun fact I just read in it: the US troops — the Doughboys who went into World War I — didn’t have any of their own artillery. No American artillery was used in World War I. It was all European artillery.
They went over there, trained on the equipment, and then mostly used French artillery. They had to convert to the metric system, these guys who had never done anything like that. What a wild fact. We had no industrial base. We built no big guns.
Jordan Schneider: Okay, so you’re still in the World War I section of Truman. I remember he was a library rat or something.
Garrett Berntsen: He was a big reader, yeah, but he also had a bunch of failed businesses. He opened a haberdashery, a men’s clothing store. He opened a mine or something that didn’t work out.
Jordan Schneider: Wasn’t there some horse venture he did?
Garrett Berntsen: There were a whole bunch of random businesses that he started. He was a very unassuming guy. It’s been interesting. I knew nothing about him. I’m only about 200 pages in, and it’s a thousand-page book.
Jordan Schneider: For dad books, there’s one called An Italian Education, by a cranky Irish guy, written in New Journalism style. He married an Italian woman, and they were living in the suburbs somewhere in northern Italy in the ’90s. It’s a very sarcastic exploration of Italian culture.
Garrett Berntsen: There’s a whole subgenre of parenting books along the lines of “I’m the American in Paris.”
Jordan Schneider: It’s still being squeezed by a publisher into something that can sell millions of copies. This book is not that. It’s just this Irish guy being cranky. I guess I like cranky dads. That’s my genre.
Garrett Berntsen: Are you a cranky dad?
Jordan Schneider: There’s so much non-crankiness in the dad literature that it’s helpful to know there are other people who are cranky at times. Michael Lewis wrote a dad book. He kept a diary of the first year of each of his three children’s lives.
Garrett Berntsen: Wow. I could never. I always feel envious when people keep diaries, because I wonder, when do you find the time? And I feel like I’ve forgotten so much.
Jordan Schneider: I do a sentence. I’m a sentence-a-day guy.
Garrett Berntsen: That’s smart.
Totally awesome conversation. Something my very brief experiences in Ukraine earlier this year absolutely reinforced to me was the observation y’all make about how imperative the act of getting technology deployed out on the edge where dudes can put it into action and figure it out really is. So, so, so often the guy best positioned to scope an application is the guy dealing with an urgent, pain-in-the-ass problem only he knows about and thinks to solve.
Thanks for having me on and for the great discussion.