In August last year I wrote that we'd gone through every app we use and switched AI off by default, that clients would opt in to how much AI touched their work, and that we wouldn't let it anywhere near live systems, repos or production. In March I wrote that it's a powerful tool that is "often very very wrong", and that I wasn't buying the hype that its capabilities would improve exponentially.
Both of those articles were honest at the time. This one is the update, because a fair bit has changed in how we work, and a fair bit hasn't. The AI Usage Policy has been rewritten to match.
Key takeaways
- We now use AI hands-on in production. Mostly shipping code, and increasingly for preparing documents and analysing research.
- The line on sensitive data has not moved. No AI touches customer data, personally identifiable information, credentials or anything secret.
- AI features in our everyday tools are still off by default. The one change is meeting transcripts are on. Video and screen recording stay off.
- The models released in the last few months, Claude's Fable and OpenAI's Astra this week, are a step change. Not a marketing 10x, an actual one, in how well they understand a brief and how good the end output is.
- The trade-off for handing over more of the work is more checking. Agents now review agents, and every pull request goes through CodeRabbit before a human looks at it.
- Creative thinking and design are still average to poor. That's unchanged, and it isn't where we point these tools.
- Our default position with clients has flipped from opt-in to a stated default they can dial down.
What actually changed
In March I described the experience of building with AI as "sometimes good at doing step 1, and very not good at doing step 2". Getting a first pass that looked like a real app was easy. Getting it to actually work, and be good, cost hours of logic loops, undone work and code I had to unpick. That was true of every model I'd used up to that point, premium or local, and it's why our policy said no end-to-end code creation and no pushing to a repo.
Then the current generation of models arrived. We've been running Claude's Fable since it landed and started on OpenAI's Astra the day it was released this week. The difference isn't incremental. On the work we do, the models understand what's being asked the first time, hold that understanding across a long task, and produce output that needs a review rather than a rebuild. The time we used to spend correcting, re-prompting and reverting has mostly gone. I'd put the improvement in understanding and end output at ten times, and I don't say that lightly given how sceptical the last two articles were.
So I'll own the March prediction. I said I wasn't buying that capabilities would improve exponentially over the next ten years, and I'm still not buying the "world order is changing" story that goes with it. The historical pattern I described, overestimate the short term and underestimate the long term, still looks right to me. But I under-called how much better the tools would get in six months. Working with what I can test myself today, as I said then, means updating when what I can test changes.
What we're doing with it now
The biggest shift is in programming. Where the previous policy stopped at "coding assistant, discrete tasks, debugging", we now give agents open briefs, let them work in our repos, open pull requests and take work through to production. That's the thing we explicitly said we wouldn't do a year ago, and it's now a large part of how development at Harvey happens.
The second shift is document preparation and research analysis. Turning workshop notes, interviews and desktop research into structured documents, first drafts of reports, and pulling apart survey data. The illusion of productivity I wrote about last year hasn't fully disappeared here, the output still needs someone who knows the subject to read it properly, but the ratio of useful output to correction time has changed enough that it's worth doing.
We're working across three model families: OpenAI, Claude, and zcode, which is in a limited pilot only while we work out whether it earns a place.
The price of handing over more of the work
At the RIAA conference in June I said that running ten agents creates ten times the review work, and that you can't just trust the output. Letting AI ship code doesn't change that, it makes it the central design problem. Our answer is what I described on that panel as a harness: a framework where the work is checked, critiqued and cross-checked before a person sees it.
In practice that means agents review other agents' work as a matter of course. Every pull request goes through CodeRabbit for an automated review before a developer reads the diff. Tests have to pass. Nothing reaches production without a person approving it, and that person has to understand what they're approving. The three rules from March still stand: understand the subject matter, give very clear direction, and monitor, intervene and review everything. We've just built more of the monitoring into the process rather than relying on whoever is at the keyboard to remember.
What hasn't changed
Sensitive data. The most important line in last year's article was about what you put in. It hasn't moved. AI does not get access to customer data, personally identifiable information, passwords, keys or anything else that could do harm if it left the building. That applies regardless of how good the model is, because the risk was never about model quality. Our Security & Data Protection Policy defines what counts.
AI in our tools is still off by default. We went through Google Workspace, Slack, HubSpot, Zoom, Sentry, GitHub and the rest last year and disabled the built-in AI features. They're still disabled, because when we've revisited them they still mostly don't add much. The one exception is meeting transcripts, which we've turned on. Video and screen recording stay off. The transcript is useful, the recording of people's faces and shared screens was never the useful part.
Creative thinking and design. I said in March it was "excellent at creating the most generic (or terrible) design you've ever seen". Still true. The newest models are a genuine leap in understanding and execution, but they produce limited novel thought, and left to their own devices they produce user and brand experiences that are average at best. Concepting, brand design and website design stay with people. Where AI is useful in design is exactly where it was: wireframing, exploring options, editing images. Not the idea.
Human review before anything reaches a client. Unchanged. And deliverables that involved AI are still identified as such.
What this means for clients
Last year our position was that AI use was opt-in. Every client chose their mode at the start of a project and the default was to hold back. That made sense when we didn't trust the tools and neither did anyone else.
It's now the other way around. We have a stated default way of working, which is Assistant for programming and data, Generator for strategy and content, and Researcher for design, and every client can dial that down as far as they like, including to no AI at all. The choice is still theirs and it's still recorded in the project documentation. What's changed is that the default reflects how we actually work, rather than how cautious we felt in 2025.
The reasons a client might dial it down are the same as before. Sensitive data, a regulatory environment, a preference for how their money is spent, or a view on the environmental and ethical cost of the AI supply chain, which I wrote about at length last year and which hasn't got better. We'd rather have that conversation openly than assume.
Where I've landed
I still think we're near the top of a hype cycle, and I still think most of the confident predictions about jobs and industries will look silly in hindsight. But I'm not going to pretend the tools didn't get dramatically better, or keep a policy that was written for the ones we had a year ago. The version of us that wrote last year's article would be surprised at how much we now let AI do, and reassured by how much we still don't.
The updated AI Usage Policy has the modes, the defaults, the harness requirements and the boundaries in full.
AI usage in this article
This article was drafted by Claude from my notes, our previous articles and our existing policy, then edited by me. Roughly one round of back and forth to get the draft, and the edits were mine.
.avif)
.avif)
