RFD 576 Using LLMs at Oxide

A thoughtful overview of how Oxide uses AI. It breaks down values, different uses, and reaches solid conclusions. It recognizes AI’s usefulness for some cases while keeping humans firmly responsible.

Has the cost of building software just dropped 90%?

AI’s explosion generates more demand for software by making it easier to build. A developer and business analyst can now ship an incredible amount at lower cost. What once required five people working for months now takes one developer and a part-time business analyst a week. Because creating internal tools and prototypes is cheap, more will get built.

Smaller teams save money another way: less coordination. Fewer status meetings. Of course, developers now coordinate agents instead.

I wonder whether ideas will be less polished when fewer people are involved before building starts. As Tim Cook says:

You can go fast alone. You can go far with a team.

Building the right thing still matters. “Because we can” is not a reason to build something.

Horse

The horse and chess analogies are interesting. They suggest that as new technology improves incrementally (~20% per year), it reaches a tipping point where it clearly becomes better than the older technology, and adoption is immediate.

Cars improved steadily until they were clearly better than horses. On the next “replacement” cycle, most people traded their horse for a car.

Chess programs improved beyond a threshold where they could beat a human most of the time.

Now, regarding jobs and AI: answering questions from a knowledge base is a task where AI excels. As the author acknowledges, that part of his job is no longer something he does. He was “replaced” – but that was only part of his job. In fact, that wasn’t even the principal part:

I was one of the first researchers hired at Anthropic.

Presumably, everyone is happier that he can now focus on actual research instead of answering onboarding questions.

My takeaway: AI is already changing some of our tasks and changing our jobs.

Prediction: AI will make formal verification go mainstream

I’ve always been interested in formal verification. The idea of mathematical proofs for algorithms and software is appealing. As noted, it is highly impractical because writing the proofs themselves is hard and requires “PhD-level training”.

Kleppmann argues that AI can change that, and not only that, can help validate the onslaught of AI-written software that is coming.