I recently noticed an unexpected N+1 in a Rails project when using #without (aliased from #excluding).

The feature is a page that lists all available programs, and a list of participants other than the current user. In its basic form it’s equivalent to:

# Controller:
programs = Program.all.includes(:participants)
# Program Load (2.0ms)  SELECT "programs".* FROM "programs"
# ProgramParticipant Load (1.0ms)  SELECT "program_participants".* FROM "program_participants" WHERE "program_participants"."program_id" IN ($1, $2)  [["program_id", 1], ["program_id", 2]]
# Person Load (0.5ms)  SELECT "people".* FROM "people" WHERE "people"."id" IN ($1, $2, $3)  [["id", 4], ["id", 2], ["id", 3]]

# View
programs.map do |program|
  program.participants.without(current_user).map { _1.first_name }.join(", ")
end
# Person Load (1.5ms)  SELECT "people".* FROM "people" INNER JOIN "program_participants" ON "people"."id" = "program_participants"."participant_id" WHERE "program_participants"."program_id" = $1 AND "people"."id" != $2  [["program_id", 1], ["id", 4]]
# Person Load (0.6ms)  SELECT "people".* FROM "people" INNER JOIN "program_participants" ON "people"."id" = "program_participants"."participant_id" WHERE "program_participants"."program_id" = $1 AND "people"."id" != $2  [["program_id", 2], ["id", 4]]
# => ["Gabriel", "Gabriel, Alex"]

Notice that participant (in people table) are being loaded, and seemingly ignoring the includes in the controller.

The N+1 was not present before this app was upgraded to Rails 7.0. That is key. We can see in the changelog the implementation of ActiveRecord::Relation#excluding (but not mentioned in the guide as a notable change). Before that, excluding (or without) was implemented in Enumerable, which didn’t create the N+1. In fact, using that method – by calling to_a on the relation – returns us to the desired behavior:

programs.map do |program|
  program.participants.to_a.without(current_user).map { _1.first_name }.join(", ")
end
# => ["Gabriel", "Alex, Gabriel"]
# --> Same result, no extra query!

Conclusion

Typically, doing more work in the database and less in Ruby brings performance improvements. In this specific case, the optimization prevented using already loaded data, which resulted in many more queries and overall worse performance. Catching these errors when upgrading Rails is difficult, because the functionality was actually not affected.

The Day FedEx Delivered Its Promise

The story is compelling. A small tweak has a big payoff. We’d all like to believe that we can do that in our own lives. It also rings as apocryphal, but I haven’t actually checked. The takeaway is that incentives matter, and changing incentives changes behavior.

The more interesting question is: how do you find the correct incentives? Getting lucky is one way. Is there a systematic methodology to design and measure incentives? It also reminds me of the adage “You optimize what you measure”. The measuring itself becomes an incentive.

Responding to “Are bugs and slow delivery ok?”

This article responds to another article, about when it’s OK to ship buggy software. I think the original was a marketing ploy by that author setting up a false dichotomy so that you would agree that you do need good quality. This author then misunderstands, I think, but it doesn’t matter. The claim is that it is in fact OK to ship slow, with bugs because:

I’ve seen (and wrote) some terrible quality code. Really bad stuff. Untested, most of it. In nearly every place I’ve worked at. I’ve seen enormous amounts of time wasted with testing for, or fixing, bugs.

You know what I haven’t seen? not once in 15 years? A company going under.

There is lots of ways a company can have bad performance, without “going under”. It’s is a false dichotomy. Akin to dismissing all diseases because they are not deadly, glossing over the nuance that you can still suffer a lot without dying.

In any case, it’s hard to think that a company like Apple would be the the most valuable company in the world, if they embraced shipping buggy software or hardware. A company can limp along and compensate, but not excel. Quickbooks comes to mind. Their software is the worst. Everyone I know that uses it hates it, but they are profitable. Why? Because they have captured the accountants market, and they make their clients use it. They survive like that, but I don’t think they actually thrive.

Online Data Type Change in PostgreSQL

This article does a good job at sequencing how to change a column type in postgres without locking the whole table. The temp table for keeping track of what needs to be backfilled works, but it can also be done without.

What was not discussed at all, is what happens to the application code in the meantime. Rails applications will see the new column an register it. The trigger takes care of writing the data for that column, but when we rename the columns in the last step (or delete them), we are changing the column information from under Rails’s proverbial rug, which will cause exceptions. Solvable problems, if you are looking for them in the first place.

I was trying to update rails in one of my projects recently. The command I ran was not quite right, so I wanted to discard changes to Gemfile.lock

I kept doing:

git restore Gemfile.lock

But the git reported the file as still dirty, and the unwanted changes were still there. I couldn’t understand what was going on!

Eventually, I noticed that:

git restore Gemfile Gemfile.lock

worked. Then it hit me. A spring server was still running an apparently running bundle install whenever the Gemfile changed, which was regenerating my unwanted changes as soon as I restored them with git.

I guess for some use cases this quiet behavior helps. In my case, I wanted to run a particular bundle update:

$ bundle update rails --conservative --patch --strict

spring. Sigh.

I’ve just read a blog post on Fly.io that says:

So, a few months back, during an integration with a 3rd party API that relied on OAuth2 tokens, we drew a line: ⚡ henceforth, hazmat shall only be removed from Rails, never added ⚡.

Hazmat. As in hazardous materials. Wikipedia defines it as:

Dangerous goods, abbreviated DG, are substances that when transported are a risk to health, safety, property or the environment. Certain dangerous goods that pose risks even when not being transported are known as hazardous materials (syllabically abbreviated as HAZMAT or hazmat)

I’ve never heard of software secrets being described as hazmat. It’s an apt metaphor. They can be dangerous, even when not being transmitted. We should be careful on how we handle them. Hazmat evokes specialized suits and gear. It’s not a bad mental image when dealing with secrets!

The second interesting part about the line in the blog post is the “only removed, never added”. That makes me think of a ratchet, defined as:

ratchet1 | ˈraCHət |
noun
1 a device consisting of a bar or wheel with a set of angled teeth in which
  a cog or tooth engages, allowing motion in one direction only
2 a situation or process that is perceived to be changing in a series of irreversible
  steps

A hazmat ratchet, if you will.

How First Principles Thinking Fails

Thinking from first principles is not a silver bullet:

When reasoning from first principles, you may fail if:

You have flawed assumptions.

You make a mistake in one of your inference steps. (An inference step is a step in the chain of reasoning in your argument).

You start from the wrong set of principles/axioms/base facts.

You reason upwards from the correct base, but end up at a ‘useless’ level of abstraction.

I’ve been thinking a lot about the third one. If you don’t know certain facts, and you don’t know that you don’t know them, you can reach wrong conclusions.

For example, if you only know Newtonian physics, you will calculate the orbits of plants very precisely, but you will come to the wrong conclusions about Mercury’s orbit. You need to know about Relativity to get that orbit correct.

In essence, the “unknown unknowns” can get you.

Speed matters: Why working quickly is more important than it seems

If you are fast at something, of course you can do more of that thing in a given time. The author proposes that you also have a lower perceived cost for doing more of that thing, and that in itself lowers he barrier to doing more of that activity. The conclusion is that if you want to do more of something, you should try getting faster at it.

How do you get faster at something? In running (the sport), you get faster by doing speed-work, sprints, and the like. You also get faster by running longer and longer distances, at a relatively slow pace. The physiology is interesting, but not my point. In guitar playing, a very effective technique for learning a fast solo or transition is to use a metronome and slow down to practice the section, say at 50%, speed until your are proficient at it, then increase to 60% and so on, until you can play it at 100%.

While I agree that doing something faster promotes you doing more of it, it is not always intuitive how to get faster.

I’m an ER doctor. Here’s how I’m already using ChatGPT to help treat patients

This resonates with me: In it’s current form, ChatGPT is already very usable to generate text that you can verify is correct. In this example, the doctor can read the text and can tell if it’s empathetic, like he requested, and correct. It saved him from having to type it, and in fact maybe even did something that he could not: Communicate with more empathy. In any case, the doctor was 100% capable of evaluating the text produced.

In my personal use of LLMs, they can and will suggest code snippets. It saves me the time of having to read API documentation of several classes. I can evaluate the result for correctness, and even more importantly, I will adapt for my use case which will include tests for formally verifying that the code indeed works as expected.

I am not sure about the drunken intern analogy: I probably would have phrased it as a sleep-deprived intern instead. The intern heuristic is useful though. Getting back to the my code usage: It is useful to think of LLMs as an intern that produces code very fast, but that I have to evaluate. “Hey intern, how do I add a custom header to a web request using Ruby’s ‘rest’ gem?”. I am capable of evaluating the result and making whatever corrections are needed. Time saved.