Eventual Business Consistency

Kent Beck talks about bi-temporal modeling. It’s a topic I’m very interested in. I am glad that Ken Beck is talking about this: He has great readership and might make this a more mainstream technique. I am not sure about renaming it to “Eventual Business Consistency”.

However, I think part of the reason it hasn’t become more popular, given the benefits it brings, is just the name. Hence my proposed rebranding to “eventual business consistency”.

I don’t see any rationale for this assertion. I doesn’t ring true for me. Bi-temporal data/modeling seems like a fine name. Programmers regularly talk about polymorphism, inheritance, dependency injection, concurrency, parallelism. As far as I can tell, bi-temporal doesn’t seem different than other technical jargon. I fail to see why it’s a disadvantage.

If I had to guess, I think that he was right in the first place:

Part of the reason it hasn’t taken off is because of the additional complexity it imposes on programmers.

Bi temporal modeling adds a lot of complexity. Most queries are “current” state queries, where NOW() can be used for both the validity range and the transaction range. The complexity comes from primary keys and foreign keys now needing to account for the ranges. It’s solvable, but most databases (Postgres, MySQL) don’t have first-class support for modeling like this. This probably could be solved with extension or application frameworks. I believe that could actually bring more usage.

Another barrier to entry, is that most applications are not designed with bi-temporal data in mind. Adding bi-temporal data to an existing model is more complicated and requires migrating data (and obviously not all data has been kept).

Just normal web things.

I nodded along while reading. Users have expectation of how the web should work and what they can do with it: Copy text, open a new window, etc. Websites shouldn’t break that! Sometimes websites are really apps that have a different UX paradigm (e.g. a photo editor). Most of the website that are coded as “apps” – and break web conventions – could easily be standard CRUD web apps. Sigh.

Play in a sandbox in production

Andy Croll advices to use rails console --sandbox in production, to avoid making unintended data changes.

The “why not” section is missing that opening a rails console with --sanbox opens a transaction that is rolled back after the console is closed. Long-running transactions can cause whole system performance degradation when there is a high load on the system.

When should you worry about this? Depends on your system. I’ve worked on systems where traffic was relatively low, and wouldn’t be a problem. I’ve also worked in systems where a long-running transaction of only 1 or 2 minutes cause request queueing that would bring the whole system down.

Is there an alternative? Yes. Open a rails console using a read-only database connection (to a read replica or configured to be read-only against the same database). That is not as easy as --sandbox, but it can be as simple as setting a postgres variable to make the connection read only.

I recently noticed an unexpected N+1 in a Rails project when using #without (aliased from #excluding).

The feature is a page that lists all available programs, and a list of participants other than the current user. In its basic form it’s equivalent to:

# Controller:
programs = Program.all.includes(:participants)
# Program Load (2.0ms)  SELECT "programs".* FROM "programs"
# ProgramParticipant Load (1.0ms)  SELECT "program_participants".* FROM "program_participants" WHERE "program_participants"."program_id" IN ($1, $2)  [["program_id", 1], ["program_id", 2]]
# Person Load (0.5ms)  SELECT "people".* FROM "people" WHERE "people"."id" IN ($1, $2, $3)  [["id", 4], ["id", 2], ["id", 3]]

# View
programs.map do |program|
  program.participants.without(current_user).map { _1.first_name }.join(", ")
end
# Person Load (1.5ms)  SELECT "people".* FROM "people" INNER JOIN "program_participants" ON "people"."id" = "program_participants"."participant_id" WHERE "program_participants"."program_id" = $1 AND "people"."id" != $2  [["program_id", 1], ["id", 4]]
# Person Load (0.6ms)  SELECT "people".* FROM "people" INNER JOIN "program_participants" ON "people"."id" = "program_participants"."participant_id" WHERE "program_participants"."program_id" = $1 AND "people"."id" != $2  [["program_id", 2], ["id", 4]]
# => ["Gabriel", "Gabriel, Alex"]

Notice that participant (in people table) are being loaded, and seemingly ignoring the includes in the controller.

The N+1 was not present before this app was upgraded to Rails 7.0. That is key. We can see in the changelog the implementation of ActiveRecord::Relation#excluding (but not mentioned in the guide as a notable change). Before that, excluding (or without) was implemented in Enumerable, which didn’t create the N+1. In fact, using that method – by calling to_a on the relation – returns us to the desired behavior:

programs.map do |program|
  program.participants.to_a.without(current_user).map { _1.first_name }.join(", ")
end
# => ["Gabriel", "Alex, Gabriel"]
# --> Same result, no extra query!

Conclusion

Typically, doing more work in the database and less in Ruby brings performance improvements. In this specific case, the optimization prevented using already loaded data, which resulted in many more queries and overall worse performance. Catching these errors when upgrading Rails is difficult, because the functionality was actually not affected.

The Day FedEx Delivered Its Promise

The story is compelling. A small tweak has a big payoff. We’d all like to believe that we can do that in our own lives. It also rings as apocryphal, but I haven’t actually checked. The takeaway is that incentives matter, and changing incentives changes behavior.

The more interesting question is: how do you find the correct incentives? Getting lucky is one way. Is there a systematic methodology to design and measure incentives? It also reminds me of the adage “You optimize what you measure”. The measuring itself becomes an incentive.

Responding to “Are bugs and slow delivery ok?”

This article responds to another article, about when it’s OK to ship buggy software. I think the original was a marketing ploy by that author setting up a false dichotomy so that you would agree that you do need good quality. This author then misunderstands, I think, but it doesn’t matter. The claim is that it is in fact OK to ship slow, with bugs because:

I’ve seen (and wrote) some terrible quality code. Really bad stuff. Untested, most of it. In nearly every place I’ve worked at. I’ve seen enormous amounts of time wasted with testing for, or fixing, bugs.

You know what I haven’t seen? not once in 15 years? A company going under.

There is lots of ways a company can have bad performance, without “going under”. It’s is a false dichotomy. Akin to dismissing all diseases because they are not deadly, glossing over the nuance that you can still suffer a lot without dying.

In any case, it’s hard to think that a company like Apple would be the the most valuable company in the world, if they embraced shipping buggy software or hardware. A company can limp along and compensate, but not excel. Quickbooks comes to mind. Their software is the worst. Everyone I know that uses it hates it, but they are profitable. Why? Because they have captured the accountants market, and they make their clients use it. They survive like that, but I don’t think they actually thrive.

Online Data Type Change in PostgreSQL

This article does a good job at sequencing how to change a column type in postgres without locking the whole table. The temp table for keeping track of what needs to be backfilled works, but it can also be done without.

What was not discussed at all, is what happens to the application code in the meantime. Rails applications will see the new column an register it. The trigger takes care of writing the data for that column, but when we rename the columns in the last step (or delete them), we are changing the column information from under Rails’s proverbial rug, which will cause exceptions. Solvable problems, if you are looking for them in the first place.

I was trying to update rails in one of my projects recently. The command I ran was not quite right, so I wanted to discard changes to Gemfile.lock

I kept doing:

git restore Gemfile.lock

But the git reported the file as still dirty, and the unwanted changes were still there. I couldn’t understand what was going on!

Eventually, I noticed that:

git restore Gemfile Gemfile.lock

worked. Then it hit me. A spring server was still running an apparently running bundle install whenever the Gemfile changed, which was regenerating my unwanted changes as soon as I restored them with git.

I guess for some use cases this quiet behavior helps. In my case, I wanted to run a particular bundle update:

$ bundle update rails --conservative --patch --strict

spring. Sigh.

I’ve just read a blog post on Fly.io that says:

So, a few months back, during an integration with a 3rd party API that relied on OAuth2 tokens, we drew a line: ⚡ henceforth, hazmat shall only be removed from Rails, never added ⚡.

Hazmat. As in hazardous materials. Wikipedia defines it as:

Dangerous goods, abbreviated DG, are substances that when transported are a risk to health, safety, property or the environment. Certain dangerous goods that pose risks even when not being transported are known as hazardous materials (syllabically abbreviated as HAZMAT or hazmat)

I’ve never heard of software secrets being described as hazmat. It’s an apt metaphor. They can be dangerous, even when not being transmitted. We should be careful on how we handle them. Hazmat evokes specialized suits and gear. It’s not a bad mental image when dealing with secrets!

The second interesting part about the line in the blog post is the “only removed, never added”. That makes me think of a ratchet, defined as:

ratchet1 | ˈraCHət |
noun
1 a device consisting of a bar or wheel with a set of angled teeth in which
  a cog or tooth engages, allowing motion in one direction only
2 a situation or process that is perceived to be changing in a series of irreversible
  steps

A hazmat ratchet, if you will.