Incremental View Maintenance Implementation as a PostgreSQL extension

I’ve written before about leveraging Postgres views, focusing on non-materialized views. Materialized views bring other benefits beside abstractions: Namely they can increase the performance of data loading, by doing some of the computations ahead of time (and trading off disk space). Using materialized views has some difficulties, because Postgres requires that the data refresh be explicitly managed, and that it refreshes a whole table at once.

The pg_ivm extension promises to make it much easier to work with materialized views:

Incremental View Maintenance (IVM) is a way to make materialized views up-to-date in which only incremental changes are computed and applied on views rather than recomputing the contents from scratch as REFRESH MATERIALIZED VIEW does. IVM can update materialized views more efficiently than recomputation when only small parts of the view are changed.

I have not used this in production, but it looks very promising

Engineering Levels at Honeycomb: Avoiding the Scope Trap

The folks at Honeycomb discuss how they think about their engineering ladder. In particular, they are mindful of falling into the “scope trap”. In said trap, only engineers that work on the biggest projects get promoted. I like their visualization of ownership vs scope and how different levels overlap.

On the Dangers of Cryptocurrencies and the Uselessness of Blockchain

I respect Bruce Schneier’s opinion on security and technology a lot:

Earlier this month, I and others wrote a letter to Congress, basically saying that cryptocurrencies are an complete and total disaster, and urging them to regulate the space. Nothing in that letter is out of the ordinary, and is in line with what I wrote about blockchain in 2019.

There is no good case for crypto: It’s a solution in search of a problem.

The GitHub blog recently announced Atom’s sunsetting. It’s been clear for a while that my text editor of choice was not receiving regular updates. It is now time to move on.

I first reviewed Atom in 2014, and switched to using is as my primary editor a few months later. I spend considerable amounts of time in my editor. I don’t believe in modal editors, and instead want to edit text with the same shortcuts that I use in my operating system.

I considered a few text editors, mainly from the list in the Ruby on Rails Community Survey. Among the things that I was looking for were a way to pipe text from the editor to any command, so I can have fast feedback loops. Most importantly, I’ve developed a habit of taking a lot of notes in a markdown, in a system of my own very heavily inspired by the Zettelkasten Method.

To hit all those objectives, it was clear I needed an editor that has a robust plugin system and is widely used. That pretty much left Sublime Text and VS Code as options. Sublime Text 3 is fast, and overall a pleasant editing experience. I was very disappointed with the available plugins. It’s package repository makes it very hard to know which packages are actually compatible with v3, and requires a lot of trial and error. I couldn’t configure it to my liking.

VS Code proved to have a very extensive marketplace of extensions that supplied the functionality I wanted. After a few days using it and tweaking as I went along, I am feeling very at home using it. I expected the same slow performance that Atom exhibits because both use Electron. I was pleasantly surprised. VS Code is much faster opening files (both cold opening without the application loaded, and warm opening). What really made the switch easier, was configuring to use the Atom keyboard shortcuts via an existing extension.

As of today, the list of extension I am using are:

$ code --list-extensions
akamud.vscode-theme-onedark
ban.spellright
BriteSnow.vscode-toggle-quotes
cescript.markdown-formula
fabiospampinato.vscode-open-in-github
hatoo.markdown-toggle-task
kortina.vscode-markdown-notes
ms-vscode.atom-keybindings
rebornix.ruby
ryu1kn.edit-with-shell
sianglim.slim
TakumiI.markdowntable
Thadeu.vscode-run-rspec-file
wingrunr21.vscode-ruby
zhuangtongfa.material-theme

I expect to continue to refine my VS Code configuration, but I am at the point where I don’t open Atom at all. In fact, I’ve uninstall it, because I was used to typing atom ... on the command line by force of habit. I am now retraining to type code ....

Distributed Systems Shibboleths

Shibboleths are historically a word or phrasing that indicate membership in a particular group or culture. Joey Lynch identifies some distributed systems shibboleths. In a way, shibboleths are used implicitly in many circles, but I rarely see them being used explicitly. The author identifies both positive (e.g. idempotent, crash-only, shard) and negative (e.g. consistent and available, exactly-once, i just need transactions, distributed lock) words and phrases that serve as shibboleths.

IBM’s asshole test

I don’t know if this really happened, or if this interview technique is really fair to candidates. The motivation resonates with me: Weed out assholes!

Mechanical Watch

Bartosz Ciechanowski has produced some of the best technical writing I’ve read on the web. This long post is a fantastical interactive description of how a mechanical watch works. The illustrations are superb, and link to the text via color coding. If you like this article, be sure to look at others in his blog!

A script that I’ve been using for years stopped working as expected after I upgraded bash and git. I use ctags to navigate code in my editor (currently Atom). To automate the generation of the tags file, I run the ctags executable from git hooks (post-commit, post-merge, and post-checkout), which fits well with my development workflow.

Some of the projects I work with are quite large, and the ctags invocation can take longer than 30 seconds. To avoid waiting that long on each commit, I background the invocation. The hook – that has worked for years – looked like this:

#!/usr/bin/env bash
# Regenerate ctags

# Only run one ctags process for this directory at the time.
# Otherwise the ctags file is corrupted
(lockfile .ctags.lock; \
 ctags -R --exclude='*.js' --exclude='*.h' --exclude='*.cpp' &> /dev/null ; \
 rm -f .ctags.lock) &

The lockfile usage prevents multiple copies of ctags running at the same time, which can happen when the hook is invoked often (like when comitting multiple times in quick succession). The (..) invoke the commands inside on a sub-shell, and the & at the end tells bash to background the work and continue.

I’ve been using this for years without issue, until I recently upgraded both git and bash on my machine. The invocation above continued to generate the tags as expected, but instead of backgrounding the work, the git hook would block until ctags finished.

I could not find anything related to that in either git or bash release notes. StackOverflow provided several tips regarding using nohup or disown but using them didn’t help.

Eventually, what did work is redirecting the output of the sub-shell, instead of redirecting the output of ctags alone:

(lockfile .ctags.lock; \
  ctags -R --exclude='*.js' --exclude='*.h' --exclude='*.cpp' ;\
  rm -f .ctags.lock) &> /dev/null &

When the sub-shell is instantiated, it’s stdout and stderr are connected to the parent process (i.e. the git hook). My best guess is that after the upgrade, the hook invocation now waited until the sub-shell existed because it’s std{out,err} was connected to the sub-shell’s. With the new invocation, the (..) &> /dev/null disconnects the output streams for the whole sub-shell from the hook’s output streams, by redirecting it to /dev/null. The hook’s process can then safely close its own std{out,errr} and exit.

The Dunning-Kruger Effect is Autocorrelation

This article is fascinating. The argument is that the well-known Dunnign-Kruger effect (i.e. unskilled people overestimate their skill), is not a psychological effect. Rather, it is a statistical mistake. It is an artifact of autocorrelation: Comparing a variable to itself.

Refactoring Ruby with Monads

Tom Stuart is a great, clear writer. This article does a great job at introducing the usefulness of monads – explaining them from the ground up, without the math pretentiousness.

Ruby Shell-Out Flow Chart

Ruby supports many ways of doing that. This excellent flow-chart from a StackOverflow answer tells you which one to use. I am reposting mainly so that I can find it again easily!