-
Code spelunking with rg and join
The power of the unix philosphy is that you can compose single-purpose tools together, to great effect. For example, let’s say we are working on a Rails app. We want to find all controllers that use
current_user
, and also inherit fromApplicationContrller
.I would run the following:
$ join \ <(rg '< ApplicationController' app/controllers --files-with-matches | sort) \ <(rg 'current_user' app/controllers --files-with-matches | sort)
Let’s break it down.
$ rg '< ApplicationController' app/controllers --files-with-matches app/controllers/pdfs_controller.rb app/controllers/language_selection_controller.rb app/controllers/info/pages_controller.rb app/controllers/secure/base_controller.rb
ripgrep (
rg
) is an excellent replacement forgrep
: It searches the content of files for a matching regex. The expression above searches for the regex< ApplicationController
to find classes inheriting fromApplicationController
inside theapp/controllers
directory. Likegrep
,rg
can return both file, line number and match information. In this case, I am directing it only return filenames with--files-with-matches
.We now have a list of files in
app/controllers
that have classes inheriting fromApplicationController
.Next, we want to find uses of
current_user
in those files. There are a few ways we can accomplish that. I decided to find all controllers that usecurrent_user
, and later compare the two lists. The second list of files is found with:$ rg 'current_user' app/controllers --files-with-matches app/controllers/secure/password_resets_controller.rb app/controllers/secure/medical_profiles_controller.rb app/controllers/secure/bus_trips_controller.rb app/controllers/secure/profiles_controller.rb app/controllers/secure/signatures_controller.rb app/controllers/secure/bus_reservations_controller.rb app/controllers/secure/base_controller.rb
With the two lists in hand, we can then turn to
join
. From theman
page:join – relational database operator
The join utility performs an ``equality join’’ on the specified files and writes the result to the standard output.
Much like a
JOIN
in SQL which find corresponding records in two tables,join
can find matching records in two files. Its usage typically requires specifying which field in each line to use for the join. Our usage is very simple though: Our lists of files only have a single field: The file name.join
expects two files as arguments. We could redirect the output of each of ourrg
calls to a file, and use those files as input tojoin
. However,bash
(and other shells too) allow for process substitution: It can take care of presenting the output of a subprocess to another process as if it was a file. That is done via the<()
syntax, used twice: Once for eachrg
search.The last bit is the usage of
sort
.join
expects the files to be sorted:When the default field delimiter characters are used, the files to be joined should be ordered in the collating sequence of sort(1)
And there it is! We used
rg
,sort
,join
, and a bit ofbash
plumbing to find files that have lines matching two different regexes:$ join \ <(rg '< ApplicationController' app/controllers --files-with-matches | sort) \ <(rg 'current_user' app/controllers --files-with-matches | sort) app/controllers/secure/base_controller.rb
-
The REPL: Issue 83 - July 2021
GoodJob
I recently found
GoodJob
, a background-job library for Ruby. It’s compatible withActiveJob
. Its main selling points is that it takes advantage of Postgres features. For projects already on Postgres this means two things: There is no need for another data store, and job scheduling can be transactional with the rest of the application.Sidekiq, the leading background job library requires the use of Redis. It scales exceptionally well. For many applications managing another data store in production is burdensome, and provides little tangible benefits, especially if the load on the database is low. GoodJob even has a mode that runs the workers in the same process as the Rails server. For smaller apps running on Heroku, this can remove the need of a separate dyno.
Regarding the transactional nature: Suppose you want to store a record and queue a background job as part of some business operation. If you write to your main database first you run the risk of failing when enqueuing the job. Enqueuing inside a transaction doesn’t work either. In case of a transaction rollback, the job will still be published, like Brandur explains. Keeping jobs in the same database as the rest of the data, allows for transactional semantics – much easier to code against.
I’ve only tried this library on a side project with little traffic, but so far I am very impressed.
Idempotency-Key IETF Standards Draft
Speaking of Brandur: He notes that the IETF has a new draft standard for an
Idempotency-Key
, already in use at Stripe and other places. He previously explains in more detail why it’s important. -
Per-project Postgres with asdf and direnv
My development computer typically has a number of different projects, each needing specific versions of some tools. Previously, I wrote about using
asdf
to manage my ruby, elixir, crystal and erlang version. I’ve been using it successfully to manage Postgres versions as well.What I get
- Precisely manage the Postgres version (e.g.
10.14
,12.5
) for each project. - The Postgres data directory lives alongside other project files
- No collision between projects. Each can have their own Postgres server running simultaneously.
- Precisely manage the Postgres version (e.g.
-
The REPL: Issue 82 - June 2021
EXPLAIN ANALYZE in PostgreSQL and how to interpret it
I’ve been working on web applications for a long time. From time to time I am called upon to figure out (and optimize) performance issues on some query.
EXPLAIN ANALYZE
lets you see what Postgres is doing under the hood. I don’t do this often enough to remember all the detail of what all the information returned means. This article explains the basics ofEXPLAIN ANALYZE
and links to some handy tools that help you focus on what matters.Do You Need Redis? PostgreSQL Does Queuing, Locking, & Pub/Sub
It’s no secret that I like Postgres a lot. This article explains some common use cases for Redis, and how Postgres can take care of them instead. It resonates with me, especially for smaller projects were the techniques outlined can avoid using another data store.
Stop worrying about PostgreSQL locks in your Rails migrations
To round off today’s Postgres love, this blog post introduces how the
safe-pg-migrations
gem can help when running migrations for databases with high-load and/or a lot of data. Traditionally, those operations can be problematic because of how Postgres acquires locks and can bring production sites down. The specifics are explained in detail in the post. Thesafe-pg-migrations
gem manages to turn what would be complicated steps to ensure safe migrations into the same exact semantics thatActiveRecord::Migration
uses.As soon as I tried this gem, I ran into an issue. The maintainer was very helpful. It turns out Ruby 3.0 support was pending. A few days later my issue was resolved.
-
Better Form Objects in Rails
ActiveModel
was introduced in Rails with the promise of being able to create model classes that interact well with the rest of the Rails ecosystem (e.g. routing, forms), but are not backed by a database table.It has some shortcomings, which are addressed by the
active_type
. Let’s look at an example.