Bash Debugging

Julia Evans, in here signature comic/zine style, explains how to debug bash. I was today years old when I learned that you can have a step debugger for a bash script.

How we migrated from Sidekiq to Solid Queue

Chirag Shah at BigBinary explain in detail how they migrated an app from Sidekiq to SolidQueue. From the configuration it seems that the queue usage is on the low side (4 types of workers, each with a single process and 3 threads).

The article doesn’t mention in the article what they hoped to achieve or what their results where. I assume positive, because they close by saying they plan on doing the migration on more apps.

If I had to guess, I would say that they gained a simpler infrastructure (because they don’t use redis for jobs), but more importantly, they gained transactionality: All-or-nothing writes for the models and the jobs, which is impossible using 2 separate data stores.

How Figma’s Databases Team Lived to Tell the Scale

The folks at Figma tell their battle story about scaling Postgres. Having a DB proxy that checks the queries and routes to the correct shard (and even aggregates among shards) is wild.

The use of logical partitioning to prove their point before doing actual physical partitioning seems very clever.

I spend a lot of time on the command line. My most frequently used commands are git (by far), and rails (mostly for rails test) or rspec (depending on the project). Most invocation of both commands require arguments. And most of those arguments are paths to files. In big projects, typing paths to files can be tedious, even with tab completion. My productivity trick for that is using Zsh Line Editor widgets.

Let’s see a widget in action:

The world is awful. The world is much better. The world can be much better.

Max Roser’s point of view resonated with me:

If we want more people to dedicate their energy and money to making the world a better place, then we should make it much more widely known that it is possible to make the world a better place.

For this we have to remember that all three statements are true at the same time: the world is awful, the world is much better, and the world can be much better.

Useful Uses of cat

I agree with the author: Using cat to start a pipelines is about composability of commands, and makes it easier to build pipelines in steps. Technically, you could be adding an extra process that you don’t need, but in day to day unix pipe operations, the performance is never an issue.

Data Looks Better Naked

Great illustration of how bad data visualization can be better, by removing design elements. My favorite is the pie chart.

I recently discovered Nushell. It’s billed as a “new kind of shell”. What caught my eye is not so much the actual shell usage: Executing commands, redirecting output, working with files, etc. I was interested in the concept of structured data:

Everything is data

Nu pipelines use structured data so you can safely select, filter, and sort the same way every time. Stop parsing strings and start solving problems.

Nushell comes with support for a variety of formats (e.g. JSON, YAML, csv):

> open package.json | get dependencies
╭──────────────────────────────────┬─────────────╮
│ @fontsource-variable/caveat      │ ^5.1.0      │
│ @fontsource-variable/raleway     │ ^5.1.0      │
│ @fortawesome/fontawesome-free    │ ^6.7.2      │
│ @hotwired/turbo-rails            │ ^8.0.12     │
│ @popperjs/core                   │ ^2.11.8     │
│ @rails/actioncable               │ ^8.0.100    │
│ @rails/activestorage             │ ^8.0.100    │
│ @rails/ujs                       │ ^7.1.3-4    │
│ @ttskch/select2-bootstrap4-theme │ ^1.5.2      │
│ autoprefixer                     │ ^10.4.20    │
│ bootstrap                        │ ^4.5.3      │
│ esbuild                          │ ^0.19.3     │
│ jquery                           │ 3.7.1       │
│ nodemon                          │ ^3.1.9      │
│ popper.js                        │ ^1.15.0     │
│ postcss                          │ ^8.4.30     │
│ postcss-cli                      │ ^10.1.0     │
│ sass                             │ ^1.67.0     │
│ select2                          │ ^4.1.0-rc.0 │
│ startbootstrap-sb-admin          │ ^6.0.2      │
╰──────────────────────────────────┴─────────────╯

Pipes can be built up:

> open package.json | get dependencies | transpose | where column0 =~ @rails
╭───┬──────────────────────┬──────────╮
│ # │       column0        │ column1  │
├───┼──────────────────────┼──────────┤
│ 0 │ @rails/actioncable   │ ^8.0.100 │
│ 1 │ @rails/activestorage │ ^8.0.100 │
│ 2 │ @rails/ujs           │ ^7.1.3-4 │
╰───┴──────────────────────┴──────────╯

Converting between formats is a breeze:

> open package.json | get dependencies | to yaml
'@fontsource-variable/caveat': ^5.1.0
'@fontsource-variable/raleway': ^5.1.0
'@fortawesome/fontawesome-free': ^6.7.2
'@hotwired/turbo-rails': ^8.0.12
'@popperjs/core': ^2.11.8
'@rails/actioncable': ^8.0.100
'@rails/activestorage': ^8.0.100
'@rails/ujs': ^7.1.3-4
'@ttskch/select2-bootstrap4-theme': ^1.5.2
autoprefixer: ^10.4.20
bootstrap: ^4.5.3
esbuild: ^0.19.3
jquery: 3.7.1
nodemon: ^3.1.9
popper.js: ^1.15.0
postcss: ^8.4.30
postcss-cli: ^10.1.0
sass: ^1.67.0
select2: ^4.1.0-rc.0
startbootstrap-sb-admin: ^6.0.2

And there is a built in explore TUI command that you can pipe data to (open file.csv | explore) to drill in and out of data interactively. Lovely.

I’m just getting started with Nushell, but already seems like a useful tool to have for data exploration and transformation.

Cheating is All You Need

Steve Yegge writes (a lot!) about AI. His take is very interesting: Basically LLMs are in a race to the bottom. Each LLM is not that different from the next. The important bit is about context: Providing the LLM with enough context to do the task at hand. I’ve only had limited experience playing around with LLMs, but I definitely think that they only work when they are provided enough context to perform the task at hand. Providing can be tedious. If you want it to help you write a unit test, well it needs access to the class in question. And it’s collaborators. And the factories that the test uses. And so on. Copy/pasting all of that is a lot. The more interesting claims about AI I’ve seen are for tools that integrate with your editor (or IDE) to provide the context to the LLM.

Nearly All Binary Searches and Mergesorts are Broken

Interesting take about algorithm correctness: The author wrote a buggy implementation of a binary search in the Java language. And the bug, is the same one that his PhD teacher had in his book! The bug was for some ranges of inputs (not all), which is why it’s so pervasive and hard to spot, until you specifically start to look for it.

We programmers need all the help we can get, and we should never assume otherwise. Careful design is great. Testing is great. Formal methods are great. Code reviews are great. Static analysis is great. But none of these things alone are sufficient to eliminate bugs: They will always be with us. A bug can exist for half a century despite our best efforts to exterminate it. We must program carefully, defensively, and remain ever vigilant.

The Mythical IO-Bound Rails App

The author makes good arguments that if Rails in the wild was really IO bound all the time, than the recent YJIT improvements in Ruby versions would not have resulted in reported ~25% improvements in real apps. The improvements suggests that a good chunk of the time was being spent in Ruby. He then goes to point out that actually measuring IO is complicated, and that CPU starvation can look like IO.