Building A Relational Database Using Kafka

Robert Yokota explores building a relational database on top of Kafka. It follows his previous article on creating an in-memory cache on backed by Kafka. RDBM systems are commonly thought of keeping track of tables and rows. The semantics of SQL reinforce the concept of rows being updatable. In practice though, most implementation use an immutable log under the hood. That is what makes transactions possible, each with its own consistent view of the world. Kafka can be thought of as an “exposed” MVCC system, and the current state of the data can be derived by consuming the messages in a topic. The article is interesting in that it assembles a relation database by using different existing open-source projects.

3 Key Ideas Behind The Erlang Thesis

Yiming Chen summarizes Joe Armstrong’s thesis: “Making reliable distributed systems in the presence of software errors”. The 3 key ideas identified: Concurrency oriented programming, abstracting concurrency, and let-it-fail philosophy. Armstrong is Erlang’s creator, and his thesis has been very influential in the Erlang and Elixir communities.

Issue 60! I’ve been posting my favorite links to tech articles every month for the last 5 years! I’ve linked to 163 articles in that time (not including the links in this post). And, now that I am looking back… I realize that I’ve made a mistake and I re-used #53 for the 2018-12 and 2019-12 issues. ¯\_(ツ)_/¯

Engineers Don’t Solve Problems

This article by Dean Chahim is not about software engineering or computer science. It’s about Mexico City’s infrastructure and the decades-long battle to prevent flooding in the city. The article stroke a chord with me: Mexico City is my home town, it’s where I went to University to obtain my degree in Civil Engineering. The article illustrates how engineers make trade-offs that might have far-reaching consequences, and are not immune from political and socio-economic influence. There are lessons there for all engineers.

How to Build Good Software

Software has characteristics that make it hard to build with traditional management techniques; effective development requires a different, more exploratory and iterative approach.

Li Hongyi writes a thoughtful article on why software projects are not the same as other engineering projects, and require different management techniques. Successful software projects are very iterative and oscillate between cycles discovery and consolidation.

Arcs of Seniority

Stevan Popovic breaks down engineering seniority into a few factors: Independence, authority, Design, and Influence. During once career each of these develops in an engineer, and mark different types of seniority. As expected, not everyone reaches the same maturity in all factors at once. Each senior engineer has it’s own mix. The illustrations on the articles are particularly helpful.

I have a love-hate relationship with spring, Rails’ application pre-loader. One one hand, it speeds up the feedback loop when doing TDD. Faster running specs, promote running them more often, which promotes writing code in smaller increments, and so forth. On the other hand, it is dark magic: In its quest to be unobtrusive, it starts automatically, and barely reports it’s being used at all. Occasionally it looses track of which code it needs to reload, causing much confusion to the user, as the code executing is different than the version saved on disk.

View-centric performance optimization for database-backed web applications

This post is a walk-through of of the academic paper with the same title. Keeping page-load time low continues to be important, but it has become an increasingly challenging task, due to the ever-growing amount of data stored in back-end systems. The authors created a view-centric development environment that provides intuitive information about the cost of each HTML element on page, along with the performance-enhancing opportunities can be highlighted. The goal is to make it easier to explore functionality and performance trade-offs.

Interestingly, the development environment, Panorama, targets the Ruby on Rails framework specifically. I look forward to trying it out soon.

Zanzibar: Google’s Consistent, Global Authorization System

This paper includes a thorough description of the architecture behind Zanzibar, a global system for storing and evaluating access control lists internal to Google. As a highly distributed system, it builds on top of other Google technology, like Spanner – a distributed NoSQL database. In particular, I was very interested in consistency model and how they provide guarantees around external consistency so that the casual ordering of events is maintained. It achieves this by providing clients with tokens after write operations (called a zookie): When a client makes a subsequent request with that token, the system guarantees that any results are at least as fresh as the timestamp encoded in the zookie.

The paper has a lot more, including how they architect for performance with caching layers, and a purpose-built indexing system for deeply nested recursive permission structures.

One of the reasons that I love TDD, is that it promotes fast feedback. You write a line, execute the tests, and see what the results are. I write outside-in-TDD most of the time. Occasionally, I don’t have a clear idea of what tests to write, or I am doing exploratory coding.