Under Deconstruction: The State of Shopify’s Monolith

The engineering team at Shopify discuss the state of their Rails monolith, how it has evolved over time, the lessons they’ve learned and what is in store for the future. Most of the information is relevant for Rails developers working in large systems, with large teams.

As part of their efforts to make their monolith more effective, they are introducing a newly-open sourced tool: Packwerk. The objective is to enforce modularity, through the use of static analysis.

Writing a book: is it worth it?

Martin Kleppmann discusses openly the economics of writing his book: Designing Data-Intensive Applications. The book has been one of my favorite technical books. The book has generated almost $500,000. It involved working on it for years – one of them without any other income, speaking at more than 50 conferences promoting the book, and was helped in no small part because of Kleppmann’s well deserved presence and reputation in the field.

Error handling with Monads in Ruby

Vitaly Pushkar goes over error handling in programming languages and the reasoning for using monads for error handling. I’ve been experimenting with that at work record with very positive results – albeit in a limited portion of our code.

There is a section where the author talks about the “pyramid of doom” – much like the dry-monads documentation. The “do notation” is presented as a solution. Maybe it’s my lack of familiarity with that notation, but I think that there is a simpler solution, which I call “railway oriented”.

Pyramid of Doom:

def call(fields)
  validate_fields(fields).bind do |fields|
    validate_email(fields['email']).bind do |email|
      find_user(fields['id']).bind do |user|
        update_user(user, {name: fields['name'], email: fields['email']}).bind do |user|
          send_email(user, :profile_updated).bind do |sent|
            Success(user)
          end
        end
      end
    end
  end
end

Do notation:

def call(fields)
  fields = yield validate_fields(fields)
  email = yield validate_email(fields['email'])
  user = yield find_user(fields['id'])
  user = yield update_user(user, {name: fields['name'], email: fields['email']})
  sent = yield send_email(user, :profile_updated)

  Success(user)
end

Note that the yield in the above solution differs from the standard Ruby meaning. It doesn’t yield to the block passed to #call but rather binds the Result and halts execution of the method on failures.

My “railway oriented” solution:

def call(fields)
  validate_fields(fields).
    bind { |fields| validate_email(fields['email']) }.
    bind { |_email| find_user(fields['id']) }.
    bind { |user| update_user(user, {name: fields['name'], email: fields['email']}) }.
    bind { |user| send_email(user, :profile_updated) }
end

The “pyramid of doom” is avoided, without having to change the Ruby semantics.

How to stop procrastinating by using the Fogg Behavior Model

The Fogg Behavior Model is a new concept for me. It presents a mental model in which:

Behaviour = Motivation + Ability + Trigger

A lack in any of the three factors can prevent behavior from occurring. The articles then talks about different strategies to boost each factor. For example, blocking time in your calendar can serve as a trigger to start working on a particular task. I’ve been very successful with that technique lately.

Simple Made Easy

This talk is not new, but it is new to me. I’ve seen reference to this talk in many places, but hadn’t watched until recently.

Rich Hickey emphasizes simplicity’s virtues over easiness’, showing that while many choose easiness they may end up with complexity, and the better way is to choose easiness along the simplicity path.

This is a fantastic talk! It describes how software that is simple is the ultimate goal. Simple is achievable, but it is not the same as easy. It goes hand-in-hand with choosing the correct abstractions and planning beforehand. It’s not about hiding complexity, it’s about abstracting it away. It reminds me of something that I distilled from somewhere else as:

An abstraction removes from your mental load. Indirection adds to it.

An abstraction lets you forget about the details and allows higher order thinking. Indirection forces you to think about the details constantly.

Result Objects - Errors Without Exceptions

Tom Dalling of Ruby Pigeon introduces his library Resonad. It aims at dealing with error handling without exceptions. The article gives a good overview why and when to use result objects. There are other libraries in the Ruby ecosystem (like dry-monads). As the author points out, it doesn’t actually take a lot to build your own. I did so recently and was pleasantly surprised how far a very simple implementation can take you:

class Failure
  attr_reader :error

  def initialize(error)
    @error = error
  end

  def successful?
    false
  end

  def failure?
    true
  end

  def and_then
    self
  end
end

class Success
  attr_reader :value

  def initialize(value)
    @value = value
  end

  def successful?
    true
  end

  def failure?
    false
  end

  def and_then
    yield(value)
  end
end

Success.
  new(1).
  and_then { |v| Success.new(1 + 1) }.
  and_then { |v| Failure.new(:boom) }.
  and_then { |v| Success.new(1 + 1) }
# => #<Failure:0x00007f9eda02f280 @error=:boom>

At work, I am evaluating the usage of result monads as part of some work to better encapsulate domain logic. I am expecting it will be very beneficial in how we handle errors.

Why Your Microservices Architecture Needs Aggregates

Dave Taubler writes a comprehensive piece about the use of Domain-Driven Design concepts in micro-service architectures. The use of aggregates, entities and invariants can prevent unwanted dependencies between objects, leaky references and delineate a clear boundary around groups of data. Eventually, the use of aggregates simplifies things like sharding, message passing, idempotent retries, caching and tracking changes.

I’ve been thinking about event schema design a lot lately, and found interesting that in the context of messaging between services via a message broker, the author advises to use the same aggregate!:

So what should we pass as our messages? As it turns out, if we’ve embraced Aggregates, then we have our clear answer. Anytime an Aggregate is changed, that Aggregate should be passed as the message.

Why Tacit Knowledge is More Important Than Deliberate Practice

The author talks about tacit knowledge:

Tacit knowledge is knowledge that cannot be captured through words alone.

Most of the article is new to me: The idea that there is knowledge that can’t be expressed through words alone, and that is distinct and separate from deliberate practice.

I can relate to the part about expertise: An experience software engineer can come up with a good design – or critique an existing one – in seconds. Clearly, they is not going down a list of heuristics one by one. In their mind, the pattern have been established and the brain quickly comes up with a solution. This is the subject of “Blink” by Malcom Gladwell.

My sense is that as a person gains expertise, they gain intuition about the field, and their brain gets wired for rapid pattern recognition. They reach this stage before they can put in words why their intuition went in that direction. That actually comes in a later part of expertise, when the person can articulate and reason about the intuition and communicate it to others.

I believe the author is saying that because people can’t articulate their intuition, that means that they hold tacit knowledge that can’t be articulated. I think that is a flawed syllogism. In fact, the author points at examples of fields were it was previously thought that apprenticeship was the only way to transfer tacit knowledge. Then someone came along and turned that knowledge into explicit knowledge.

I believe that knowledge can be gained by learning explicit and implicit knowledge. On one end, reading books and academic material goes a long way, but at times it can be disconnected from application. The proverbial ivory tower. On the other end, there is apprenticeships that focus on following rules and procedures, without real understanding, that can nevertheless bring proficiency: Most people learn to play the guitar like this. Of course, there is a hybrid: Academic project-based learning that aims to cover both modes of learning. In my experience all 3 can work, even for the same person. It depends a lot on familiarity with the material adjacent to the new material that we are learning, and how the new materials fall into the existing subject’s mental model.

There are interesting points in the article. My takeaway is that the real jump in understanding is when you can turn tacit knowledge into explicit knowledge, which is distinct from the author’s point about embracing tacit knowledge.

Schema evolution in Avro, Protocol Buffers and Thrift

Martin Kelppmann goes writes about schema evolution in binary protocols – namely Avro, Protocol Buffers, and Thrift. Schema evolution is an important concern when building systems connected via event stream and immutable logs. This post inspired me to dig deeper about Avro Schema Evoltion

Why does writing matter in remote work?

The pandemic has shifted a lot of people to working remotely. Writing is an important part of remote work.

While writing forces people to think clearly, writing also forces teams to think clearly.

I think this is one of the most important points in the article: Writing things down helps develop a train of thought and connect things together. Holes in logic or implicit conjectures become evident. Writing for the consumption of others, increases this effect.

I’ve found that collaborating on well-written work is easier than collaborating on work that is hard to follow, doesn’t spell out assumptions, doesn’t show examples, etc.

Tools for better thinking

Collection of different tools for problem solving, systems thinking, and decision making. Some are new to me, but all interesting and useful in separate situations. Great to have as a reference.