Ylan Segal

Chasing a Segmentation Fault

Recently, I chased down a segmentation fault occurring in one of our production servers. A segmentation fault cannot be triggered by code is that written completely in Ruby, barring a bug in Ruby itself. The VM manages the memory, making it impossible to access memory in violation of the OS rules.

In computing, a segmentation fault or access violation is a fault, or failure condition, raised by hardware with memory protection, notifying an operating system the software has attempted to access a restricted area of memory.

Not surprisingly, I tracked it down to a gem with a C extension: A database driver for MS SQL. The issue can be reproduced by attempting to read results from a connection after it has been closed. I don’t expect to be able to read the results, but I expected an exception to be raised, not for the whole process to crash. I reported the bug.

Interestingly, I can also reproduce the segfault by running the garbage collector (GC) manually. The way we interact with the gem is by instantiating a TinyTds::Client object, and executing some SQL. It then returns a TinyTds::Result object. The segfault is triggered when (1) the client object is no longer in scope (thus eligible for garbage collection), (2) the GC runs, and (3) then the result object is used. Since normally the GC runs at Ruby’s pleasure, we see non-deterministic segfaults in production, with varying stack traces.

The gem hasn’t been fixed yet, but I believe I can solve our particular issue by re-organizing my code so that the client and result objects are always in scope at the same time. The most expedient solution was to read all the results into Ruby as soon as possible. I was concerned that this would increase the memory usage. This was a good opportunity to use benchmark-memory.

1
2
3
4
5
6
7
8
require 'benchmark/memory'

Benchmark.memory do |x|
  x.report('with workaround') { Existing.run  }
  x.report('without workaround') { Workaround.run }

  x.compare!
end

Its output is handy.

1
2
3
4
5
6
7
8
9
10
11
Calculating -------------------------------------
     with workaround     1.338B memsize (    19.558M retained)
                        17.715M objects (     6.234k retained)
                        50.000  strings (    50.000  retained)
  without workaround     1.397B memsize (    12.303k retained)
                        18.828M objects (   109.000  retained)
                        50.000  strings (    19.000  retained)

Comparison:
     with workaround: 1337820417 allocated
  without workaround: 1396586267 allocated - 1.04x more

Looks like there was no need for concern.

RailsConf 2019 Talk: Bug-Driven Development

Last week I had the pleasure of attending RailsConf 2019 for the first time, and the honor to be a speaker. My talk was titled “Bug-Driven Development”. On the surface it’s ostensibly a war story about fixing a particularly nasty bug. At a deeper level, it is about software design evolution. Software is an iterative endeavor, perpetually in a state of flux. Requirements change, new features are added, external APIs are deprecated, scaling demands adjustments. In the talk, I try to thread the needle between a specific bug fix and the broader applicability of design patterns, proper abstractions, and the role of testing. My goal was for audience members to be able to see both the proverbial forest and the trees: to connect the ivory-tower, abstract design concepts with the day-to-day practice of writing code, test-driven development, and fixing bugs.

RailsConf itself was a great experience. The organizers did a great job, both from an attendee, and a speaker perspective. Attending the conference gave me an opportunity to learn about new topics, dive deeper on familiar ones, connect with other rubyists, and hopefully contribute my own grain of sand to the community from which I have gained so much. This may have been my first RailsConf, but I am sure I’ll come back in the future.

Last — but not least, I am grateful of all the support received from Procore. The continuous drive to engineering excellence is inspiring and contagious. Its commitment and investment to the personal development of the engineering team made this talk possible.

Links:

The REPL: Issue 56 - April 2019

Technical Debt Is Like Tetris

Eric Higgins makes an analogy that resonates: Technical debit is like a Tetris. It accumulation can always be seen, but it not necessarily detrimental until there is enough of it. At some point it can make it impossible to make progress, or even to keep the game going. I guess starting a new game is equivalent for rewriting your software :-).

Storing UTC is not a silver bullet

One doesn’t have to program for long before encountering the sharp edges of date, time, and time zones handling. For a lot of use cases, storing date/time in UTC is a good solution. In this post, Jon Skeet explains why it doesn’t solve all the possible problems. The biggest takeaway is that Time zone definition changes constantly, and if your code doesn’t account for it explicitly, it will do so implicitly – sometimes with unexepcted results.

Using GNU Stow to manage your dotfiles

I manage my Unix dotfiles with care, manage them with source control, and take them with me from machine to machine. I spend a lot of time in my Unix environment and my configuration allows me to be productive. Until recently, I managed my dotfiles with a hand-crafted script that created symlinks. In the past I tried a few different solutions for the same problem, but nothing worked well for both files, directories and arbitrary locations. In this post, Brandon Invergo explains how to leverage a new-to-me utility: GNU stow. It’s purpose is to manage symlinks, usually in the context of activating different version of the same software in the same system. It works great for dotfiles!

The REPL: Issue 55 - March 2019

What causes Ruby memory bloat?

Hongli Lai writes a provocative post about his research into Ruby memory bloat. Long-running Ruby process, typically grow in their memory usage over time. It’s common to see Ruby servers being re-started once a day (like on Heroku) to avoid this very issue. In this post, the author proposes that the reason behind memory bloat is not memory fragmentation, like a lot of people in the community assume. Instead, he points to the memory allocator (malloc) not releasing memory. I am not very familiar with memory allocation, so I am left wondering if other languages that also use malloc suffer from the same issue as Ruby.

HIML

Akira Matsuda released a new templating engine with a novel take. The standard template engine in Ruby is ERB. It’s the default in Rails when rendering HTML. A user typically starts with HTML and sprinkles in special tags that get processed as ruby and replaced. HAML, is an alternative that was designed to avoid writing inline code in a web document. It always generates valid HTML, because it functions as a DSL. The down-side is that one must learn the new syntax. That is where HIML comes in: It produces valid HTML and provides some of the conveniences of HAML (e.g. autoclosing tags), but as intuitive as HTML and ERB.

1
2
3
<section> <!-- This will auto-close! -->
  <div> <!-- As will this -->
    <%= post.content %> <!-- Standard way to interpolate ruby code -->

Defining a Distinguished Engineer

Jess Frazelle explains the qualities that a distinguished engineer or technical fellow should exhibit. The list is solid and resonates with my views. My only bone to pick is with the often-repeated adage: Have strong opinions, loosely held. I don’t think that is quite accurate. Opinions held in strength proportional to the weight of the evidence doesn’t have the same ring to it. I understand if it doesn’t catch on. Note that the actual description under the heading is spot on:

They do not need to have opinions on everything, that would be pedantic. Technical leaders should be able to use their experience to help others succeed, while also empowering others to own solutions.

On Being Efficient: My Mac Setup

As a Software Engineer, I spend most of my day working on a computer. Over the years, I’ve come to customize my setup to align better to the way I work and become more efficient. Efficiency is important because of the time you save, but not because you can do more with the time saved. It’s allows you to keep focus on what you want to accomplish, instead of the mechanics of accomplishing it.

The following describes what I use on my Mac, but the principles apply to any operating system with a graphical interface. The metaphors they use are mostly the same: desktop, windows, application, a dock, etc.

Application Launcher

I launch apps through an application launcher. It involves a keyboard shortcut to invoke the launcher, typing a few keys (usually 1 to 3) to fuzzy find the app I am looking for, and hitting enter. For me, typing 3 to 5 keys is very fast. I have been using Alfred for many years, but I’ve know people that thing the built-in Spotlight works just as well for this purpose.

Switching Applications

MacOS (formerly OS X) includes a keyboard shortcut to switch applications: cmd + tab. Hitting it once and releasing switches you immediately to the last application used. It use this the most to switch back-and-forth between my editor and terminal, documentation and editor, etc.

Hitting cmd+tab, but keeping cmd pressed, brings up a modal list of the icons of all running application, ordered by how recently they have been used. While still holding cmd, pressing tab moves the selector to the next app in line (shift + tab moves back one spot). When cmd is let go, the selected app moves into focus. In this way, it becomes really fast to switch to another running application.

Alternatively, instead of finding the running app, I sometimes just use the application launcher. If the application is already running, it just switches to it.

For applications that use more than one window, you can use cmd + ` to cycle between them. I use this often when I have more of one window for my text editor.

Hide the Dock

I don’t use my Dock. I have it set vertically to the far right of my screen and it’s hidden by default, unless I move my mouse over it. This doesn’t happen a lot, and usually it’s by mistake. If there was a way to completely turn it off, I would.

As far as I can see, the main raison d'être for it is to launch favorite applications (see above), to switch applications (see above), or to see app badges (e.g. unread count for email). I limit aggressively the number of badges I allow in my applications. For those that I do use, it’s enough for me to see them when switching applications.

As a side benefit of not having a permanent Dock, is that I get some of my screen real-estate back.

Clipboard History

The built-in clipboard works very simply: When you paste cmd + v it inserts the last thing you copied cmd + c or cut cmd + x. A clipboard with a history, maintains a history off what you have copied and allows you pasting from it. This is one of those things that after using for a short while you wonder how you ever managed without.

Alfred provides that functionality for me as well. It’s implementation is particulary nice, since it doesn’t modify the standard paste command. Instead, it has a keyboard shortcut to open the history and a fuzzy finder to find entries. This works really well. A particularly nice touch is that it’s configurable to not add things to the history from an application, like a password manager.

Windows Tiling

The traditional way of positioning and sizing windows in MacOS is by using the mouse. It’s slow and difficult to get just right. A window-tiling application let’s you snap your windows into predefined positions. I use 3 sizes: Full screen*, half-screen vertical, third-screen vertical. I align them left, right, or centered as needed. My preferred tool is Spectacle which allows configuring all of this with keyboard-shortcuts.

I don’t use MacOS full-screen mode: It removes the menu bar, which I like. More importantly it has an animation going to and from that is slower than just switching applications. I find that distracting. Instead I use Spectacle to enlarge the screen to fill the screen (equivalent of option-clicking on the green dot on a standard Mac window).

Shortcuts for Text Editing

Last, but not least. I type a lot of text: Code, emails, ticketing systems. wikis, GitHub, etc. By default, all Mac text input areas allow the use of the same text editing shortcuts. For example, moving to the begging of the current line (cmd + left-arrow), moving back a word (option + left-arrow), delete previous word (cmd + delete), etc. Those shortcuts make text editing much faster. Knowing those shortcuts is one of the reason why I don’t learn vim.


Customizing your working environment, even in small ways can have a big cumulative effect. I don’t recommend trying all of the above at once, but focusing on one at the time. Try it out for a while and tweak to your liking until it feels like a natural part of your workflow. For me, that can be anywhere from 2 to 4 weeks.