Ylan Segal

The REPL: Issue 30 - January 2017

The four indicators of a healthy Ruby On Rails project

Software consultants often need to evaluate existing projects. Błażej Kosmowski writes a thoughtful post on four things he focuses on: Outside-In TDD, use of design patterns, a balance between DRYness and readability and automation (around code quality). I like all of those and would add a few more around project deployment, like automation of deployment, staging to production promotion of code, etc.

The Disciplines of Web Design

Joshua Ogle writes a good introduction of what the different disciplines of web design are (visual design, user interface, user experience) and how to interact with each other. I found it very informative. I usually focus on backend programming, but work with designers and their concepts often.

Instrumentation: The First Four Things You Measure

In the context of computer programming, instrumentation refers to an ability to monitor or measure the level of a product's performance

Instrumentation provides insight into what exactly is happening to your running code. Antoine Grondin lays out what are the basics of what to measure, to be able to quickly understand the situation in times of trouble. From experience, I can say that just modest investments in instrumentation provide great value.

The REPL: Issue 29 - December 2016

Learn Graphviz and Up Your Diagramming Game

I’ve bumped with Graphviz in the past only indirectly. After reading this article, I will definitely keep it in mind when needing to create diagrams in the future. I like the idea of specifying the relationships in a diagram in text – and manage it with source control – and then generating a graph from it.

Microservices? Please, Don’t

Sean Kelly explores why some of the often-repeated arguments for microservices. Certainly, some of the benefits of microservices can be achieved without needing to separate them into different web applications, which in itself can brings complications in deployment, coordination and increased network interaction.

RbNaCL: The Ruby Cryptography Library

This repository is a great idea: Provide a cryptographic library that makes it easy and straightforward to use high-level cryptography correctly, and avoid the many pitfalls of trying to assemble a secure system from cryptographic primitives. The APIs are designed to provide abstractions like public-key / secret-key encryption, digital signatures, etc.

Enforcing Style

Most programming languages provide some flexibility on what they consider valid syntax. Take a boring piece of code:

1
2
3
4
5
def print(values:)
  values.each do |value|
    puts "[#{Time.now}] #{value} processed"
  end
end

In those four lines of Ruby code, there are plenty of style decisions. I used a keyword argument, but could easily have used a standard argument. I used parenthesis for the method definition, even if Ruby will parse the line just fine without them. I used a do end block, but could have used {}. I used 2 spaces for indentation. I used double-quoted strings with interpolated values.

A programming style guide makes explicit what you intend to adhere to. It’s largely about esthetics: What looks like Ruby. What feels like Ruby. It can also be about semantics and conventions. I can’t objectively defend all of my style preferences, because they are not objective. They are a balance between readability, efficiency, verbosity, etc. Don’t mistake subjectivity with lack of value.

When team members work on the same code base, but use wildly different style, there is often an unspoken turf war in source files. Everyone makes changes in their own style. It can either results in inconsistencies that are unsatisfying and confusing, or constant noise in pull request that flip from style to style. More subtly, and more concerning, it can also cause team members to treat large portions of the code as someone else’s problem. Lack of ownership can quickly deteriorate into dysfunction. I would characterize lack of a common style as a symptom of larger issues in team dynamics, not a cause.

The process of adding a style guide, can help resolve some of the underlying issues, by forcing the team to agree on something that is largely a matter of personal preference. The process should be inclusive and respectful of all team members. It will require compromise. The goal is to make a stronger team. Better looking code is nice, to.

Creating Your Own Style Guide

Now that you have decided – as a team – to embark on creating a shared style guide, let’s get to the details. First, creating a style guide can be a daunting. I recommend starting with the comprehensive Ruby Style Guide maintained by Bozhidar Batsov. Start by forking the style guide into your own project. This will be the basis of the conversation for your team. Use the same tools you use to work as a team – version control and pull requests.

At this point, the people work begins. Maybe you don’t like sometimes using single-quoted strings and sometimes double-quoted strings. Or you think that 80 characters is ridiculous as a line-length-limit for modern screens. Maybe I think those things. In any case, the important part here is to start a civil, cordial conversation with your teammates on what conventions you want to follow and adopt as your own. Coming to an agreement and establishing ownership are the most important benefits of this whole exercise. Expect this process to take some time and to be include a good amount of bike shedding. A lot of these decisions are subjective and arguments for them are not much better than personal preference. Stick with it.

Checking Style

Bozhidar Batsov also maintains an amazing gem called rubcop. It’s a static code analyzer, that seeks to enforce the contents of the style guide. It includes many different “cops”, each enforcing different aspects of the code. Examples are Metrics/LineLength, Style/EmptyLines or Style/EmptyElse. rubocop is highly configurable and extensible.

Your first step, is to run rubocop without any configuration in one of your projects. I recommend starting with a small project. rubocop is very opinionated and there are bound to be many offenses in any given source file.

1
2
3
4
5
6
7
8
9
10
11
$ rubocop --show-cops example.rb
Inspecting 1 file
C

Offenses:

example.rb:1:1: C: Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols.
"This is a line"
^^^^^^^^^^^^^^^^

1 file inspected, 1 offense detected

In this first phase, our goal is to hand-craft a configuration that reflects the text in your style guide. Because of that, I used the --show-cops option. It will show the name of the cop that is marking the offenses. Each cop can be configured (or disabled). Look in the excellent documentation for information on each one.

Once familiar with the syntax, you can start adding entries to a .rubocop.yml file in the project root directory. At this point, I don’t recommend making any changes to source files to comply with cops. That will come later. A simple configuration, looks like this:

1
2
3
4
5
Metrics/LineLength:
  Max: 120

Style/StringLiterals:
  EnforcedStyle: double_quotes

Auto-correcting

rubocop has the ability to auto-correct some of the cops. This helps with some of the tediousness of having to make many edits. I recommend running a single cop at a time with auto-correct. I find it easier to inspect the changes that way before committing, as well as smaller commits with better descriptions.

1
$ rubocop --auto-correct --only Style/StringLiterals

Ensure that you review your changes before committing. The last thing you want is to introduce bugs into your application. You do have a good test suite, right?

Managing Large Code Bases

Sometimes, for very large projects, removing all style offenses can be a daunting tasks. rubocop allows you to declare a bankruptcy of sorts, to let you keep moving in the new direction. The idea is that current offenses will not be flagged, but new ones will. You can auto generates a “to do” file, that marks current violations and make a plan to address them later. For example, if editing a file in the future for a different reason, you can make a commitment to fix the style first, and then do the original work. That way, correcting the style becomes a part of the daily routine and not a larger-than-life tasks.

1
$ rubocop --auto-gen-config --exclude-limit 99999999

The above will generate a .rubocop_todo.yml with exclusions to current offenses. Using that file with rubocop will result in zero offenses, by definition. The --exclude-limit is set to a high number, so all exclusions are logged, instead of whole files being ignored. As with your regular rubocop config, this should be checked in to version control.

As you fix existing offenses, be sure to run --auto-gen-config to ratchet down the exclusions in your project. Decide with your team when the “ratcheting” will happen (e.g. on each pull request, once a week).

Continuos Integration

You now have everything in place to enforce style on a project. As with specs or unit tests, you want to run the enforcement often and fail loudly. Your CI server is the perfect place for it. If your project does not have an .rubocop_todo.yml file, then all you need to do is to invoke rubocop on the build.

If you do have a .rubocop_todo.yml file, I recommend that you create a specific configuration for CI, like so:

1
2
3
4
# .rubocop_ci.yml
inherit_from:
  - .rubocop.yml
  - .rubocop_todo.yml

And configure CI to use it:

1
$ rubocop --config .rubocop_ci.yml

This will pick-up your settings from .rubocop.yml and then apply any exclusions in .rubocop_todo.yml. For local development, I have my editor run rubocop when I save any file, but pointing to the regular .rubocop.yml file. That way, I can see any offenses on any file I work on and are reminded to fix them as I go.

I am currently using Atom with the linter-rubocop package.

Sharing Rubocop Configuration Among Projects

If you are working in multiple code bases, you probably don’t want to manually maintain separate configuration files for each project and have sync changes to your style config for each one. rubocop provides a few mechanisms to deal with this problem. My favorite is it’s ability to inherit configuration from a file inside a gem. This allows us to create a gem that holds the configuration and use it from all our other projects. This not only removes the duplication, but you can also take advantage of gem versioning to control when new style guidelines are brought into each project. Of course, the downside of that is that you still need to remember to upgrade the gem version on each project using it. I see that as less of an issue, because I recon that most teams already have a process in place to update their projects dependencies on a regular basis.

A .rubocop.yml file with gem inheritance:

1
2
inherit_gem:
  your_team_style_gem_name: .rubocop.yml

rubocop itself changes often and new versions tend to bring in new cops that where not there before. Even if your config or your code hasn’t changed, updating rubocop can cause new offenses to show up. That is undesirable, especially if running on CI. However, by managing rubocop config in a gem, you can also pin the specific version of rubocop in the gemspec.

Conclusion / Parting Thoughts

The process of deciding on a style guide and enforcing it can be difficult. A team challenge, as opposed to a technical one. It can bring a lot of value to a team. It’s about the journey. The destination can be pretty nice, too.

Renewing a Let's Encrypt Certificate

I previously wrote about changing my certificate authority to Let’s Encrypt. About the only downside I found about using it with my hosting service, Nearly Free Speach, is the need to manually renew every 3 months. Today, I went through the process and found it to be relatively simple.

To renew the certificate, I ran certbot like this on my machine:

1
$ sudo certbot certonly --manual

This initiated the in-terminal process, with instructions on adding some specific content to the domain to ensure you control it. After the verification the certificates where issued and placed in /etc/letsencrypt/live/ylan.segal-family.com/.

The generated certificate contents looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
$ openssl x509 -in cert.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:b3:26:9f:ed:b0:58:d7:57:6f:ba:0d:0c:8e:85:cb:f0:d8
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Dec  7 20:42:00 2016 GMT
            Not After : Mar  7 20:42:00 2017 GMT
        Subject: CN=ylan.segal-family.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:b2:3c:2d:4e:2b:cc:ae:c2:75:08:43:08:f7:2d:
                    cc:fa:05:06:36:e6:06:9f:48:1a:71:a2:aa:c0:4f:
                    01:9d:e7:b2:a0:1b:18:ec:48:ef:59:ec:98:93:89:
                    e5:7a:a7:e4:62:b1:32:85:cf:60:72:40:81:46:71:
                    b8:6f:8d:d1:bb:5d:7d:d4:cd:9c:49:ad:94:8d:ba:
                    98:ad:01:db:d1:7f:f9:98:e3:c2:50:43:97:50:f0:
                    b0:a8:58:8a:e9:5d:f3:d9:88:4a:63:77:4a:06:b2:
                    5d:16:a2:66:6d:1d:b7:2b:c2:90:8f:30:90:18:3d:
                    23:09:8d:fb:07:4c:32:c5:bf:3b:3a:3b:fd:f5:49:
                    7e:e9:2e:82:e5:31:59:3c:b7:c3:e8:07:b9:a8:b6:
                    c7:11:f0:53:36:0a:d9:58:a5:26:09:42:51:b7:9c:
                    78:8c:c0:01:e8:0d:44:7c:eb:66:c8:b4:49:09:22:
                    69:48:94:68:9c:d5:ce:c1:9d:bf:e1:b1:4c:b3:ff:
                    f2:eb:c0:66:e4:7b:1a:1c:4e:24:71:bf:f5:e9:8a:
                    bb:8d:e7:50:5c:f6:01:32:09:53:fd:fe:2f:96:eb:
                    f2:6a:44:7f:dc:5e:f6:c6:18:1a:02:99:b2:0b:45:
                    53:25:b2:97:1c:c4:67:61:99:2b:d7:2a:d6:52:e0:
                    90:2b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                17:5A:1E:01:42:05:20:F2:CA:AE:CB:BA:84:49:DB:53:02:7D:76:3E
            X509v3 Authority Key Identifier:
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access:
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org/
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name:
                DNS:ylan.segal-family.com
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org
                  User Notice:
                    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

    Signature Algorithm: sha256WithRSAEncryption
        35:98:2b:c4:ed:e3:93:5b:2a:61:f0:cb:1d:23:3f:d8:15:79:
        eb:92:f4:79:e9:a2:31:2b:d1:35:bc:0c:5d:89:ad:ec:ed:56:
        2c:d2:77:bc:f0:19:64:7c:04:9b:76:6c:16:84:23:b5:94:9a:
        74:2e:2e:3c:18:47:ee:73:6e:d9:b5:2c:dd:89:c5:1e:ec:a4:
        c0:c4:e7:8c:35:9f:a0:af:ec:87:ea:78:81:45:6b:e5:db:b3:
        60:0a:02:08:4e:92:5a:da:5c:d8:95:d3:45:7d:d7:3f:07:2e:
        0c:a3:dc:a8:4f:1a:e8:e7:9b:d7:09:2e:d7:f3:2c:c2:c0:ba:
        78:70:11:12:62:37:80:e6:e3:cb:a0:04:e6:19:f3:0a:eb:74:
        64:50:e6:90:e5:60:32:f6:f6:d4:e8:db:94:6a:47:76:25:34:
        23:e0:13:2b:19:a0:de:33:7c:33:a2:fb:7e:03:79:d6:30:ab:
        f8:85:3d:27:3c:d4:69:9a:f9:da:b4:5c:88:b5:1c:95:5d:64:
        8f:2c:26:eb:73:f9:08:4c:ec:30:d8:91:82:a7:0a:ed:9c:82:
        b5:24:b7:54:38:04:61:47:5e:ac:02:83:81:cf:d7:29:d6:74:
        b2:90:3d:0c:75:2a:e3:12:f2:3d:53:96:9d:ca:48:c4:bc:b3:
        a8:94:5a:d2

As was the case when first installing the certificates, Nearly Free Speach requires them to be concatanated and pasted into their form:

1
$ cat fullchain.pem privkey.pem | pbcopy

The changes take place immeditely. I now have 3 more months of a valid TLS certificate for this web-site.

The REPL: Issue 28 - November 2016

Open-Sourcing Yelp’s Data Pipeline

The Yelp Engineering team has been posting regularly about they structure data consumption between different teams. The backbone of their system is Apache Kafka, but they have created a lot of tooling around it. In this announcement, they have open sourced (Apache License 2.0) many of these tools. MySQL Streamer pipes data from MySQL to Kafka. Schematizer stores and tracks the various data schemas used throughout their pipeline. There is a lot to learn in this projects and the blog itself, which provides an overview of how they approach dealing with data.

Offshoring roulette: lessons from outsourcing to India, China and the Philippines

Troy Hunt writes a lengthy post about his experience offshoring development work to teams in India, China and the Philippines. He goes through the motivation for offshoring in the first place, the challanges and rewards, and the differences he encountered in different countries. His conclusion:

if you’re looking at hourly rate as a metric for outsourcing success, you’re doing it very, very wrong!

NIST’s new password rules – what you need to know

The United States National Institute for Standards and Technology has come up with new guidelines for password policies. If you are wondering which password rules to follow in your product, these are a great baseline. Note that the NIST policies contradict the FBI’s. While you are at it, consider if you actually need to store a password at all. Medium, for example, emails you a link to log in. Because of “forgot your password” functionality in most sites, access to your email is essentially equal to access to the site. Medium just made it explicit and removed the need for them to store users passwords. Those passwords are probably re-used elsewhere. If you don’t store them, you can’t loose them. Right?