Practical Unix For Ruby & Rails

SDRuby - September, 2014

Ylan Segal

2

The Unix Philosophy

Many UNIX programs do quite trivial things in isolation, but, combined with other programs, become general and useful tools.

The UNIX Programming Environment, Brian Kernighan and Rob Pike

3

Bash

You already have it on Mac and Linux.

On Windows: Use Cygwin or VM

4

Tail The Log

$ tail -f log/development.log
Started GET "/tokens/new" for 127.0.0.1 at 2014-08-27 20:03:43 -0700
Processing by TokensController#new as HTML
  User Load (0.3ms)  SELECT  "users".* FROM "users"  WHERE "users"."id" = 3  ORDER BY "users"."id" ASC LIMIT 1
  Rendered tokens/_form.html.haml (78.0ms)
  Rendered tokens/new.html.haml within layouts/application (86.9ms)
  Rendered layouts/_navigation.html.haml (0.3ms)
  Rendered layouts/_messages.html.haml (0.1ms)
Completed 200 OK in 119ms (Views: 115.8ms | ActiveRecord: 0.3ms)


Started POST "/tokens" for 127.0.0.1 at 2014-08-27 20:03:46 -0700
Processing by TokensController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"+/a7r/ubDAn1vSf5CjhrgxPzSHfemFhUxJhrapbR2BA=", "token"=>{"name"=>"ad", "secret"=>"[FILTERED]"}, "commit"=>"Create Token"}
  User Load (0.2ms)  SELECT  "users".* FROM "users"  WHERE "users"."id" = 3  ORDER BY "users"."id" ASC LIMIT 1
   (0.1ms)  begin transaction
  SQL (32.5ms)  INSERT INTO "tokens" ("created_at", "name", "secret", "updated_at", "user_id") VALUES (?, ?, ?, ?, ?)  [["created_at", "2014-08-28 03:03:46.418899"], ["name", "ad"], ["secret", "asdasdasd"], ["updated_at", "2014-08-28 03:03:46.418899"], ["user_id", 3]]
   (1.8ms)  commit transaction
Redirected to http://localhost:3000/tokens
Completed 302 Found in 41ms (ActiveRecord: 34.6ms)
...

5

Turn Down The Noise

$ tail -f log/development.log | grep '^Started'
Started GET "/" for 127.0.0.1 at 2013-09-06 13:57:46 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 13:59:02 -0700
Started DELETE "/users/sign_out" for 127.0.0.1 at 2013-09-06 14:54:53 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 14:54:53 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 14:55:04 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 14:55:08 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 14:55:40 -0700
Started GET "/users/sign_in" for 127.0.0.1 at 2013-09-06 14:55:40 -0700
Started POST "/users/sign_in" for 127.0.0.1 at 2013-09-06 14:55:53 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 14:55:53 -0700
Started GET "/users/edit" for 127.0.0.1 at 2013-09-06 15:10:13 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 15:10:28 -0700
Started GET "/users/edit" for 127.0.0.1 at 2013-09-06 15:10:56 -0700
Started GET "/users/edit" for 127.0.0.1 at 2013-09-06 15:11:08 -0700
Started GET "/users" for 127.0.0.1 at 2013-09-06 15:11:11 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 15:11:15 -0700
Started GET "/users/edit" for 127.0.0.1 at 2013-09-06 15:11:16 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 15:11:17 -0700
Started GET "/users/edit" for 127.0.0.1 at 2013-09-06 15:11:20 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 15:13:52 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 15:26:23 -0700
Started GET "/tokens/new" for 127.0.0.1 at 2013-09-06 15:26:24 -0700
Started GET "/tokens/new" for 127.0.0.1 at 2013-09-06 15:27:16 -0700
Started POST "/tokens" for 127.0.0.1 at 2013-09-06 16:05:08 -0700
Started POST "/tokens" for 127.0.0.1 at 2013-09-06 16:08:05 -0700
Started POST "/tokens" for 127.0.0.1 at 2013-09-06 16:08:17 -0700

6

How many times has each endpoint been hit?

7

Extract Information

$ grep '^Started' log/development.log
Started GET "/" for 127.0.0.1 at 2013-09-06 13:29:47 -0700
Started GET "/users/sign_in" for 127.0.0.1 at 2013-09-06 13:29:55 -0700
Started GET "/users/sign_up" for 127.0.0.1 at 2013-09-06 13:29:56 -0700
Started POST "/users" for 127.0.0.1 at 2013-09-06 13:30:21 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 13:30:21 -0700
Started GET "/users/1" for 127.0.0.1 at 2013-09-06 13:30:30 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 13:30:35 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 13:45:04 -0700
Started GET "/users/1" for 127.0.0.1 at 2013-09-06 13:45:23 -0700
Started GET "/" for 127.0.0.1 at 2013-09-06 13:45:29 -0700
Started GET "/users/1" for 127.0.0.1 at 2013-09-06 13:45:51 -0700
...

8

Extract Information

$ grep '^Started' log/development.log | grep --only-matching '".*"'
"/"
"/users/sign_in"
"/users/sign_up"
"/users"
"/"
"/users/1"
"/"
"/"
"/users/1"
"/"
"/users/1"
...

9

Extract Information

$ grep '^Started' log/development.log | grep -o '".*"' | sort
"/"
"/"
"/"
"/"
"/"
"/"
"/"
"/"
"/"
"/"
"/"
...

10

Extract Information

$  grep '^Started' log/development.log | grep -o '".*"' | sort | uniq
"/"
"//users/sign_in"
"/session/sign_out"
"/token"
"/tokens"
"/tokens/"
"/tokens/1"
"/tokens/1/edit"
"/tokens/10"
"/tokens/10/edit"
"/tokens/11"
"/tokens/11/edit"
"/tokens/14/edit"
"/tokens/2"
"/tokens/2/edit"
"/tokens/3"
"/tokens/3/edit"
"/tokens/4"
"/tokens/5"
"/tokens/5/edit"
"/tokens/6"
"/tokens/6/edit"
"/tokens/7"
"/tokens/7/edit"
"/tokens/9"
"/tokens/new"
"/users"
"/users/1"
"/users/edit"
"/users/sign_in"
"/users/sign_out"
"/users/sign_up"

11

Extract Information

$ grep '^Started' log/development.log | grep -o '".*"' \
| sed --regexp-extended 's/[0-9]+/:id/'
"/"
"/users/sign_in"
"/users/sign_up"
"/users"
"/"
"/users/:id"
"/"
"/"
"/users/:id"
"/"
"/users/:id"
...

12

Extract Information

$ grep '^Started' log/development.log | grep -o '".*"' \
| sed -r 's/[[:digit:]]+/:id/' | sort | uniq -c
 113 "/"
   1 "//users/sign_in"
   1 "/session/sign_out"
   2 "/token"
 267 "/tokens"
   1 "/tokens/"
  85 "/tokens/:id"
  57 "/tokens/:id/edit"
  30 "/tokens/new"
   6 "/users"
   3 "/users/:id"
  13 "/users/edit"
  35 "/users/sign_in"
   8 "/users/sign_out"
   3 "/users/sign_up"

13

Extract Information

$ grep '^Started' log/development.log | grep -o '".*"' \
| sed -r 's/[[:digit:]]+/:id/' | sort | uniq -c | sort --reverse
 267 "/tokens"
 113 "/"
  85 "/tokens/:id"
  57 "/tokens/:id/edit"
  35 "/users/sign_in"
  30 "/tokens/new"
  13 "/users/edit"
   8 "/users/sign_out"
   6 "/users"
   3 "/users/sign_up"
   3 "/users/:id"
   2 "/token"
   1 "/tokens/"
   1 "/session/sign_out"
   1 "//users/sign_in"

Extra Credit: Normalize routes ending in '/' and starting in '//'

14

Find Inside Files

grep has a few competitors: ack and ag

$ ag 'FactoryGirl'
spec/controllers/home_controller_spec.rb
7:    let(:user) { FactoryGirl.create(:user) }

spec/controllers/tokens_controller_spec.rb
6:  let(:user) { FactoryGirl.create(:user) }
7:  let(:token) { FactoryGirl.create(:token, :user => user) }
29:      post :create, :token => FactoryGirl.attributes_for(:token)

spec/factories/tokens.rb
1:FactoryGirl.define do

spec/factories/users.rb
3:FactoryGirl.define do

spec/models/token_spec.rb
4:  subject(:token) { FactoryGirl.build(:token) }

spec/views/tokens/index.html.haml_spec.rb
5:    token = FactoryGirl.create(:token)

15

Lets Run Some Specs

Filter specs only:

$ ag 'FactoryGirl' | grep '^spec/.*_spec.rb'
spec/controllers/home_controller_spec.rb:7:    let(:user) { FactoryGirl.create(:user) }
spec/controllers/tokens_controller_spec.rb:6:  let(:user) { FactoryGirl.create(:user) }
spec/controllers/tokens_controller_spec.rb:7:  let(:token) { FactoryGirl.create(:token, :user => user) }
spec/controllers/tokens_controller_spec.rb:29:      post :create, :token => FactoryGirl.attributes_for(:token)
spec/models/token_spec.rb:4:  subject(:token) { FactoryGirl.build(:token) }
spec/views/tokens/index.html.haml_spec.rb:5:    token = FactoryGirl.create(:token)

ag formats output differently when piped

16

Lets Run Some Specs

Trim the fat:

$ ag 'FactoryGirl' | grep -o '^spec/.*_spec.rb' | sort | uniq
spec/controllers/home_controller_spec.rb
spec/controllers/tokens_controller_spec.rb
spec/models/token_spec.rb
spec/views/tokens/index.html.haml_spec.rb

17

Lets Run Some Specs

Convert lines into arguments:

$ ag 'FactoryGirl' | grep -o '^spec/.*_spec.rb' | sort | uniq | xargs rspec
rspec spec/controllers/home_controller_spec.rb spec/controllers/tokens_controller_spec.rb spec/models/token_spec.rb spec/views/tokens/index.html.haml_spec.rb
..............

Finished in 0.41011 seconds (files took 1.53 seconds to load)
14 examples, 0 failures

xargs passes lines as arguments to any command

18

Different Flavors of RSpec

$ rspec
$ bundle exec rspec
$ bin/rspec
$ rspec --drb
$ jruby --ng -S rspec
$ zeus rspec

19

Different Flavors of RSpec

Run binstub if we have them:

$ chmod +x smart_rspec

$ cat smart_rspec
#! /bin/bash

# Looking for binstubs
if [ -f ./bin/rspec ]; then
  RSPEC="bin/rspec"
else
  RSPEC="bundle exec rspec"
fi

CMD="$RSPEC $@" # Passing all arguments on to rspec
echo $CMD
$CMD

20

Different Flavors of RSpec

$ lsof -i :3000
COMMAND   PID      USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
ruby    58963 ylansegal   10u  IPv4 0x7f4d291ee5382f61      0t0  TCP *:hbci (LISTEN)
$ echo $?
0

$ lsof -i :3001
$ echo $?
1

21

Different Flavors of RSpec

Looking for spork server:

$ cat smart_rspec
#! /bin/bash

SPORK_PORT=8989

# Looking for binstubs
if [ -f ./bin/rspec ]; then
  RSPEC="bin/rspec"
else
  RSPEC="bundle exec rspec"
fi

# Looking for spork
lsof -i :$SPORK_PORT > /dev/null
if [ $? == 0 ]; then
  RSPEC="$RSPEC --drb"
fi

CMD="$RSPEC $@" # Passing all arguments on to rspec
echo $CMD
$CMD

22

Different Flavors of RSpec

Looking for nailgun server:

$ cat smart_rspec
#! /bin/bash

NAILGUN_PORT=2113
SPORK_PORT=8989

# Looking for binstubs
# ...

# Looking for nailgun
lsof -i :$NAILGUN_PORT > /dev/null
if [ $? == 0 ]; then
  RSPEC="jruby --ng -S $RSPEC"
fi

# Looking for spork
lsof -i :$SPORK_PORT > /dev/null
if [ $? == 0 ]; then
  RSPEC="$RSPEC --drb"
fi

#...

23

Different Flavors of RSpec

$ cat smart_rspec
#...
# Looking for zeus socket
if [ -S ./.zeus.sock ]; then
  RSPEC="zeus rspec"
else
  # Looking for binstubs
  if [ -f ./bin/rspec ]; then
    RSPEC="bin/rspec"
  else
    RSPEC="bundle exec rspec"
  fi
  # Looking for nailgun
  lsof -i :$NAILGUN_PORT > /dev/null
  if [ $? == 0 ]; then
    RSPEC="jruby --ng -S $RSPEC"
  fi
  # Looking for spork
  lsof -i :$SPORK_PORT > /dev/null
  if [ $? == 0 ]; then
    RSPEC="$RSPEC --drb"
  fi
fi

24

Different Flavors of RSpec

$ smart_rspec spec/controllers/index_controller_spec.rb
jruby --ng -S bundle exec rspec --drb spec/controllers/index_controller_spec.rb
...

$ smart_rspec spec/controllers/index_controller_spec.rb
zeus rspec spec/controllers/index_controller_spec.rb
....

25

Environment Configuration

$ export RAILS_SECRET_TOKEN=b938f...
$ export DB_HOST=db.host.com
$ rails server

or

$ DB_HOST=db.host.com RAILS_SECRET_TOKEN=b938f... rails server

26

Environment Configuration

$ cat tokenator/.env
RAILS_SECRET_TOKEN=b938f...
DB_HOST=localhost

$ cat ~/.profile
...
# Overriding cd function. Looking for .env and sourcing it if found
function cd {
    builtin cd "$@"
  if [[ -f .env ]]; then
    echo "### Setting up environment variables from .env"
    cat .env | grep -v '#' | grep -v '^$' | grep -v 'PATH'| while read line; do
      echo $line
      export $line
    done
  fi
}
...

27

Per-project Environment Configuration

$ cd tokenator
### Setting up environment variables from .env
RAILS_SECRET_TOKEN=b938f...
DB_HOST=localhost

28

Conclusions

A little UNIX goes a long way
The whole is greater than the sum

29

Further Inspiration

www.destroyallsoftware.com (Gary Bernhardt)
linuxcommand.org
cli.learncodethehardway.org

30

Practical Unix For Ruby & Rails

SDRuby - September, 2014

Ylan Segal

The Unix Philosophy

Bash

Tail The Log

Turn Down The Noise

How many times has each endpoint been hit?

Extract Information

Extract Information

Extract Information

Extract Information

Extract Information

Extract Information

Extract Information

Find Inside Files

Lets Run Some Specs

Lets Run Some Specs

Lets Run Some Specs

Different Flavors of RSpec

Different Flavors of RSpec

Different Flavors of RSpec

Different Flavors of RSpec

Different Flavors of RSpec

Different Flavors of RSpec

Different Flavors of RSpec

Environment Configuration

Environment Configuration

Per-project Environment Configuration

Conclusions

Further Inspiration

Ylan Segal

@ylansegal

http://ylan.segal-family.com

ylan@segal-family.com