developer blog

Back to Originate.com

Casual APIs

This blog post is inspired by some ideas from Avdi Grimm’s excellent book Confident Ruby. Definitely read it if this topic interests you. It contains a ton more best practices.

Modern software projects typically combine different code bases like frameworks, libraries, or toolkits. The problem is that these independent code bases usually comes with their own subjective domain models, each one specialized for the task the respective library is supposed to perform. Hence, combining different code bases forces developers to deal with several overlapping domain models within their applications.

If the individual libraries are too strict and narrow-minded about the types of parameters they accept as input, combining them into a cohesive code base becomes cumbersome. The code starts to feel like a bureaucracy where despite you having the right data, it is not accepted because it is not in a particular format, and just out of principle there cannot be any exceptions. Adding the necessary explicit type conversations to method calls makes them more cryptic than necessary. Extrapolating the converting of types into explicit adapter layers adds bloat to the architecture.

Wouldn’t it be nicer if libraries were a bit more casual, flexible, and “forgiving”? That with a pragmatic “can-do” attitude they accept parameters in any usable format, and do the right thing in any situation, like a friendly and helpful human operator would do?

This blog post describes a convention to accomplish this through transparently massaging input data dynamically into the needed formats. The examples are in Ruby, but the described techniques apply to most dynamically typed environments and even some reasonably sophisticated statically typed ones.

Implicit and Explicit Conversions

Let’s say we write a text processor application. As part of that we have a method that moves the cursor on the screen. It needs the number of characters by which to move forward, and given the discrete nature of characters this should be an integer.

But we want this method to be casual. If our data happens to be another type of number, i.e. a floating-point, decimal, or even our own custom classes that represent numbers, we still want to be able to use it here. Some examples:

1
2
3
4
5
6
7
8
9
10
11
move_cursor 10                              # Integer parameter

move_cursor column_width * 1.1              # Float parameter while drawing
                                            # a table

move_cursor account.balance * column_width  # BigDecimal parameter while
                                            # drawing a chart

move_cursor configuration.border_width      # We know its the right number,
                                            # but aren't sure which exact data
                                            # type configuration values have

Ruby provides a number of conventions that allow us to do this elegantly. The first one is converter methods. Let’s use one in our method to make it accept all types of numbers and convert them to the type it needs:

1
2
3
4
def move_cursor pixels
  pixels = pixels.to_int
  # pixels is now an integer, and we can move the cursor based on it
end

By calling to_int on the argument, our method now works not only with integers, but with any number.

Notice that we use to_int here, not to_i. The difference between the two is that the former is an implicit conversion, while the latter is an explicit conversion.

Implicit converter methods convert closely related types into each other. This means types which can easily stand in for each other. If you need an integer, receiving a float is probably okay. You can (and should) expect that any Ruby code base calls implicit converter methods for you, hence their name.

Explicit converter methods convert unrelated classes, i.e. classes that shoudn’t be converted without the developer knowing about it. As an example, DateTime does not implicitly convert into an integer, because points in time are not really numbers, and shouldn’t pretend to be. It should still be possible to convert DateTime into a reasonable number if necessary, but you have to do that explicitly. Don’t expect that other code does such far-fetched things for you implicitly!

The cool thing about these converter methods is that they allow for architectures that are open for extension. This means you can make your own objects behave like numbers everywhere, even in other libraries, simply by adding a to_int method to them. And bam! you can use instances of your classes anywhere numbers are required! No more manual converting!

Below is a list of Ruby’s built-in converter methods. Use them as much as you can, and don’t reinvent them!

target class implicit method explicit method
Array to_ary to_a
Complex to_c
Enumerator to_enum
Hash to_hash to_h
Integer to_int to_i
IO to_io, to_open
String to_str when needing strings
to_path when needing filenames
to_s
Proc to_proc
Rational to_r
Symbol to_sym

Dynamic Type Casting

One side effect of these converter methods is that your classes have to be aware that they can be used as a number, and have to consciously implement the converter methods. That’s usually a good thing. But what if that’s not possible, like when you don’t have control over who calls your method? Or you use a library that doesn’t implement this protocol, and you don’t want to monkey-patch all of its internal classes?

Ruby provides another set of powerful converter methods for this case. These methods have the same name as the class they convert into. They can be seen as the bigger brothers of type-casting in statically typed systems. Bigger because these methods not only cast types, but also convert their parameter into the target class.

They are also siblings of constructor methods (funny how everything seems related here), with the difference that constructors always create a new instance based on the given value, while converters don’t create a new instance if that’s not necessary, i.e. when given a usable instance already. This makes them more efficient. Let’s play with Ruby’s built-in Integer method a bit:

1
2
3
4
5
6
Integer 100         # = 100  (ints get passed through)
Integer 100.1       # = 100  (floats are converted to int)
Integer "100"       # = 100  (parses strings)
Integer "0b100"     # = 4    (parses binary-encoded strings)
Integer Time.now    # = 133422 (calls to_i and to_int on the object)
Integer({})         # TypeError: can't convert Hash into Integer

As you can see, this method uses every trick in the book to convert its argument into an integer, and if that doesn’t work, it detonates an exception. Very convenient! Ruby provides such converter methods for many of the built-in classes: Array, Float, String, Integer, URI, or Pathname.

So if we have a print method that needs to convert its arguments into a string no matter what is given to it (everything that is given to it should be printed), we would best write it like this:

1
2
3
4
def print value
  value = String value   # Convert the parameter to a String, or fail here
  # print the string value now
end

Custom Converters

Let’s build such a converter method ourselves! Let’s say our text processor application should be modern and support colors. So we create our own Color class. Anytime a color is needed, developers should be able to provide one comfortably with whatever data about colors they have at that time. Lets take the method to set the current color as an example. We should be able to call it with parameters like these:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# HTML color strings
set_color '#102030'

# integer RGB values
set_color 10, 20, 30

# floating point RGB values
set_color 10.1, 20.1, 30.1

# existing Color instances
set_color red

# an array of RGB values
set_color [10, 20, 30]

# any object that has a 'to_color' method
set_color parent_container

That’s what I call a nice to use method that works with me! Here is how easily the method performs these conversions:

1
2
3
4
def set_color *args
  color = Color *args
  # do something with the 'color' variable here
end

As is visible, the method first massages its parameter(s) into the format it needs, in this case a Color instance, and then does its job with that value. Here is its implementation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
# Converts the argument(s) into a Color instance
def Color *args
  case args.first

    when Color
      args.first

    when Integer
      Color.new(*args)

    when Float
      Color.new(*args.map(&:to_i))

    # Parse HTML color code string of the form '#112233'
    when String
      Color.new(*args.first.scan(/^#(\w{2})(\w{2})(\w{2})$/)[0].map{|x| x.to_i 16 })

    # Call to_color method if the parameter has one
    when ->(arg) { arg.respond_to? :to_color }
      args.first.to_color

    # Convert other color classes to our class by using their RGB values
    when ->(arg) { arg.respond_to? :to_rgb }
      Color.new args.first.to_rgb

    # Fail if there is no known type match
    else
      raise TypeError, "Cannot convert #{args.inspect} into a Color"
  end
end

Simple, straight, easy to understand Ruby, with clear expression of intent.

Best Practices

  • Make your APIs casual, and thereby easy and intuitive to use. If there are different legitimate types of parameters, your API should do the work to support them, not your users. This makes your API robust, meaning your users can throw all sorts of different input at them, and it does the right thing.

  • When unsure about the parameters given to your methods, for example when they are called by other people’s code, convert the parameters into the expected type or fail.

  • Don’t give up too early. Try all suitable type conversions before you reject a parameter.

  • You can take it easy on data types for parameters, but you should fail reliably for bad arguments. For example, if you allow strings to be provided for colors, and the string does not contain a color code, your code should fail, because that’s most likely a bug. If you get an id for a product, and the referenced product does not exist, there should be an immediate failure as well.

  • Use integration tests (aka feature specs) to make sure everything is plumbed together correctly, and the application functions as it should.

  • To keep things efficient, only convert types at the public boundaries of your code, not in each internal method. Similar to how the visa of tourists is only checked when they enter a foreign country at its border, and not on each bus ride inside the city.

  • Document non-implicit conversions so that other developers know about them.

Conclusion

Done correctly, casual APIs are both robust and intuitive to use. Like a helpful human operator, they do the right thing in all situations no matter what input you throw at them. This allows developers to focus on calling libraries with the right data instead of the right data format.

There are several ways to implement this pattern, depending on how strict you want to be about your input. Done consistently, casual APIs allow for casual collaboration of independent libraries in modern, open-source based code bases.

Happy dynamic type massaging! :)

Idiomatic Scala: Your Options Do Not Match

Idiomatic Scala: Your Options Do Not Match

Imagine you are reviewing a teammate’s work when you are confronted with the following piece of code. What would you think?

1
2
3
4
5
if (expr == true) {
  return true
} else {
  return false
}

Cast the first semicolon who never saw (or maybe even wrote) similar code. While it works and is strictly speaking correct, the code above disappoints for expressing what it does a little too literally. It is unsophisticated and ingenuous, a neophyte code smell almost. Experienced developers probably prefer the more compact idiom:

1
return expr

Yet, it is very common to see even seasoned Scala developers decomposing Option via pattern matching:

1
2
3
4
def f[A](opt: Option[A]) = opt match {
  case Some(a) => ???
  case None => ???
}

Pattern matching on Option is almost never necessary as usually there are cleaner and more expressive ways to attain the same results with monadic functions. This is what the Option Scaladoc has to say:

The most idiomatic way to use an scala.Option instance is to treat it as a collection or monad and use map, flatMap, filter, or foreach […] A less-idiomatic way to use scala.Option values is via pattern matching

Any expression of the form

1
2
3
4
opt match {
  case Some(a) => foo
  case None => bar
}

can be written more concisely and precisely equivalent as

1
opt map foo getOrElse bar

Furthermore, since Scala 2.10 an even more compact alternative is available, fold:

1
opt.fold(bar)(foo)

fold even gives us the additional benefit of being more type-strict than either of the previous alternatives. For instance, while a and b in the example below have a type of Any (with value 2), c fails to compile:

1
2
3
4
5
6
7
8
9
10
val opt = Option(1)

val a = opt match {
  case Some(x) => x + 1
  case None => "a"
}

val b = opt map (_ + 1) getOrElse "a"

val c = opt.fold("a")(_ + 1)

On the not-so-bright side, fold may not be as readable as the alternatives and a little error-prone, too, since it is not clear what comes first, None or Some?

Readable or not, we still can do a lot better than fold: Option offers a few dozen methods, many of which can be used to better express more specific transformations. For example, opt.fold(true)(_ => false) is equivalent to opt.isEmpty. The converse is either opt.isDefined or its alias opt.nonEmpty.

The Ultimate Scala Option Cheat Sheet

We reproduce below a table highlighting the most relevant Option methods. Whenever you feel like writing case Some(a) => foo; case None => bar consult this table for the equivalent foo and bar in the Some(a) and None columns to find its corresponding method. It is also recommended that you refer to the Option Scaladoc for detailed descriptions and additional methods.

First, a few definitions. For arbitrary types A and B, let:

1
2
3
4
val a: A
def p(a: A): Boolean // a predicate
def f[B](a: A): B // a mapping function
def g[B](a: A): Option[B] // an optional mapping function
Method Meaning Some(a) None
map(f) Applies f to its content Some(f(a)) None
getOrElse(default) Retrieves its content or a default value a default
fold(default)(f) (2.10) Same as map f getOrElse default f(a) default
isEmpty Is it a None? false true
isDefined Is it a Some? true false
nonEmpty (2.10) Same as isDefined true false
size Same as isDefined, but returning 0 or 1 1 0
flatMap(g) Same as map, except g returns Option g(a) None
orElse(Option(default)) Same as getOrElse, but inside an Option Some(a) Option(default)
orNull For legacy code that expects null a null
filter(p) Turns a Some into None if its content does not conform to p Some(a) if p(a), otherwise None None
find(p) Same as filter Some(a) if p(a), otherwise None None
filterNot(p) (2.9) Same as filter(!p) Some(a) if !p(a), otherwise None None
contains(b) (2.11) Whether its content is equal to b a == b false
exists(p) Does its content conform to p? p(a) false
forall(p) (2.10) Same as exists, except None returns true (see note below) p(a) true
count(p) Same as exists, but returning 0 or 1 1 if p(a), otherwise 0 0
foreach(f) Applies side-effectful f to its content Unit (calls f(a)) Unit
flatten (2.10) “Unnest” a nested Option. Compile error if a itself is not an Option a None
toRight(default) Converts to Either Right(a) Left(default)
toLeft(default) Converts to Either Left(a) Right(default)
toList Converts to List List(a) Nil
toSeq Same as toList List(a) Nil
toSet Converts to Set Set(a) Set()
get Do not bother about what this is supposed to do, never use it :–) a NoSuchElementException

.exists and .forall

The most attentive readers may be baffled by why None.forall returns true.

If you have your favorite predicate logic introductory book at hand, you may want to read the sections on “universal quantification” and “vacuous truth”. If not, trust us, the math checks out :–)

Here is a colloquial explanation. Basically, it boils down to forall being the reciprocal negative (this may not be the correct mathematical term) of exists. This is easier to understand with an example:

1
2
3
4
5
def isPrime(n: Int): Boolean = ???

val a = Seq(2, 3, 5, 7)

val b = Seq(1, 2, 4, 8)

We can clearly see that a.forall(isPrime) is true, while a.forall(!isPrime) is false. However, for list b, both b.forall(isPrime) and b.forall(!isPrime) are false. Negating the predicate does not necessarily negates the result. The negation of forall(isPrime) is not forall(!isPrime), but exists(!isPrime). In other words:

1
forall(isPrime) == !exists(!isPrime)

It makes perfect sense in plain English: to say “all numbers are prime” is equivalent to saying “there does not exist a number that is not prime”.

Back to Option, None.exists(p) is always false, as is None.exists(!p). Therefore, by the identity derived above:

1
2
3
None.forall(p) == !None.exists(!p)
               == !false
               == true

As unintuitive and surprising as this result may be, it is true and correct, proven by much more rigorous mathematical derivation. As a matter of fact, this apparent oddity can be put to some pretty good use, as we will see next.

A more elaborate example

Let us start with a simple method: tracking events on a webpage. The only distinctive characteristic here is that whether we have a user logged in to the site, we record their ID.

1
2
3
4
def track(user: Option[User]) = user match {
  case Some(u) => Tracker.track(u.id)
  case None => Tracker.track(GUEST)
}

Now, due to privacy concerns, we have to allow users to opt out of tracking. Easy-peasy:

1
2
3
4
def track(user: Option[User]) = user match {
  case Some(u) if u.canTrack => Tracker.track(u.id)
  case _ => Tracker.track(GUEST)
}

All was going well, until one day we got a call from our lawyers: if a user opts out, they should never be tracked at all, not even anonymously. Hence, our final version:

1
2
3
4
5
def track(user: Option[User]) = user match {
  case Some(u) if u.canTrack => Tracker.track(u.id)
  case None => Tracker.track(GUEST)
  case _ => // do not track
}

Now let us convert everything to not use pattern matching. The first one is trivial:

1
Tracker.track(user map (_.id) getOrElse GUEST)

Not only did we replace the match with a more compact single-line expression, we do not have to write Tracker.track twice anymore. It may seem inconsequential, but over time such small things tend to grow in complexity, making maintenance harder.

The second version is more interesting. Whenever we see pattern guards, filter may be required. Overall, it does not change much from above:

1
Tracker.track(user filter (_.canTrack) map (_.id) getOrElse GUEST)

Finally, something more interesting. Now we have not two, but three case clauses. Clearly, there is no single method in Option that handles three possible outcomes. The answer, however, is surprisingly simple once you know it:

1
2
if (user forall (_.canTrack))
  Tracker.track(user map (_.id) getOrElse GUEST)

Knowing that None.forall is always true was crucial to allow us to come up with a simple expression that handles all cases.

Tip: Prefer Option(a) over Some(a)

You may have noticed in some examples that we wrote val a = Option(1) instead of val b = Some(1). Usually, one should prefer Option(x) over Some(x). First, a is inferred as type Option[Int], while b is Some[Int]. Secondly, and more importantly, is that Option(x) is more type-safe than Some(x). Again, let us see why with an example:

1
2
3
4
5
6
// We are enthusiastic about newcomers
def hello(name: Option[String]) = s"Hello, ${name getOrElse "stranger" toUpperCase}!"

def a(name: String) = hello(Option(name))

def b(name: String) = hello(Some(name))

Which is better? Method a or method b? Call a(null) and b(null) and see the answer for yourself.

Trivia

All Option methods in the Scala standard library are implemented as single line methods that are slight variations of the form:

1
if (isEmpty) bar else foo

Most methods are also marked as @inline final, which means that using Option higher-order functions should be as efficient as safe-guarding for null in Java, especially after HotSpot does its magic. How cool is that?

Addendum: Do not abuse pattern matching

Pattern matching is a very expressive and powerful tool. However, as frequently is the case with the finest hammers, we must watch out for the proverbial nail.

Pattern match all the things

Pattern matching “all the things” must be considered a Scala code smell. Especially for some common cases, pattern matching is not always as readable or concise as other constructs, and often the bytecode generated by the Scala compiler is less efficient, too.

In particular, do not pattern match on Booleans; use good old, true and tested if else:

1
2
3
4
5
6
cond match {
  case true => Ok
  case false => Error
}

if (cond) Ok else Error

Avoid pattern matching on a single value followed by a defaut case:

1
2
3
4
5
6
7
8
n match {
  case 0 => "Cart is empty"
  case _ => "View cart"
}

if (n > 0) "View cart" else "Cart is empty"

s"Your cart has ${if (n > 0) n else "no"} item(s)."

Finally, for most data structures like Seq and String, consider methods like isEmpty, headOption, tail, etc. And for enumerations, use withName.

Enjoy your Options!

Deep Linking in iOS

Deep linking consists of using a hyperlink that links to a specific piece of content within an app. The specific content could be a specific view, a particular section of a page, or a certain tab.

To see an example, download the Twitter app. Log into it and close it. Now, try opening the following url in Safari Mobile on your device: twitter://timeline. iOS will switch to the Twitter app and go directly to your timeline.

Now try the following url in Safari Mobile: twitter://post?message=using%20deeplinking. This time Twitter will open up with the message “using deep linking” already composed and ready to be tweeted.

You can use deep linking for:

  • Displaying a different landing page after launch.
  • Inter-app communication by launching an app from another app and passing information.
  • Creating a web-like URI based navigation scheme throughout your app.
  • Enabling integration with other apps by letting them launch your app directly.
  • Recording and analyzing user behavior to determine where your users launch your app from.

Given the above uses, let us look at a basic implementation of deep linking. In this tutorial we are going to:

  1. Enable deep linking in an app.
  2. Handle launch URLs to show a different landing page based on the url.

Note: This tutorial assumes a basic knowledge of creating simple apps in iOS

In this post we’ll be referring to the sample app, which is located here: https://github.com/vipulvpatil/deeplinking-in-ios.

Create an app and enable deep linking

Create a basic app in XCode consisting of a main ViewController pushed on a UINavigation ViewController. Also create some additional ViewControllers to be used later.

To enable deep linking, go to the Info tab in the Xcode project. In the URL Types section, click on the + button, and then add an identifier and a URL scheme. Ensure that the identifier and URL scheme you select are unique. Take note of the URL scheme you enter, as this is how iOS knows to open a link in your app. The sample app registers the following url scheme:

dlapp

To confirm that your URL scheme has been registered, check Info.plist for an entry named ‘URL Types’. Expanding it will show you the new URL scheme you just registered. You can check that this is working by typing the following url into Safari Mobile: your-url-scheme:// ( For the sample app, it would be: dlapp://). This should open up your app. If not, please go through this section again before moving on.

Handling the opening of registered urls within your app

Now that you have ensured that deep linking is working, we need to handle the url used to launch the app. In it’s current state, your app can be launched using a simple url, but it can’t do much beyond that. To do more, we need to override the following function in AppDelegate:

1
2
3
4
-(BOOL)application:(UIApplication *)application
          openURL:(NSURL *)url
          sourceApplication:(NSString *)sourceApplication
          annotation:(id)annotation

Note that this is not present by default, and needs to be added. This function gets called every time your app is launched using the registered url-scheme. The passed in arguments are:

  • url: The complete url used to launch the app.
  • sourceApplication: The bundle ID for the application from which the url was called.
  • annotation: A property list object that can be used to pass additional info along with the url.

The format of the url is as follows: [scheme]://[host]/[path]

Breaking this down, we get:

  • scheme: The url scheme tells iOS what app to launch. Note that the url scheme should be registered with the device for it to be able to handle this (this is what we did in the previous section).
  • host: The host is analogous to a website/server name on the web. You can handle multiple hosts within your app.
  • path: The path enables you to provide additional information regarding the location within your app.

In general, you would use the host and path parameters to determine what the user intends to do.

The exact contents of this method largely depend on your needs, but for the purpose of this post, we will check the host and then based on the path, load a particular ViewController.

1
2
3
4
5
6
7
-(BOOL)application:(UIApplication *)application openURL:(NSURL *)url sourceApplication:(NSString *)sourceApplication annotation:(id)annotation{
  if([[url host] isEqualToString:@"page"]){
    if([[url path] isEqualToString:@"/page1"]){
      [self.mainController pushViewController:[[Page1ViewController alloc] init] animated:YES];
    }
 return YES;
}

Line 2 checks to see if the host within the url is what we are expecting, i.e. ‘page’ in this case. After that, it matches the url path and loads the view controller for that page. In this way, you can handle, every expected url within the function and show a different screen for each. One thing to note is that the app can be launched with a url regardless of whether this function can handle it or not. In such cases returning NO from the method tells iOS that the url was not handled by the app. The result in general, is that the app just opens in the last state it was in.

In the sample app, we handle 4 different urls:

  • dlapp://page/main
  • dlapp://page/page1
  • dlapp://page/page2
  • dlapp://page/page3

Here is the complete code to handle the above urls:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
if([[url host] isEqualToString:@"page"]){
    if([[url path] isEqualToString:@"/main"]){
      [self.mainController setViewControllers:@[[[DLViewController alloc] init]] animated:YES];
    }
    else if([[url path] isEqualToString:@"/page1"]){
      [self.mainController pushViewController:[[Page1ViewController alloc] init] animated:YES];
    }
    else if([[url path] isEqualToString:@"/page2"]){
      [self.mainController pushViewController:[[Page2ViewController alloc] init] animated:YES];
    }
    else if([[url path] isEqualToString:@"/page3"]){
      [self.mainController pushViewController:[[Page3ViewController alloc] init] animated:YES];
    }
    else{
      return NO;
    }
    return YES;
  }
  else{
    return NO;
  }

Conclusion

This post shows you how to respond to urls passed into your app, but you can do a lot more. By taking apart the url in the callback, you can do quite a lot. For example, you could:

  • customize the look and feel of the page opened
  • pre-populate certain elements on the page (the way the Twitter app did)
  • track how users are getting to your app

Happy Coding!

Refactoring Git Branches

TL/DR

I describe a technique that allows one to extract commits from a larger Git branch into separate branches. This “git branch refactoring” provides numerous benefits:

  • A fast-track for integrating urgent changes (like refactorings or bug fixes) that were created as part of feature development into the main Git branch, before the feature is merged.

  • Improvements to the quality, efficiency, and fun factor of code reviews, by allowing your team to review and merge individual changes within larger feature branches individually, by different reviewers, at different times.

Feature branches contain many different changes

When code classes or methods do too many things, we refactor them by extracting individual pieces of functionality into separate classes or methods. The same applies to feature branches in Git. Ideally a Git feature branch should also perform just one particular change, be it a bug fix, a refactoring, or adding a new feature.

We all know, however, that while adding the next cool feature to a code base, we come across existing code that needs to be improved along the way. So, in the spirit of Continuous Improvement and Kaizen, or simply because we depend on improvements to happen before we can continue developing our feature, we

  • fix bugs in existing code that we depend on,
  • add missing functionality to existing code that our feature requires,
  • refactor existing code, for example by extracting pieces of it to make them available separately,
  • do good in numerous other ways, for example by cleaning up messes, reducing smells, paying down technical debt, or improving code quality and spec coverage.

As a result of this process, many feature branches end up looking similar to the one depicted here (time flows from the bottom up here, i.e. new commits are added at the top):

bloated feature branch

Our feature branch called kg_feature_1 is cut from the development branch (we follow the terminology of the NVIE model here), which is our main branch that is shared by all developers here. This development branch only contains the framework so far. Our feature branch contains a number of commits for the feature itself (feature 1feature 4), a bug fix for an external code module (“bug fix 1”), and a more generic refactoring that was done in two separate steps (“refactoring 1a” and “refactoring 1b”).

Changes should be reviewed individually

We don’t want to just send this many-headed monster of a feature branch off as a single code review. It contains too many different things! This makes it hard for the reviewer to get an overview of what we are doing here. Its also pretty hard to have meaningful conversations about several different things at the same time. This is just a recipe for confusion, cognitive overload, and missing important details. It also reduces the fun and the power of code reviews.

We also want to review and merge the bug fix and the refactorings into the development branch right away, so that other developers can incorporate them into their work before their work deviates too much from our changes. If we waited with that until the whole feature is done, it will be too late, and we will have to deal with a lot more merge conflicts than necessary.

These different types of changes in different parts of the code base should probably also be reviewed by different people. The bug fix gets reviewed by the author/maintainer of the affected module, the refactoring by the architect or tech lead, and the feature by another team member.

Let’s refactor our Git branch!

Extracting commits into dedicated branches

The change that most urgently needs to get merged into development here is the refactoring, because it might touch code that other people are currently working on. Lets extract it into a separate branch.

1
2
3
4
5
6
# Cut a new branch for the refactoring off the development branch.
$ git checkout -b kg_refactoring development

# Move the refactoring commits into the "kg_refactoring" branch.
$ git cherry-pick [SHA1 of the "refactoring 1a" commit]
$ git cherry-pick [SHA1 of the "refactoring 1b" commit]

Our kg_refactoring branch now looks like this.

refactoring branch

It only contains both refactoring commits. Perfect! Now we can push this branch to the Git repo, get it reviewed by the architect or tech lead, and merge it into the development branch.

Once this is done, we rebase our feature branch against the development branch to pick up all the changes that happened in it recently, like our extracted refactoring. We do this regularly anyways, since it is a good practice to keep our feature branches synchronized with ongoing development, and solve merge conflict as soon as they happen, rather than all at once when the branch is done.

1
2
$ git checkout kg_feature_1
$ git rebase development

If you know Git, you know that our branch situation looks like this now.

after refactoring merge

This looks much better already. The refactorings are now part of the development branch, and our feature branch still has access to them, and uses them!

The only foreign body left in the feature branch is the bug fix. Once we have extracted it using the exact same technique, our branch situation looks like this.

refactoring branch

Both bug fix and refactorings have been individually reviewed and merged into the development branch, separately from the ongoing feature development. The refactored feature branch contains only feature-related commits, and can be reviewed now, or developed further. If we do more bug fixes or refactorings, we can repeat this procedure as needed.

When to do this

Extracting autonomous changes into their own Git feature branches can significantly improve the structure of your branches, the performance of your code reviews, and the efficiency of your workflows. In order to work well, however, it is dependent on a number of things:

  • Each commit is well described and addresses a single change (feature, bug fix, refactoring, etc).

  • You can rebase your feature branches. Only do that with branches that only you use! Read Linus Torvalds’ explanation if you are unsure what this means. In our case we marked our branches as private by prefixing their names with our initials, and according to our team policy this makes them private.

  • You have enough specs, especially feature specs (aka integration tests) to verify that everything still works after your branch refactorings.

All these are good engineering practices to live by anyways, so give this a shot, and let me know what you think in the comments!

Lightning Fast Prototyping With Harp

What is Harp?

Harp is a static web server built on Node. It serves Jade, Markdown, EJS, CoffeeScript, Sass, LESS and Stylus as HTML, CSS & JavaScript with no additional configuration or plugins. You get a powerful asset pipeline with a few extra perks. This makes it great for prototyping and as a frontend playground.

Why use Harp?

It’s blazing fast, easy to get setup, and dead simple. It’s also more powerful than things like CodePen or jsFiddle (more than just a “widget builder”). If you end up liking what you’ve built, you can then compile the static assets or actually deploy it to Harp’s own dropbox based hosting platform. Because of it’s speed and versatility, Harp is an efficient and effective tool for prototyping UI concepts and workflows.

Getting Started

  • Install Node or brew install node
  • Install Harp npm install -g harp
  • Create a new Harp project harp init project-name (will create a folder with your project name in the current directory)
  • Start Harp server harp server project-name
  • Access your site at localhost:9000

Now you can write using your favorite frontend technologies! (This article will be using Jade/Sylus for all code examples.)

Note: Any time you would like to generate the static files for your project, you can run harp compile and your site will be generated in the /www folder in the root of your project.

Public vs. Private

In Harp, anything prefixed with an underscore will not be publicly available via the server or compilation. This is useful for partials and/or private layout files. Anything without an underscore will be compiled and publicly available. Making things private when possible will also improve compile time (not that you need it). Folders that are prefixed with an underscore will be private as well as anything that is contained within.

Layouts and Partials

You’ll have layout files which have the underscore prefix. The _layout file will be applied to anything on that folder level. eg: /_layout.jade would apply to anything on the root level. /posts/_layout.jade would be applied to any page view within the /posts directory.

Partials are just as easy. If you have a partial.jade file in the _partials/ folder you can simply:

1
!= partial("_partials/partial")

Static Data Structures

With Harp you also get flat JSON data structures to play with. You can create a _data.json file in any subdirectory and then have access to that data in your templates. Eg: if you have /posts/_data.json you can then iterate over that json collection.

with /posts/_data.json:

1
2
3
4
5
6
7
8
9
10
11
12
{
  "post1-slug" : {
    "name":      "Post 1",
    "author":    "John Doe",
    "content":   "Lorem ipsum..."
  },
  "post2-slug" : {
    "name":      "Post 2",
    "author":    "Kevin Smith",
    "content":   "Some Post Content"
  }
}

Then in your template:

1
2
3
4
each post, slug in public.posts._data
  .post
    h3
      a.link(href="/posts/#{ slug }")= post.name

This will enumerate over the top-level keys using the key name as the slug.

Output:

1
2
3
4
5
6
7
<div class="post">
  <h3><a class="link" href="/posts/post1-slug">Post 1</a></h3>
</div>

<div class="post">
  <h3><a class="link" href="/posts/post2-slug">Post 2</a></h3>
</div>

It should be noted here that the _data.json file is intended only for meta data. If you were blogging, like in this example, you would also need to include a view file in the posts directory (with a file name that’s the same as the slug) that contains the actual post itself. The _data.json would include things like the slug, title, tags, etc. It’s easy to see that you can use _data.json files to mimic database results for prototyping very easily.

You also get a global variables JSON file located at /_harp.json which is useful for layout variables (eg: site title) and other settings that you’ll need available within scope.

The Current Object

On any template or layout you will have access to current which will give you brief information about the current location of the app. The current object has a path and source property. Eg: if you were accessing /posts/post-name the current object may look like:

1
2
3
4
{
  "path":   ["posts", "post-name"],
  "source": "post-name"
}

This is very useful if you would like to have an active-state in your site’s navigation:

1
2
3
4
5
ul
  li(class="#{ current.source == 'index' ? 'active' : '' }")
    a(href="/") Home
  li(class="#{ current.source == 'posts' ? 'active' : '' }")
    a(href="/posts") Posts

A Boilerplate Project

If you prefer to learn by example, you can clone my harp boilerplate. It will set you up with a layout, a partial, some useful stylus mixins, icon fonts, and other helpful examples.

For more information you should check out the harp documentation

Secure Your Software for Battle

A World Wide War is being fought across the software world. The warring factions are engaging in battles in IT centers, corporate offices and your homes; between security experts on one side and cyber-criminals on the other.

This world we live in is getting increasingly connected and computerized. Consumers, organizations and governments alike are conducting business electronically at an ever increasing pace. This requires organizations to store information about their consumers. This includes financial, personal data as well as data about their online and spending habits.

Corporations need to secure this data because

  • privacy laws demand it
  • consumers demand it
  • it gives them a competitive advantage

Naturally, this data is prized by hackers, criminals, competitive entities and unfriendly state agents. This battle of wits between the security experts and the cyber-criminals is an ongoing one and is only going to get bigger.

The systems that process and store data also become targets — either to disable them, or to access, corrupt or destroy data.

Securing the data and the systems that manage it is now a high priority and high profile task. Or, it should be if they mean business. It is no longer a nice-to-have or even a should-have. It is now a must-have.

Companies that don’t have their act together … are in for a crude awakening in the internet age, to significant damage to their business and reputation (http://www.businessweek.com/articles/2014-03-13/target-missed-alarms-in-epic-hack-of-credit-card-data), and in some cases to an end of their very existence (http://www.forbes.com/sites/cameronkeng/2014/02/25/bitcoins-mt-gox-shuts-down-loses-409200000-dollars-recovery-steps-and-taking-your-tax-losses).

The War Zone

The increasing mobility of data in corporations is one of the biggest challenges faced in the last couple of years. Users are feeling empowered to access data from anywhere. The BYOD (Bring Your Own Device) trend is bringing in new attack vectors. The data access end-point is also going through a transformation. No longer are these Windows-based, but now a diverse mix of different systems. Some of them are still in their infancy compared to long established platforms like Windows/Unix. These platforms are new targets, and malware for mobile devices is now a rapidly growing threat.

Employees are bringing in their smartphones, tablets and next-gen ultrabooks in to the corporate networks. IT professionals are being required to secure sensitive data on devices the have little control over.

The growing popularity of the “Internet of things” (smartphones, tablets, wearable gadgets, and interconnected devices) makes a fluid situation even more dynamic for security experts. They present ripe targets for cyber-criminals looking to either find access to your data or cause a large-scale disruption.

Cyber-criminals are getting sophisticated themselves. They are using underground online marketplaces to sell their wares and services. The Blackhole exploit kit accounted for almost a third of all malware infestations in 2012. And, it spawned a new generation of botnet exploits that drew on its source code for inspiration.

The Attacks

As the arms race unfolds between the cyber-criminals and security experts, here are some security threats that are at the top the list.

Cloud

Cloud computing continues to grow in popularity — and so too will the number of security threats targeting the cloud. It’s important that businesses realize that whilst they may outsource the handling and storage of their data, they can’t outsource the responsibility for the data itself. If their provider’s systems are breached, and data is exposed, the businesses are responsible.

Therefore, any risks need to be assessed in the same way as they would if they were holding the data internally. Other issues organizations should consider include: where will the data be stored, what happens to any data if organizations switch providers and what steps are being taken to secure the data on their provider’s systems, including how they prevent other customers from accessing it.

The bad guys will look to targeting the cloud in 2014, with cloud service provider employees the main focus. Successful phishing attacks on staff, especially if the password re-use jackpot is hit, have been the door to entry of many online member databases during 2013.

APT

APT or Advanced Persistent Threat is a set of well-camouflaged, focused, continuous hacking activities against a single target. APT activities require a high-level of stealth sustained over a period of time. These employ advanced malware to exploit vulnerabilities in the targets. And these are usually orchestrated for business or political motives. Flame/Gauss and Stuxnet are examples of recent APT attacks used to attack financial, government and military institutions.

Social Engineering

It’s a tried-and-true tactic in both the physical and digital worlds – social engineering. Before the digital age, this meant sneaking one’s way past the security desk with glib talk vs. a cleverly-worded email. Now social engineering has moved onto social networks, including Facebook and LinkedIn.

Attackers are increasing their use of social engineering, which goes beyond calling targeted employees and trying to trick them into giving up information. Cyber-criminals will try to hide malware using deceitful tactics to trick you into installing it.

The task of cybercriminal is a lot simpler — check if details are already posted on social networks. Social networks are about networking people; a convincing-looking profile of a company or person followed by a friend/connection request can be enough to get a social engineering attack started. Sarah Palin’s Yahoo email account was compromised using information that was publicly available on her social media profiles (birth name, date of birth etc).

Password Management

We live in a world where people are increasingly connected through social networking, cloud services & smart devices. This presents a password management headache for users and, by extension, the services they use. People tend to reuse passwords. More than half users reuse passwords for their online accounts. A data breach in one vendor/service provider can potentially put their accounts at other services at risk. If that weren’t scary enough, over a quarter tend to use easy-to-remember passwords such as birthdays or people’s names, opening the door for their online accounts to be hacked into by criminals. It’s worrying aspect for corporations that so many people are making life so easy for cyber-criminals hackers. Especially because simple to make strong password security a part of one’s everyday life.

BYOD (Bring Your Own Device )

This is one threat vector that will give security professionals many sleepless nights. BYOD (Bring Your Own Device) increases data leakage potential especially from devices unprotected by device specific counter-measures such as passcode/passphrase protection. etc. Most organizations going down the BYOD path aren’t implementing appropriate training for employees.

Attackers will try and circumvent app review and detection mechanisms on these devices. If possible their apps will mimic the UI of the native settings page and trick the user in to granting them admin privileges on the device. The device sensors such as GPS, microphone and cameras coupled with the ability to network over WIFI and mobile networks, could become tools for engineering further attacks.

The main areas of risk include: * Data loss — lost or stolen devices * ID theft — thieves logging in to your online accounts using saved credentials on your stolen device * Data leakage — via bogus WIFI access points and malware

Botnets

Essentially, a botnet is a collection of networked computers running a program to perform a task. Initially, most botnets were employed to perform legitimate tasks such controlling IRC channels. Later they were deployed to distribute spam, or perform DDos attacks.

Botnet operators are beginning to design systems that are more adaptive and redundant than many corporate and government networks. Botnets such as Gameover have replaced the traditional command and control link with a peer-to-peer network of infected systems. Controlling this agile attack vector before it can be used as an advanced persistent threat (APT) and migrates into smart mobile devices is crucial.

We’re also likely to see more mobile botnets, of the sort created using the RootSmart backdoor in Q1 2012. In order to prevent falling victim to mobile malware, businesses should install anti-malware protection on their Android devices, secure the data held on them and make sure that this can be wiped remotely if the device is lost or stolen. Businesses should also develop a policy for staff on how to reduce the risks from mobile devices. This should include not rooting the device, avoiding public Wi-Fi networks for confidential transactions, not relying solely on a simple PIN and only installing apps from trusted sources.

Spam

People still send emails. So do bad guys, and they will keep doing so long as people keep using email. You see spam that link to financial scams that are now mostly ignored, spam that link to malware designed to install botnet agents, spam that attempt to seem legit by linking to current events. Phishing via deceptive links in spam is a very common attack. Spammers are also wising up to having their bots and servers taken down. Snowshoe spam is their innovation against this countermeasure. This involves distributing their spamming across multiple ip addresses, spreading the load. Hence, the metaphor.

The Defense

So what are organizations to do to deflect security attacks? It used to be about securing the perimeter in the past. Now the perimeter is blurred due to BYOD and cloud services trends.

There is no one good answer. But most of them would include adopting treating security as a feature, good engineering practices, implementing systems, good employer training, a good plan to manage responses to attacks and finally transparency.

At Originate, when we work with partners, we emphasize security up front as a key element of the systems we build. In most cases, we have been able to educate and received buy-in from our partners. In one case however, we received push back on implementing a policy to enforce strong passwords — the partners feared having a strong password policy would impact user registration. Guess what, on the first day of launch, the first account to be compromised was the admin account. Fortunately, we were monitoring all activity and were able to retake control of the account and secure it. Needless to say, we got the go-ahead to implement the strong-password policy.

Make security a feature ( and not a Non-functional requirement )

When writing applications, security aspects should be first-class requirements. It cannot be a bolt-on later. Build the app with the best security practices from the start. According to security firms FireEye and Secunia, the real security problems in this decade are not in our operating systems but in the applications we run on them.

Design Software with Secure Features

Security issues in design and semantic flaws (ones that are not syntactic or code related), such as business logic flaws, cannot be detected in code and need to be inspected by performing threat models and abuse cases modeling during the design stage.

Threat modeling is an iterative technique used to identify the threats to the software being built. It starts by identifying the security objectives of the software and profiles it. It breaks the software into physical and logical constructs generating the software context that includes data flow diagrams, and end-to-end deployment scenarios, identifying entry and exit points, protocols, components, identities, and services.

Threat Modeling is performed during the design stage so that necessary security controls (safeguards) can be developed during the development phase of the software.

Develop Software with Secure Features

Follow the Saltzer and Schroeder list of principles for building good secure software:

Design principle What does it mean? Example
Economy of mechanism Keep the design as simple and small as possible. Modular, minimalistic code developed using TDD (only write code to make the tests pass),centralized services
Fail-safe defaults Access denied by default, and permitted explicitly Firewalls are configured such that the default is to deny access
Complete mediation Every access to every object checked for authority. Rely as little as possible on access decisions retrieved from a cache. File permissions tend to reflect this model: the operating system checks the user requesting access against the file’s ACL.
Open design The design should not be secret, the implementation of safeguards is. Cryptographic algorithms
Separation of privilege More than one condition is required to authorize a task Bank lockers, nuclear launch systems (I hope!), that use two separate keys to operate
Least privilege Invoke minimum privileges running web servers using accounts with reduced privileges
Least common mechanisms Minimize the amount of mechanism common to more than one user and depended on by all users. Role based dynamic libraries
Psychological acceptability Requires the policy interface to reflect the user’s mental model of protection Help dialogs, intuitive iconography

Use secure-coding best practices

A stitch in time saves nine. It is always cheaper to build secure applications then to correct the security bugs later. Making security a check-off item for code reviews. OwASP and Apple have good checklists that software development organizations would do well to follow.

Use source code analysis tools

Whenever possible integrate source-code analysis tools (Fortify, FindBugs, etc.) in to your development process. However, it is not a substitute for testing. And it can generate a lot of false positives initially. But it can be a useful tool in locating security vulnerabilities.

Use testing techniques to stress security features

Focus on abuse/negative test cases as well: test the evil path in unit and integration tests using. Running penetration tests against new releases as part of the release process using tools like SkipFish, MetaSploit etc.

Password and Session Management

Passwords are the keys to a user’s account. It is often a target to gain unauthorized access. Password strength is key in thwarting such attacks. Enforce a strong password policy. However, it does present an inconvenience to users who now have to remember long complex passwords. A good strategy around that is to not have to manage passwords and authentication at all. Implement single sign-on, so that users need to only have to authenticate once. Or, offload it to an 3rd party authentication authority using OAuth. However, if you do implement authentication ensure that

  • user credentials are never stored in clear text — makes sure that they are hashed and salted
  • never send user credentials over an un-encrypted connection

Session tokens should always be generated on the server. Use existing application frameworks for session management as much as possible. For example, Rails, Play, Lift, J2EE/Spring, ASP.Net etc., come with features around strongly encrypted sessions, protections against attacks like CSRF, XSS, SQL injection, session replay, timeouts etc. That wheel’s been invented and rolls well.

The bad guys will still be targeting the cloud in 2014, with cloud service provider employees the main focus. Successful phishing attacks on staff, especially if the password re-use jackpot is hit, have been the door to entry of many online member databases during 2013. Ultimately, with innovation and planning, cloud services could reduce business risks by providing greater flexibility, resiliency and security

Input validation

One of the most effective safeguards against hacking attacks is to perform input validation. Input validation should take place on the server side for all data. Validation criteria should be set for input fields. All data should be encoded before use. Input validation techniques along with proper encoding help block against many attack vectors like XSS, SQL Injection and remote file inclusion.

Use open source cryptography

The only way to secure architectures is to use systems, algorithms, and implementations that are well known to be safe, and which have been peer reviewed in the open by lots of people, and which have withstood a lot of hacking attempts in production already. Custom or proprietary solutions often have not gone through this process, and as a result are often riddled with security weaknesses that can and will be exploited.

Configuring for security

What does this mean? It is simply managing and defining a process to ensure that the systems the software runs on have been secured adequately. Harden your infrastructure by:

  1. Ensuring all software is updated to the latest security patches. This includes the OS, Web/App server, databases. And, any other component in the system architecture. Define a process to keep these components abreast of the latest updates and patches and deploying them to all deployed environments in a timely manner. The process should also include seemingly innocuous peripherals such as networked printers, VOIP phones
  2. Turning off or disabling all unused and unnecessary features (e.g. ports, services, accounts )
  3. Changing all default passwords and or accounts.
  4. Ensuring that error handling does not reveal any stack traces or other sensitive information to the users.

In 2013, a third of the most serious threats facing enterprises were associated with worms (Win32/Conficker, INF/Autorun, Win32/Dorkbot) infecting windows systems. Worms are commonly spread through network drives, abusing the Autorun feature or exploiting weak passwords.

The Conficker worm exploited weak passwords. Once it compromised a system, it could steal the credentials of an IT administrator to spread on the internal network.

Manage data exposure

Analyze your data characteristics and determine which data is sensitive enough to require extra protection ( e.g. passwords, credit card numbers, health records, etc.). Ensure that such data is never backed up or transmitted in clear text. Ensure that strong/updated cryptographic algorithms are used to encrypt such data. Don’t store data unnecessarily. Data you don’t have is impossible to steal. For example, don’t store credit card numbers if you use a payment gateway, simply store only the last 4 digits for tracking purposes. Also, no sensitive information such as IP addresses in logs. Ensure passwords are stored with algorithms specifically designed for password encryption such as bcrypt, PBKDF2 or scrpyt.

Manage the end-points

BYOD is here to stay. CEOs want to use the latest in mobile technology — the President of the United States uses a tablet. It has it’s benefits – reduced capital expense, and productivity gains. While there are some businesses that may do well to steer away from this trend, most corporations can benefit if they manage this trend with their organization, rather than fight it. Put in a plan to train employees & raise awareness regarding the risks using their devices to access corporate data. Consider implementing a MDM (Mobile Device Management) solution to manage access to corporate apps and access on employee devices. MobileIron, Airwatch/VmWare, Citrix among others have MDM suites to manage BYOD issues. Also, consider reigning it in using a CYOD (Choose your Own Device) option — where the business has a list of approved devices the employees chose from, to gain access to corporate systems.

Employee Training

Education is a good risk-mitigator as well. Training developers to follow good design principles (including the ones mentioned above) helps build good secure software. Training operations to configure for security means there will be fewer if not none, changes that could compromise security, for example, inadvertently removing access restrictions on network drives, which happens more often than one would imagine.

Stay Paranoid

It is impossible to protect anything that is in the users hands from being read in its completeness. For example, the attempts to encrypt digital media like DVDs have all been hacked with ease. There is no platform or app that hasn’t been hacked and studied in detail by the community, even if it requires modifying the hardware that the code is running on, either for establishing a street cred or for financial gain. Sony’s Playstation was protected on the hardware level against hacks. Even that wasn’t enough protection apparently — you could buy extra chips (called flashers) could solder onto chips on the motherboard. You could then downgrade the firmware and load pirated games onto it.

Stay vigilant for any signs of abnormal activity on your network. No alarm should be treated as to insignificant to investigate. Just recently, a large store chain revealed that they did not take early warning signs of abnormal activity too seriously, eventually resulting in a data breach that impacted 70 million customers, and more importantly a loss of trust and goodwill.

Every piece of code, data, or hardware that is in the users hands must be considered read and understood in its completeness, even if that means grinding away a hardware chip layer by layer and photographing the internals. People do that. Obfuscation might be helpful to reduce the size of the shipped binaries, but it doesn’t prevent reading and manipulation of the compiled code.

No input from the outside world, even if it seems to come from our own client apps over secure channels, must be trusted. The clients might be (and will be) compromised, and will send malicious data, no matter how obfuscated or protected they are. If there are truly security-relevant things, they have to remain on the servers, and have to happen there.

Further reading

Using ScalaTest in a Play Framework Application

The Play Framework is configured out of the box for specs2. However, some people (including the author) prefer ScalaTest. Play actually doesn’t really care what testing framework you use, but there is one gotcha you need to know about.

The default Play configuration sets the test options sequential true junitxml console, which are specific to specs2. Because ScalaTest doesn’t understand these options, you need to specify testOptions in Test := Nil in your build definition.

Starting with Play 2.2, build.sbt is used for the build definition. Simply add the line:

1
testOptions in Test := Nil

For Play 2.0 or 2.1, project/Build.scala is used. You’ll have to do something like the following:

1
2
3
4
val main = PlayProject(appName, appVersion, appDependencies, mainLang = SCALA).settings(
  // other project settings here
  testOptions in Test := Nil
)

Also, don’t forget to add ScalaTest as a project dependency. At the time of writing, the latest version can be specified with the following:

1
"org.scalatest" % "scalatest_2.10" % "2.0" % "test"

Recursive Type Signatures in Scala

Have you seen a type signature like this before?

1
trait T[U <: T[U]]

If you’re like me, you’ve come across this type signature, and you’re wondering what the heck it means. You likely Googled something like “recursive type” or “self-referential type” and ended up here.

So what does this type signature mean? Why not use trait T[U]?

To understand the meaning of T[U <: T[U]], we’ll work through a simple example.

Example

Suppose we want database entities with CRUD methods. We could define them like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// I don't know about you, but I like to name my fruit.
case class Apple(name: String, price: Double) {
  def create(entityData: String): Apple
  def read(id: String): Option[Apple]
  def update(f: Apple => Apple): Apple
  def delete(id: String): Unit
}

case class Bird(name: String, birthday: DateTime) {
  def create(entityData: String): Bird
  def read(id: String): Option[Bird]
  def update(f: Bird => Bird): Bird
  def delete(id: String): Unit
}

But we can see that these classes look nearly identical. In addition, any new CRUD entities we add will expose the same CRUD methods. Now, should we abstract the interface into a trait? Let’s see what happens when we do:

1
2
3
4
5
6
7
8
9
10
trait CrudEntity {
  def create(entityData: String): CrudEntity
  def read(id: String): Option[CrudEntity]
  def update(f: CrudEntity => CrudEntity): CrudEntity
  def delete(id: String): Unit
}

case class Apple(name: String, age: Int) extends CrudEntity

case class Bird(name: String, hobby: String) extends CrudEntity

Well this sucks. Our method signatures don’t fully express what we want. We’ve lost the ability to ensure that e.g. calling update on an Apple returns an Apple. As is, it can return any CrudEntity. Let’s try to regain some specificity by adding a type parameter to our CrudEntity trait:

1
2
3
4
5
6
7
8
9
10
trait CrudEntity_2[E] {
  def create(entityData: String): E
  def read(id: String): Option[E]
  def update(f: E => E): E
  def delete(id: String): Unit
}

case class Apple(name: String, age: Int) extends CrudEntity_2[Apple]

case class Bird(name: String, hobby: String) extends CrudEntity_2[Bird]

Okay, better. But we still haven’t locked this down. Our types don’t yet express exactly what we want. Do you see the problem?

The problem is that someone can extend CrudEntity_2 in a way we didn’t intend them to:

1
case class Orange(name: String, bankAccount: Double) extends CrudEntity_2[FloobyDust]

Whoa! In the code above, CrudEntity_2[E] does not restrict the type of E, so they can use anything they want, without complaint from the compiler — FloobyDust, Potato, BurritoAstronaut, you name it.

This is no bueno. Instead, we’d like them to get a big, fat compiler error if they try extending anything other than CrudEntity_2[Orange]. How do we ensure that E matches the class we’re defining?

Let’s try defining CrudEntity again. This time, we’ll use type bounds:

1
2
3
4
5
6
7
8
9
10
trait CrudEntity_3[E <: CrudEntity_3[E]] {
  def create(entityData: String): E
  def read(id: String): Option[E]
  def update(f: E => E): E
  def delete(id: String): Unit
}

case class Apple(name: String, age: Int) extends CrudEntity_2[Apple]

case class Bird(name: String, hobby: String) extends CrudEntity_2[Bird]

Better. Now we’ve constrained E to be a subtype of CrudEntity. No more FloobyDust. But there’s one last problem, and you can probably guess what it is. We haven’t yet ensured that E matches our class type, only that it subclasses CrudEntity. CrudEntity is still open for abuse:

1
case class Orange(name: String, age: Int) extends CrudEntity_3[Apple]

Yuck! To take care of this, we need a way to ensure that e.g. Orange extends CrudEntity_3[Orange]. For this assurance, we’ll use a self type.

Here is our final definition of CrudEntity, which uses a self type:

1
2
3
4
5
6
trait CrudEntity[E <: CrudEntity[E]] { self: E =>
  def create(entityData: String): E
  def read(id: String): Option[E]
  def update(f: E => E): E
  def delete(id: String): Unit
}

self: E => ensures that any concrete class that extends CrudEntity must be of type E and that code like

1
case class Orange(name: String, age: Int) extends CrudEntity[Apple]

will get rejected by the compiler because Orange is not of type Apple.

Now we can rest, confident that our definition of CrudEntity ensures that any subtype of CrudEntity[E] must in fact be an E. This definition gives us the semantics we desire, and enlists the compiler to reject all code in violation.

Speed Up Your Rails Specs by 10x

One of the primary reasons people end up being lax in letting specifications drive the development of their Rails applications is the time it takes to get feedback from running the suite of specifications. A number of tools have been built to help alleviate this pain like Spork, Zeus, and Spring. In fact Rails 4.1 will now come with Spring standard. Unfortunately, these tools are just crutches that tackle the symptoms of the problem rather than the problem itself. The actual problem is writing tightly coupled code that expects to have the full Rails framework always present, which is slow to start up.

Developing Decoupled Code

The solution is to write code that is isolated and decouple your components from as much of the system as possible. In other words, write SOLID Rails code. As a specific example, one might typically directly use a model class to create an instance. Instead we can use dependency injection to remove hard coded references to classes. We just need to make sure we safely reference the defaults using either block notation or a lazy evaluating ||=. Below we have a service that needs to create Widgets which happen to be ActiveRecord models. Instead of directly referencing the Widget class, we use lazy evalutation in our chosen injection method. This allows us to decouple our code and not need ActiveRecord loaded.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# A tightly coupled class. We don't want this.
class MyService
  def create_widget(params)
    Widget.create(params)
  end
end

# We can inject in the initializer
class MyService
  attr_reader :widget_factory

  def initialize(dependencies={})
    @widget_factory = dependencies.fetch(:widget_factory) { Widget }
  end

  def create_widget(params)
    widget_factory.create(params)
  end
end

# Or we can explictly inject via a setter with a lazy reader
class MyService
  attr_writer :widget_factory

  def widget_factory
    @widget_factory ||= Widget
  end

  def create_widget(params)
    widget_factory.create(params)
  end
end

# A specification injecting the dependency using the second method
describe MyService do
  subject(:service) { MyService.new }
  let(:widget_factory) { double 'widget_factory', create: nil }
  before { service.widget_factory = widget_factory }

  it 'creates a widget with the factory' do
    service.create_widget({name: 'sprocket'})
    expect(widget_factory).to have_received(:create).with({name: 'sprocket'})
  end
end

A Base Rails-free Configuration

When writing your applications in this way you can then start to restructure how you setup your specifications and minimize the required environment to run both your specification and your code fulfilling the specification. The typical spec_helper.rb will have a line like this:

1
require File.expand_path("../../config/environment", __FILE__)

This is what loads your entire Rails application and slows down the running of your tests. To make your specifications faster, you need to use a configuration file that does not contain this line. So let’s start by creating a very light weight base_spec_helper.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
ENV["RAILS_ENV"] ||= 'test'
require 'rubygems'

RAILS_ROOT = File.expand_path('../..', __FILE__)
Dir[File.join(RAILS_ROOT, 'spec/support/**/*.rb')].each {|f| require f}

RSpec.configure do |config|
  config.mock_with :rspec
  config.order = 'random'
  # Your prefered config options go here
end

require 'active_support'
require 'active_support/dependencies'

We are requiring active_support and active_support/dependencies so we can have access to the autoloader Rails uses without actually loading up all of Rails. It is fairly light weight and the convienence outweighs the cost. In each spec helper which requires this base we will add the relevant portions of our app into the ActiveSupport::Dependencies.autoload_paths.

Plain Ruby Object Specifications

Depending on the part of application that you are specifying, you can create spec helpers specific to what you need in any one context. For example, the simplest would be one for specifying any type of pure Ruby class such as a service class. A sample services_spec_helper.rb might be:

1
2
3
require 'base_spec_helper'
Dir[File.join(RAILS_ROOT, "spec/support_services/**/*.rb")].each {|f| require f}
ActiveSupport::Dependencies.autoload_paths << "#{RAILS_ROOT}/app/services"

Decorator Specifications

For your decorators, you might choose to use Draper and your decorators_spec_helper.rb might look like:

1
2
3
4
5
require 'base_spec_helper'
require 'draper'
Draper::ViewContext.test_strategy :fast
Dir[File.join(RAILS_ROOT, "spec/support_decorators/**/*.rb")].each {|f| require f}
ActiveSupport::Dependencies.autoload_paths << "#{RAILS_ROOT}/app/decorators"

Model Specifications

Testing models needs a little bit more. Assuming you are using ActiveRecord you’ll need to include that as well as establish a connection to your database. We won’t include factory_girl or database_cleaner as most of your tests should not be actually creating database objects. In fact, the only place you really need to actually create an object in the database is when testing uniqueness validations. When you do need to create something you can just manually clean it up or use a transaction. So a sample models_spec_helper.rb can look like this:

1
2
3
4
5
6
7
8
9
10
11
12
require 'base_spec_helper'
require 'active_record'
# RSpec has some nice matchers for models so we'll pull them in
require 'rspec/rails/extensions/active_record/base'
Dir[File.join(RAILS_ROOT, "spec/support_models/**/*.rb")].each {|f| require f}

# Manually connect to the database
ActiveRecord::Base.establish_connection(
  YAML.load(File.read(RAILS_ROOT + '/config/database.yml'))['test']
)

ActiveSupport::Dependencies.autoload_paths << "#{RAILS_ROOT}/app/models"

Feature Specifications

Finally, when creating feature specs, we do need our full Rails stack and our feature_spec_helper.rb is going to look very similar to what your current spec_helper.rb looks like.

Summary

I found myself using varitions on the above spec helpers in projects I work on and decided I would write a set of generators to make it easier to bootstrap the project. The gem can be found at https://github.com/Originate/rails_spec_harness

While introducing these changes into existing projects I have found speed increases of 8-12 times. The worst project experienced a 27x increase once these changes and the corresponding changes in coding habits where applied. As an example I made a specification with 4 examples for a plain Ruby class. I then used the time command line utility to measure running rspec with the minimal spec helper as well as the full Rails spec helper and found the following:

Spec Helper Real User Sys RSpec Reported
Full Rails 4.913s 2.521s 1.183s 0.0706s
Minimal 0.492s 0.407s 0.080s 0.0057s

Write SOLID code, isolate your specifications and enjoy a fun and sane development experience.

TDD Is BS**

** BS = Behavior Specification

tl/dr

The misleading terminology around TDD is responsible for a good amount of confusion and hesitation in this area, as well as over- and undertesting. Calling tests specifications makes the concept more intuitively available in several ways.

One does not simply test something before it is built

testing thin air

One cannot test something before it is built. Asking people to write tests before they write code is like asking them to test-drive a new car before it even exists. This isn’t possible. Similarly, a piece of code can only be tested after it is completely written. We can’t test thin air.

Asking people to write tests before they write code is counter-intuitive and misleading. Resistance or discomfort with the idea is only natural.

specifications instead of tests

A better perspective is to treat tests as specifications for the application in the works, i.e. descriptions of what our code should do.

It is completely plausible to reflect upfront about what I’m going to build – and to write down these thoughts – before starting to build anything serious, costly, and/or sizeable. Especially when working within a team.

It is similarly intuitive that starting to think about these specifications – or writing them down – after a product has already been built makes very little sense, provides very little benefit to the development process, and is actually more a hassle than useful at that point. Specifications need to be defined, agreed upon, and considered before we build the actual system.

behavior-driven development

Specifications just define expected behavior. They don’t dictate how in particular to implement things. The idea of “testing”, on the other hand, naturally means checking a particular implementation in all its gory details.

This distinction is important. Ideally, our tests/specs should not be concerned with implementation details and private methods. Otherwise they become brittle, and will break unnecessarily each time we refactor or clean up something. All that we really care about when running our specs is that the code still works, not how it still works. The how should be discussed in code reviews.

user perspective

Developers tend to occasionally forget about the user perspective while being deeply immersed in the code. Specifications give them a natural place to muse over the user-visible behavior of the application, from the perspective of the user of their code. Specifications thereby form the missing link between business requirements and their technical implementation. Features specs (aka “integration tests”) are runnable user stories.

Tests, on the other hand, just assert implementation details from a technical perspective. Even integration tests just check that the different parts and layers of the application have been integrated properly.

conclusion

While there is often no real technical difference between “tests” and “specs”, the term “test-driven development” is misleading, while “specification-driven development” is intuitive and describes what we actually do here better. We should consider using the latter term in favor of the former. Let’s write “specs” instead of “tests”, and “feature specifications” instead of “integration tests” from now on.

Much love, and happy coding! :)