Developer Blog

Secure Your Software for Battle

A World Wide War is being fought across the software world. The warring factions are engaging in battles in IT centers, corporate offices and your homes; between security experts on one side and cyber-criminals on the other.

This world we live in is getting increasingly connected and computerized. Consumers, organizations and governments alike are conducting business electronically at an ever increasing pace. This requires organizations to store information about their consumers. This includes financial, personal data as well as data about their online and spending habits.

Corporations need to secure this data because

  • privacy laws demand it
  • consumers demand it
  • it gives them a competitive advantage

Naturally, this data is prized by hackers, criminals, competitive entities and unfriendly state agents. This battle of wits between the security experts and the cyber-criminals is an ongoing one and is only going to get bigger.

The systems that process and store data also become targets — either to disable them, or to access, corrupt or destroy data.

Securing the data and the systems that manage it is now a high priority and high profile task. Or, it should be if they mean business. It is no longer a nice-to-have or even a should-have. It is now a must-have.

Companies that don’t have their act together … are in for a crude awakening in the internet age, to significant damage to their business and reputation (http://www.businessweek.com/articles/2014-03-13/target-missed-alarms-in-epic-hack-of-credit-card-data), and in some cases to an end of their very existence (http://www.forbes.com/sites/cameronkeng/2014/02/25/bitcoins-mt-gox-shuts-down-loses-409200000-dollars-recovery-steps-and-taking-your-tax-losses).

The War Zone

The increasing mobility of data in corporations is one of the biggest challenges faced in the last couple of years. Users are feeling empowered to access data from anywhere. The BYOD (Bring Your Own Device) trend is bringing in new attack vectors. The data access end-point is also going through a transformation. No longer are these Windows-based, but now a diverse mix of different systems. Some of them are still in their infancy compared to long established platforms like Windows/Unix. These platforms are new targets, and malware for mobile devices is now a rapidly growing threat.

Employees are bringing in their smartphones, tablets and next-gen ultrabooks in to the corporate networks. IT professionals are being required to secure sensitive data on devices the have little control over.

The growing popularity of the “Internet of things” (smartphones, tablets, wearable gadgets, and interconnected devices) makes a fluid situation even more dynamic for security experts. They present ripe targets for cyber-criminals looking to either find access to your data or cause a large-scale disruption.

Cyber-criminals are getting sophisticated themselves. They are using underground online marketplaces to sell their wares and services. The Blackhole exploit kit accounted for almost a third of all malware infestations in 2012. And, it spawned a new generation of botnet exploits that drew on its source code for inspiration.

The Attacks

As the arms race unfolds between the cyber-criminals and security experts, here are some security threats that are at the top the list.

Cloud

Cloud computing continues to grow in popularity — and so too will the number of security threats targeting the cloud. It’s important that businesses realize that whilst they may outsource the handling and storage of their data, they can’t outsource the responsibility for the data itself. If their provider’s systems are breached, and data is exposed, the businesses are responsible.

Therefore, any risks need to be assessed in the same way as they would if they were holding the data internally. Other issues organizations should consider include: where will the data be stored, what happens to any data if organizations switch providers and what steps are being taken to secure the data on their provider’s systems, including how they prevent other customers from accessing it.

The bad guys will look to targeting the cloud in 2014, with cloud service provider employees the main focus. Successful phishing attacks on staff, especially if the password re-use jackpot is hit, have been the door to entry of many online member databases during 2013.

APT

APT or Advanced Persistent Threat is a set of well-camouflaged, focused, continuous hacking activities against a single target. APT activities require a high-level of stealth sustained over a period of time. These employ advanced malware to exploit vulnerabilities in the targets. And these are usually orchestrated for business or political motives. Flame/Gauss and Stuxnet are examples of recent APT attacks used to attack financial, government and military institutions.

Social Engineering

It’s a tried-and-true tactic in both the physical and digital worlds – social engineering. Before the digital age, this meant sneaking one’s way past the security desk with glib talk vs. a cleverly-worded email. Now social engineering has moved onto social networks, including Facebook and LinkedIn.

Attackers are increasing their use of social engineering, which goes beyond calling targeted employees and trying to trick them into giving up information. Cyber-criminals will try to hide malware using deceitful tactics to trick you into installing it.

The task of cybercriminal is a lot simpler — check if details are already posted on social networks. Social networks are about networking people; a convincing-looking profile of a company or person followed by a friend/connection request can be enough to get a social engineering attack started. Sarah Palin’s Yahoo email account was compromised using information that was publicly available on her social media profiles (birth name, date of birth etc).

Password Management

We live in a world where people are increasingly connected through social networking, cloud services & smart devices. This presents a password management headache for users and, by extension, the services they use. People tend to reuse passwords. More than half users reuse passwords for their online accounts. A data breach in one vendor/service provider can potentially put their accounts at other services at risk. If that weren’t scary enough, over a quarter tend to use easy-to-remember passwords such as birthdays or people’s names, opening the door for their online accounts to be hacked into by criminals. It’s worrying aspect for corporations that so many people are making life so easy for cyber-criminals hackers. Especially because simple to make strong password security a part of one’s everyday life.

BYOD (Bring Your Own Device )

This is one threat vector that will give security professionals many sleepless nights. BYOD (Bring Your Own Device) increases data leakage potential especially from devices unprotected by device specific counter-measures such as passcode/passphrase protection. etc. Most organizations going down the BYOD path aren’t implementing appropriate training for employees.

Attackers will try and circumvent app review and detection mechanisms on these devices. If possible their apps will mimic the UI of the native settings page and trick the user in to granting them admin privileges on the device. The device sensors such as GPS, microphone and cameras coupled with the ability to network over WIFI and mobile networks, could become tools for engineering further attacks.

The main areas of risk include: * Data loss — lost or stolen devices * ID theft — thieves logging in to your online accounts using saved credentials on your stolen device * Data leakage — via bogus WIFI access points and malware

Botnets

Essentially, a botnet is a collection of networked computers running a program to perform a task. Initially, most botnets were employed to perform legitimate tasks such controlling IRC channels. Later they were deployed to distribute spam, or perform DDos attacks.

Botnet operators are beginning to design systems that are more adaptive and redundant than many corporate and government networks. Botnets such as Gameover have replaced the traditional command and control link with a peer-to-peer network of infected systems. Controlling this agile attack vector before it can be used as an advanced persistent threat (APT) and migrates into smart mobile devices is crucial.

We’re also likely to see more mobile botnets, of the sort created using the RootSmart backdoor in Q1 2012. In order to prevent falling victim to mobile malware, businesses should install anti-malware protection on their Android devices, secure the data held on them and make sure that this can be wiped remotely if the device is lost or stolen. Businesses should also develop a policy for staff on how to reduce the risks from mobile devices. This should include not rooting the device, avoiding public Wi-Fi networks for confidential transactions, not relying solely on a simple PIN and only installing apps from trusted sources.

Spam

People still send emails. So do bad guys, and they will keep doing so long as people keep using email. You see spam that link to financial scams that are now mostly ignored, spam that link to malware designed to install botnet agents, spam that attempt to seem legit by linking to current events. Phishing via deceptive links in spam is a very common attack. Spammers are also wising up to having their bots and servers taken down. Snowshoe spam is their innovation against this countermeasure. This involves distributing their spamming across multiple ip addresses, spreading the load. Hence, the metaphor.

The Defense

So what are organizations to do to deflect security attacks? It used to be about securing the perimeter in the past. Now the perimeter is blurred due to BYOD and cloud services trends.

There is no one good answer. But most of them would include adopting treating security as a feature, good engineering practices, implementing systems, good employer training, a good plan to manage responses to attacks and finally transparency.

At Originate, when we work with partners, we emphasize security up front as a key element of the systems we build. In most cases, we have been able to educate and received buy-in from our partners. In one case however, we received push back on implementing a policy to enforce strong passwords — the partners feared having a strong password policy would impact user registration. Guess what, on the first day of launch, the first account to be compromised was the admin account. Fortunately, we were monitoring all activity and were able to retake control of the account and secure it. Needless to say, we got the go-ahead to implement the strong-password policy.

Make security a feature ( and not a Non-functional requirement )

When writing applications, security aspects should be first-class requirements. It cannot be a bolt-on later. Build the app with the best security practices from the start. According to security firms FireEye and Secunia, the real security problems in this decade are not in our operating systems but in the applications we run on them.

Design Software with Secure Features

Security issues in design and semantic flaws (ones that are not syntactic or code related), such as business logic flaws, cannot be detected in code and need to be inspected by performing threat models and abuse cases modeling during the design stage.

Threat modeling is an iterative technique used to identify the threats to the software being built. It starts by identifying the security objectives of the software and profiles it. It breaks the software into physical and logical constructs generating the software context that includes data flow diagrams, and end-to-end deployment scenarios, identifying entry and exit points, protocols, components, identities, and services.

Threat Modeling is performed during the design stage so that necessary security controls (safeguards) can be developed during the development phase of the software.

Develop Software with Secure Features

Follow the Saltzer and Schroeder list of principles for building good secure software:

Design principle What does it mean? Example
Economy of mechanism Keep the design as simple and small as possible. Modular, minimalistic code developed using TDD (only write code to make the tests pass),centralized services
Fail-safe defaults Access denied by default, and permitted explicitly Firewalls are configured such that the default is to deny access
Complete mediation Every access to every object checked for authority. Rely as little as possible on access decisions retrieved from a cache. File permissions tend to reflect this model: the operating system checks the user requesting access against the file’s ACL.
Open design The design should not be secret, the implementation of safeguards is. Cryptographic algorithms
Separation of privilege More than one condition is required to authorize a task Bank lockers, nuclear launch systems (I hope!), that use two separate keys to operate
Least privilege Invoke minimum privileges running web servers using accounts with reduced privileges
Least common mechanisms Minimize the amount of mechanism common to more than one user and depended on by all users. Role based dynamic libraries
Psychological acceptability Requires the policy interface to reflect the user’s mental model of protection Help dialogs, intuitive iconography

Use secure-coding best practices

A stitch in time saves nine. It is always cheaper to build secure applications then to correct the security bugs later. Making security a check-off item for code reviews. OwASP and Apple have good checklists that software development organizations would do well to follow.

Use source code analysis tools

Whenever possible integrate source-code analysis tools (Fortify, FindBugs, etc.) in to your development process. However, it is not a substitute for testing. And it can generate a lot of false positives initially. But it can be a useful tool in locating security vulnerabilities.

Use testing techniques to stress security features

Focus on abuse/negative test cases as well: test the evil path in unit and integration tests using. Running penetration tests against new releases as part of the release process using tools like SkipFish, MetaSploit etc.

Password and Session Management

Passwords are the keys to a user’s account. It is often a target to gain unauthorized access. Password strength is key in thwarting such attacks. Enforce a strong password policy. However, it does present an inconvenience to users who now have to remember long complex passwords. A good strategy around that is to not have to manage passwords and authentication at all. Implement single sign-on, so that users need to only have to authenticate once. Or, offload it to an 3rd party authentication authority using OAuth. However, if you do implement authentication ensure that

  • user credentials are never stored in clear text — makes sure that they are hashed and salted
  • never send user credentials over an un-encrypted connection

Session tokens should always be generated on the server. Use existing application frameworks for session management as much as possible. For example, Rails, Play, Lift, J2EE/Spring, ASP.Net etc., come with features around strongly encrypted sessions, protections against attacks like CSRF, XSS, SQL injection, session replay, timeouts etc. That wheel’s been invented and rolls well.

The bad guys will still be targeting the cloud in 2014, with cloud service provider employees the main focus. Successful phishing attacks on staff, especially if the password re-use jackpot is hit, have been the door to entry of many online member databases during 2013. Ultimately, with innovation and planning, cloud services could reduce business risks by providing greater flexibility, resiliency and security

Input validation

One of the most effective safeguards against hacking attacks is to perform input validation. Input validation should take place on the server side for all data. Validation criteria should be set for input fields. All data should be encoded before use. Input validation techniques along with proper encoding help block against many attack vectors like XSS, SQL Injection and remote file inclusion.

Use open source cryptography

The only way to secure architectures is to use systems, algorithms, and implementations that are well known to be safe, and which have been peer reviewed in the open by lots of people, and which have withstood a lot of hacking attempts in production already. Custom or proprietary solutions often have not gone through this process, and as a result are often riddled with security weaknesses that can and will be exploited.

Configuring for security

What does this mean? It is simply managing and defining a process to ensure that the systems the software runs on have been secured adequately. Harden your infrastructure by:

  1. Ensuring all software is updated to the latest security patches. This includes the OS, Web/App server, databases. And, any other component in the system architecture. Define a process to keep these components abreast of the latest updates and patches and deploying them to all deployed environments in a timely manner. The process should also include seemingly innocuous peripherals such as networked printers, VOIP phones
  2. Turning off or disabling all unused and unnecessary features (e.g. ports, services, accounts )
  3. Changing all default passwords and or accounts.
  4. Ensuring that error handling does not reveal any stack traces or other sensitive information to the users.

In 2013, a third of the most serious threats facing enterprises were associated with worms (Win32/Conficker, INF/Autorun, Win32/Dorkbot) infecting windows systems. Worms are commonly spread through network drives, abusing the Autorun feature or exploiting weak passwords.

The Conficker worm exploited weak passwords. Once it compromised a system, it could steal the credentials of an IT administrator to spread on the internal network.

Manage data exposure

Analyze your data characteristics and determine which data is sensitive enough to require extra protection ( e.g. passwords, credit card numbers, health records, etc.). Ensure that such data is never backed up or transmitted in clear text. Ensure that strong/updated cryptographic algorithms are used to encrypt such data. Don’t store data unnecessarily. Data you don’t have is impossible to steal. For example, don’t store credit card numbers if you use a payment gateway, simply store only the last 4 digits for tracking purposes. Also, no sensitive information such as IP addresses in logs. Ensure passwords are stored with algorithms specifically designed for password encryption such as bcrypt, PBKDF2 or scrpyt.

Manage the end-points

BYOD is here to stay. CEOs want to use the latest in mobile technology — the President of the United States uses a tablet. It has it’s benefits – reduced capital expense, and productivity gains. While there are some businesses that may do well to steer away from this trend, most corporations can benefit if they manage this trend with their organization, rather than fight it. Put in a plan to train employees & raise awareness regarding the risks using their devices to access corporate data. Consider implementing a MDM (Mobile Device Management) solution to manage access to corporate apps and access on employee devices. MobileIron, Airwatch/VmWare, Citrix among others have MDM suites to manage BYOD issues. Also, consider reigning it in using a CYOD (Choose your Own Device) option — where the business has a list of approved devices the employees chose from, to gain access to corporate systems.

Employee Training

Education is a good risk-mitigator as well. Training developers to follow good design principles (including the ones mentioned above) helps build good secure software. Training operations to configure for security means there will be fewer if not none, changes that could compromise security, for example, inadvertently removing access restrictions on network drives, which happens more often than one would imagine.

Stay Paranoid

It is impossible to protect anything that is in the users hands from being read in its completeness. For example, the attempts to encrypt digital media like DVDs have all been hacked with ease. There is no platform or app that hasn’t been hacked and studied in detail by the community, even if it requires modifying the hardware that the code is running on, either for establishing a street cred or for financial gain. Sony’s Playstation was protected on the hardware level against hacks. Even that wasn’t enough protection apparently — you could buy extra chips (called flashers) could solder onto chips on the motherboard. You could then downgrade the firmware and load pirated games onto it.

Stay vigilant for any signs of abnormal activity on your network. No alarm should be treated as to insignificant to investigate. Just recently, a large store chain revealed that they did not take early warning signs of abnormal activity too seriously, eventually resulting in a data breach that impacted 70 million customers, and more importantly a loss of trust and goodwill.

Every piece of code, data, or hardware that is in the users hands must be considered read and understood in its completeness, even if that means grinding away a hardware chip layer by layer and photographing the internals. People do that. Obfuscation might be helpful to reduce the size of the shipped binaries, but it doesn’t prevent reading and manipulation of the compiled code.

No input from the outside world, even if it seems to come from our own client apps over secure channels, must be trusted. The clients might be (and will be) compromised, and will send malicious data, no matter how obfuscated or protected they are. If there are truly security-relevant things, they have to remain on the servers, and have to happen there.

Further reading

Using ScalaTest in a Play Framework Application

The Play Framework is configured out of the box for specs2. However, some people (including the author) prefer ScalaTest. Play actually doesn’t really care what testing framework you use, but there is one gotcha you need to know about.

The default Play configuration sets the test options sequential true junitxml console, which are specific to specs2. Because ScalaTest doesn’t understand these options, you need to specify testOptions in Test := Nil in your build definition.

Starting with Play 2.2, build.sbt is used for the build definition. Simply add the line:

1
testOptions in Test := Nil

For Play 2.0 or 2.1, project/Build.scala is used. You’ll have to do something like the following:

1
2
3
4
val main = PlayProject(appName, appVersion, appDependencies, mainLang = SCALA).settings(
  // other project settings here
  testOptions in Test := Nil
)

Also, don’t forget to add ScalaTest as a project dependency. At the time of writing, the latest version can be specified with the following:

1
"org.scalatest" % "scalatest_2.10" % "2.0" % "test"

Recursive Type Signatures in Scala

Have you seen a type signature like this before?

1
trait T[U <: T[U]]

If you’re like me, you’ve come across this type signature, and you’re wondering what the heck it means. You likely Googled something like “recursive type” or “self-referential type” and ended up here.

So what does this type signature mean? Why not use trait T[U]?

To understand the meaning of T[U <: T[U]], we’ll work through a simple example.

Example

Suppose we want database entities with CRUD methods. We could define them like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// I don't know about you, but I like to name my fruit.
case class Apple(name: String, price: Double) {
  def create(entityData: String): Apple
  def read(id: String): Option[Apple]
  def update(f: Apple => Apple): Apple
  def delete(id: String): Unit
}

case class Bird(name: String, birthday: DateTime) {
  def create(entityData: String): Bird
  def read(id: String): Option[Bird]
  def update(f: Bird => Bird): Bird
  def delete(id: String): Unit
}

But we can see that these classes look nearly identical. In addition, any new CRUD entities we add will expose the same CRUD methods. Now, should we abstract the interface into a trait? Let’s see what happens when we do:

1
2
3
4
5
6
7
8
9
10
trait CrudEntity {
  def create(entityData: String): CrudEntity
  def read(id: String): Option[CrudEntity]
  def update(f: CrudEntity => CrudEntity): CrudEntity
  def delete(id: String): Unit
}

case class Apple(name: String, age: Int) extends CrudEntity

case class Bird(name: String, hobby: String) extends CrudEntity

Well this sucks. Our method signatures don’t fully express what we want. We’ve lost the ability to ensure that e.g. calling update on an Apple returns an Apple. As is, it can return any CrudEntity. Let’s try to regain some specificity by adding a type parameter to our CrudEntity trait:

1
2
3
4
5
6
7
8
9
10
trait CrudEntity_2[E] {
  def create(entityData: String): E
  def read(id: String): Option[E]
  def update(f: E => E): E
  def delete(id: String): Unit
}

case class Apple(name: String, age: Int) extends CrudEntity_2[Apple]

case class Bird(name: String, hobby: String) extends CrudEntity_2[Bird]

Okay, better. But we still haven’t locked this down. Our types don’t yet express exactly what we want. Do you see the problem?

The problem is that someone can extend CrudEntity_2 in a way we didn’t intend them to:

1
case class Orange(name: String, bankAccount: Double) extends CrudEntity_2[FloobyDust]

Whoa! In the code above, CrudEntity_2[E] does not restrict the type of E, so they can use anything they want, without complaint from the compiler — FloobyDust, Potato, BurritoAstronaut, you name it.

This is no bueno. Instead, we’d like them to get a big, fat compiler error if they try extending anything other than CrudEntity_2[Orange]. How do we ensure that E matches the class we’re defining?

Let’s try defining CrudEntity again. This time, we’ll use type bounds:

1
2
3
4
5
6
7
8
9
10
trait CrudEntity_3[E <: CrudEntity_3[E]] {
  def create(entityData: String): E
  def read(id: String): Option[E]
  def update(f: E => E): E
  def delete(id: String): Unit
}

case class Apple(name: String, age: Int) extends CrudEntity_3[Apple]

case class Bird(name: String, hobby: String) extends CrudEntity_3[Bird]

Better. Now we’ve constrained E to be a subtype of CrudEntity. No more FloobyDust. But there’s one last problem, and you can probably guess what it is. We haven’t yet ensured that E matches our class type, only that it subclasses CrudEntity. CrudEntity is still open for abuse:

1
case class Orange(name: String, age: Int) extends CrudEntity_3[Apple]

Yuck! To take care of this, we need a way to ensure that e.g. Orange extends CrudEntity_3[Orange]. For this assurance, we’ll use a self type.

Here is our final definition of CrudEntity, which uses a self type:

1
2
3
4
5
6
trait CrudEntity[E <: CrudEntity[E]] { self: E =>
  def create(entityData: String): E
  def read(id: String): Option[E]
  def update(f: E => E): E
  def delete(id: String): Unit
}

self: E => ensures that any concrete class that extends CrudEntity must be of type E and that code like

1
case class Orange(name: String, age: Int) extends CrudEntity[Apple]

will get rejected by the compiler because Orange is not of type Apple.

Now we can rest, confident that our definition of CrudEntity ensures that any subtype of CrudEntity[E] must in fact be an E. This definition gives us the semantics we desire, and enlists the compiler to reject all code in violation.

Speed Up Your Rails Specs by 10x

One of the primary reasons people end up being lax in letting specifications drive the development of their Rails applications is the time it takes to get feedback from running the suite of specifications. A number of tools have been built to help alleviate this pain like Spork, Zeus, and Spring. In fact Rails 4.1 will now come with Spring standard. Unfortunately, these tools are just crutches that tackle the symptoms of the problem rather than the problem itself. The actual problem is writing tightly coupled code that expects to have the full Rails framework always present, which is slow to start up.

Developing Decoupled Code

The solution is to write code that is isolated and decouple your components from as much of the system as possible. In other words, write SOLID Rails code. As a specific example, one might typically directly use a model class to create an instance. Instead we can use dependency injection to remove hard coded references to classes. We just need to make sure we safely reference the defaults using either block notation or a lazy evaluating ||=. Below we have a service that needs to create Widgets which happen to be ActiveRecord models. Instead of directly referencing the Widget class, we use lazy evalutation in our chosen injection method. This allows us to decouple our code and not need ActiveRecord loaded.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# A tightly coupled class. We don't want this.
class MyService
  def create_widget(params)
    Widget.create(params)
  end
end

# We can inject in the initializer
class MyService
  attr_reader :widget_factory

  def initialize(dependencies={})
    @widget_factory = dependencies.fetch(:widget_factory) { Widget }
  end

  def create_widget(params)
    widget_factory.create(params)
  end
end

# Or we can explictly inject via a setter with a lazy reader
class MyService
  attr_writer :widget_factory

  def widget_factory
    @widget_factory ||= Widget
  end

  def create_widget(params)
    widget_factory.create(params)
  end
end

# A specification injecting the dependency using the second method
describe MyService do
  subject(:service) { MyService.new }
  let(:widget_factory) { double 'widget_factory', create: nil }
  before { service.widget_factory = widget_factory }

  it 'creates a widget with the factory' do
    service.create_widget({name: 'sprocket'})
    expect(widget_factory).to have_received(:create).with({name: 'sprocket'})
  end
end

A Base Rails-free Configuration

When writing your applications in this way you can then start to restructure how you setup your specifications and minimize the required environment to run both your specification and your code fulfilling the specification. The typical spec_helper.rb will have a line like this:

1
require File.expand_path("../../config/environment", __FILE__)

This is what loads your entire Rails application and slows down the running of your tests. To make your specifications faster, you need to use a configuration file that does not contain this line. So let’s start by creating a very light weight base_spec_helper.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
ENV["RAILS_ENV"] ||= 'test'
require 'rubygems'

RAILS_ROOT = File.expand_path('../..', __FILE__)
Dir[File.join(RAILS_ROOT, 'spec/support/**/*.rb')].each {|f| require f}

RSpec.configure do |config|
  config.mock_with :rspec
  config.order = 'random'
  # Your prefered config options go here
end

require 'active_support'
require 'active_support/dependencies'

We are requiring active_support and active_support/dependencies so we can have access to the autoloader Rails uses without actually loading up all of Rails. It is fairly light weight and the convienence outweighs the cost. In each spec helper which requires this base we will add the relevant portions of our app into the ActiveSupport::Dependencies.autoload_paths.

Plain Ruby Object Specifications

Depending on the part of application that you are specifying, you can create spec helpers specific to what you need in any one context. For example, the simplest would be one for specifying any type of pure Ruby class such as a service class. A sample services_spec_helper.rb might be:

1
2
3
require 'base_spec_helper'
Dir[File.join(RAILS_ROOT, "spec/support_services/**/*.rb")].each {|f| require f}
ActiveSupport::Dependencies.autoload_paths << "#{RAILS_ROOT}/app/services"

Decorator Specifications

For your decorators, you might choose to use Draper and your decorators_spec_helper.rb might look like:

1
2
3
4
5
require 'base_spec_helper'
require 'draper'
Draper::ViewContext.test_strategy :fast
Dir[File.join(RAILS_ROOT, "spec/support_decorators/**/*.rb")].each {|f| require f}
ActiveSupport::Dependencies.autoload_paths << "#{RAILS_ROOT}/app/decorators"

Model Specifications

Testing models needs a little bit more. Assuming you are using ActiveRecord you’ll need to include that as well as establish a connection to your database. We won’t include factory_girl or database_cleaner as most of your tests should not be actually creating database objects. In fact, the only place you really need to actually create an object in the database is when testing uniqueness validations. When you do need to create something you can just manually clean it up or use a transaction. So a sample models_spec_helper.rb can look like this:

1
2
3
4
5
6
7
8
9
10
11
12
require 'base_spec_helper'
require 'active_record'
# RSpec has some nice matchers for models so we'll pull them in
require 'rspec/rails/extensions/active_record/base'
Dir[File.join(RAILS_ROOT, "spec/support_models/**/*.rb")].each {|f| require f}

# Manually connect to the database
ActiveRecord::Base.establish_connection(
  YAML.load(File.read(RAILS_ROOT + '/config/database.yml'))['test']
)

ActiveSupport::Dependencies.autoload_paths << "#{RAILS_ROOT}/app/models"

Feature Specifications

Finally, when creating feature specs, we do need our full Rails stack and our feature_spec_helper.rb is going to look very similar to what your current spec_helper.rb looks like.

Summary

I found myself using varitions on the above spec helpers in projects I work on and decided I would write a set of generators to make it easier to bootstrap the project. The gem can be found at https://github.com/Originate/rails_spec_harness

While introducing these changes into existing projects I have found speed increases of 8-12 times. The worst project experienced a 27x increase once these changes and the corresponding changes in coding habits where applied. As an example I made a specification with 4 examples for a plain Ruby class. I then used the time command line utility to measure running rspec with the minimal spec helper as well as the full Rails spec helper and found the following:

Spec Helper Real User Sys RSpec Reported
Full Rails 4.913s 2.521s 1.183s 0.0706s
Minimal 0.492s 0.407s 0.080s 0.0057s

Write SOLID code, isolate your specifications and enjoy a fun and sane development experience.

TDD Is BS**

** BS = Behavior Specification

tl/dr

The misleading terminology around TDD is responsible for a good amount of confusion and hesitation in this area, as well as over- and undertesting. Calling tests specifications makes the concept more intuitively available in several ways.

One does not simply test something before it is built

testing thin air

One cannot test something before it is built. Asking people to write tests before they write code is like asking them to test-drive a new car before it even exists. This isn’t possible. Similarly, a piece of code can only be tested after it is completely written. We can’t test thin air.

Asking people to write tests before they write code is counter-intuitive and misleading. Resistance or discomfort with the idea is only natural.

specifications instead of tests

A better perspective is to treat tests as specifications for the application in the works, i.e. descriptions of what our code should do.

It is completely plausible to reflect upfront about what I’m going to build – and to write down these thoughts – before starting to build anything serious, costly, and/or sizeable. Especially when working within a team.

It is similarly intuitive that starting to think about these specifications – or writing them down – after a product has already been built makes very little sense, provides very little benefit to the development process, and is actually more a hassle than useful at that point. Specifications need to be defined, agreed upon, and considered before we build the actual system.

behavior-driven development

Specifications just define expected behavior. They don’t dictate how in particular to implement things. The idea of “testing”, on the other hand, naturally means checking a particular implementation in all its gory details.

This distinction is important. Ideally, our tests/specs should not be concerned with implementation details and private methods. Otherwise they become brittle, and will break unnecessarily each time we refactor or clean up something. All that we really care about when running our specs is that the code still works, not how it still works. The how should be discussed in code reviews.

user perspective

Developers tend to occasionally forget about the user perspective while being deeply immersed in the code. Specifications give them a natural place to muse over the user-visible behavior of the application, from the perspective of the user of their code. Specifications thereby form the missing link between business requirements and their technical implementation. Features specs (aka “integration tests”) are runnable user stories.

Tests, on the other hand, just assert implementation details from a technical perspective. Even integration tests just check that the different parts and layers of the application have been integrated properly.

conclusion

While there is often no real technical difference between “tests” and “specs”, the term “test-driven development” is misleading, while “specification-driven development” is intuitive and describes what we actually do here better. We should consider using the latter term in favor of the former. Let’s write “specs” instead of “tests”, and “feature specifications” instead of “integration tests” from now on.

Much love, and happy coding! :)

How Does Caching Work in AFNetworking? : AFImageCache & NSUrlCache Explained

If you are an iOS developer using Mattt Thompson’s ‘delightful networking framework’ AFNetworking (and if you aren’t, what are you waiting for?), perhaps you have been been curious or confused about the caching mechanism employed and how you can tweak it to your advantage.

AFNetworking actually takes advantage of 2 separate caching mechanisms:

  • AFImagecache: a memory-only image cache private to AFNetworking, subclassed off of NSCache

  • NSURLCache: NSURLConnection's default URL caching mechanism, used to store NSURLResponse objects : an in-memory cache by default, configurable as an on-disk persistent cache

In order to understand how each caching system works, let’s look at how they are defined:

How AFImageCache Works

AFImageCache is a part of the UIImageView+AFNetworking category. It is a subclass of NSCache, storing UIImage objects with a URL string as its key (obtained from an input NSURLRequest object).

AFImageCache definition:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
@interface AFImageCache : NSCache <AFImageCache>

// singleton instantiation :

+ (id <AFImageCache>)sharedImageCache {
    static AFImageCache *_af_defaultImageCache = nil;
    static dispatch_once_t oncePredicate;
    dispatch_once(&oncePredicate, ^{
        _af_defaultImageCache = [[AFImageCache alloc] init];

// clears out cache on memory warning :

    [[NSNotificationCenter defaultCenter] addObserverForName:UIApplicationDidReceiveMemoryWarningNotification object:nil queue:[NSOperationQueue mainQueue] usingBlock:^(NSNotification * __unused notification) {
        [_af_defaultImageCache removeAllObjects];
    }];
});

// key from [[NSURLRequest URL] absoluteString] :

static inline NSString * AFImageCacheKeyFromURLRequest(NSURLRequest *request) {
    return [[request URL] absoluteString];
}

@implementation AFImageCache

// write to cache if proper policy on NSURLRequest :

- (UIImage *)cachedImageForRequest:(NSURLRequest *)request {
    switch ([request cachePolicy]) {
        case NSURLRequestReloadIgnoringCacheData:
        case NSURLRequestReloadIgnoringLocalAndRemoteCacheData:
            return nil;
        default:
            break;
    }

    return [self objectForKey:AFImageCacheKeyFromURLRequest(request)];
}

// read from cache :

- (void)cacheImage:(UIImage *)image
        forRequest:(NSURLRequest *)request {
    if (image && request) {
        [self setObject:image forKey:AFImageCacheKeyFromURLRequest(request)];
    }
}

AFImageCache is a private implementation of NSCache. There is no customization that you can do outside of editing the implementation in the the UIImageView+AFNetworking category, directly. It stores all accessed UIImage objects into its NSCache. The NSCache controls when the UIImage objects are released. If you wish to observe when images are released, you can implement NSCacheDelegate’s cache:willEvictObject method.

Edit (03.14.14) : Mattt Thompson has gratiously informed me that as of AFNetworking 2.1, AFImageCache is configurable. There is now a public setSharedImageCache method. Here’s the full AFN 2.2.1 UIImageView+AFNetworking specification.

How NSURLCache Works

Since AFNetworking uses NSURLConnection, it takes advantage of its native caching mechanism, NSURLCache. NSURLCache caches NSURLResponse objects returned by server calls via NSURLConnection.

Enabled by Default, but Needs a Hand

An NSURLCache sharedCache is enabled by default and will be used by any NSURLConnection objects fetching URL contents for you.

Unfortunately, it has a tendency to hog memory and does not write to disk in its default configuration. To tame the beast and potentially add some persistance, you can simply declare a shared NSURLCache in your app delegate like so:

1
2
3
4
NSURLCache *sharedCache = [[NSURLCache alloc] initWithMemoryCapacity:2 * 1024 * 1024
                                              diskCapacity:100 * 1024 * 1024
                                              diskPath:nil];
[NSURLCache setSharedURLCache:sharedCache];

Here we declare a shared NSURLCache with 2mb of memory and 100mb of disk space

Setting the Cache Policy on NSURLRequest Objects

NSURLCache will respect the caching policy (NSURLRequestCachePolicy) of each NSURLRequest object. The policies are defined as follows :

  • NSURLRequestUseProtocolCachePolicy: specifies that the caching logic defined in the protocol implementation, if any, is used for a particular URL load request. This is the default policy for URL load requests

  • NSURLRequestReloadIgnoringLocalCacheData: ignore the local cache, reload from source

  • NSURLRequestReloadIgnoringLocalAndRemoteCacheData: ignore local & remote caches, reload from source

  • NSURLRequestReturnCacheDataElseLoad: load from cache, else go to source.

  • NSURLRequestReturnCacheDataDontLoad: offline mode, load cache data regardless of expiration, do not go to source

  • NSURLRequestReloadRevalidatingCacheData: existing cache data may be used provided the origin source confirms its validity, otherwise the URL is loaded from the origin source.

Caching to Disk with NSURLCache

Cache-Control HTTP Header

Either the Cache-Control header or the Expires header MUST be in the HTTP response header from the server in order for the client to cache it (with the existence of the Cache-Control header taking precedence over the Expires header). This is a huge gotcha to watch out for. Cache Control can have parameters defined such as max-age (how long to cache before updating response), public / private access, or no-cache (don’t cache response). Here is a good introduction to HTTP cache headers.

Subclass NSURLCache for Ultimate Control

If you would like to bypass the requirement for a Cache-Control HTTP header and want to define your own rules for writing and reading the NSURLCache given an NSURLResponse object, you can subclass NSURLCache.

Here is an example that uses a CACHE_EXPIRES value to judge how long to hold on to the cached response before going back to the source:

(Thanks to Mattt Thompson for the feedback and code edits!)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
@interface CustomURLCache : NSURLCache

static NSString * const CustomURLCacheExpirationKey = @"CustomURLCacheExpiration";
static NSTimeInterval const CustomURLCacheExpirationInterval = 600;

@implementation CustomURLCache

+ (instancetype)standardURLCache {
    static CustomURLCache *_standardURLCache = nil;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        _standardURLCache = [[CustomURLCache alloc]
                                 initWithMemoryCapacity:(2 * 1024 * 1024)
                                 diskCapacity:(100 * 1024 * 1024)
                                 diskPath:nil];
    }

    return _standardURLCache;
}

#pragma mark - NSURLCache

- (NSCachedURLResponse *)cachedResponseForRequest:(NSURLRequest *)request {
    NSCachedURLResponse *cachedResponse = [super cachedResponseForRequest:request];

    if (cachedResponse) {
        NSDate* cacheDate = cachedResponse.userInfo[CustomURLCacheExpirationKey];
        NSDate* cacheExpirationDate = [cacheDate dateByAddingTimeInterval:CustomURLCacheExpirationInterval];
        if ([cacheExpirationDate compare:[NSDate date]] == NSOrderedAscending) {
            [self removeCachedResponseForRequest:request];
            return nil;
        }
    }
}

    return cachedResponse;
}

- (void)storeCachedResponse:(NSCachedURLResponse *)cachedResponse
                 forRequest:(NSURLRequest *)request
{
    NSMutableDictionary *userInfo = [NSMutableDictionary dictionaryWithDictionary:cachedResponse.userInfo];
    userInfo[CustomURLCacheExpirationKey] = [NSDate date];

    NSCachedURLResponse *modifiedCachedResponse = [[NSCachedURLResponse alloc] initWithResponse:cachedResponse.response data:cachedResponse.data userInfo:userInfo storagePolicy:cachedResponse.storagePolicy];

    [super storeCachedResponse:modifiedCachedResponse forRequest:request];
}

@end

Now that you have your NSURLCache subclass, don’t forget to initialize it in your AppDelegate in order to use it :

1
2
3
4
CustomURLCache *URLCache = [[CustomURLCache alloc] initWithMemoryCapacity:2 * 1024 * 1024
                                                   diskCapacity:100 * 1024 * 1024
                                                                 diskPath:nil];
[NSURLCache setSharedURLCache:URLCache];

Overriding the NSURLResponse before caching

The -connection:willCacheResponse delegate is a place to intercept and edit the NSURLCachedResponse object created by NSURLConnection before it is cached. In order to edit the NSURLCachedResponse, return an edited mutable copy as follows (code from NSHipster blog):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
- (NSCachedURLResponse *)connection:(NSURLConnection *)connection
                  willCacheResponse:(NSCachedURLResponse *)cachedResponse {
    NSMutableDictionary *mutableUserInfo = [[cachedResponse userInfo] mutableCopy];
    NSMutableData *mutableData = [[cachedResponse data] mutableCopy];
    NSURLCacheStoragePolicy storagePolicy = NSURLCacheStorageAllowedInMemoryOnly;

    // ...

    return [[NSCachedURLResponse alloc] initWithResponse:[cachedResponse response]
                                                    data:mutableData
                                                userInfo:mutableUserInfo
                                           storagePolicy:storagePolicy];
}

// If you do not wish to cache the NSURLCachedResponse, just return nil from the delegate function:

- (NSCachedURLResponse *)connection:(NSURLConnection *)connection
                  willCacheResponse:(NSCachedURLResponse *)cachedResponse {
    return nil;
}

Disabling NSURLCache

Don’t want to use the NSURLCache? Not Impressed? That’s okay. To disable the NSURLCache, simply zero out memory and disk space in the shared NSURLCache definition in your appDelegate:

1
2
3
4
NSURLCache *sharedCache = [[NSURLCache alloc] initWithMemoryCapacity:0
                                              diskCapacity:0
                                              diskPath:nil];
[NSURLCache setSharedURLCache:sharedCache];

Summary

I wanted to write this blog post for the benefit of the iOS community, to summarize all of the information I found dealing with caching releated to AFNetworking. We had an internal app loading a lot of images that had some memory issues and performance problems. I was tasked with trying to diagnose the caching behavior of the app. During this exercise, I discovered the information on this post through scouring the web and doing plenty of debugging and logging. It is my hope that this post summarizes my findings and provides an opportunity for others with AFNetworking experience to add additional information. I hope that you have found this helpful.

One Simple Trick to Level Up Your Code

As software developers we can’t stop bad code but we can make it better. There are a number of ways we try to make better code. We try to follow best practices like TDD. We’ve all read a number of books on design patterns, code standards, rules, and guidelines. It is a fact though, that the real world gets in the way. Deadlines are tight. Projects get rushed. Code is written quickly and sloppily. None of these things are going away but there is one simple thing we can do no matter how tight the deadline.

When you write a piece of code, take a brief pause and look at what you just wrote. Ask yourself:

Can I make this code better?

Chances are you’ll see at least one thing that could be changed. A large method that could be broken up. A convoluted logic expression. Or any number of small things. If you take just a moment to change that one thing before moving on, all of your code will be cleaner, easier to maintain, and in general much higher quality.

Scrap Your Cake Pattern Boilerplate: Dependency Injection Using the Reader Monad

There are a number of ways to do dependency injection in Scala without adding a framework. The cake pattern is probably the most popular approach and is used in the Scala compiler itself. Using implicit parameters is less popular and is used in the Scala concurrency libraries.

An approach that doesn’t get as much mention in the blogosphere but gets much love from those who have tried it is dependency injection via the Reader Monad.

To illustrate the differences between the Reader Monad approach and the more popular approaches, let’s look at examples of using each to solve the same dependency problem. These examples will all be injecting a dependency for getting User objects from a repository:

1
2
3
4
trait UserRepository {
  def get(id: Int): User
  def find(username: String): User
}

The Cake Pattern

In the cake pattern we represent the dependency as a component trait. We put the thing that is depended on into the trait along with an abstract method to return an instance.

1
2
3
4
5
6
7
8
9
trait UserRepositoryComponent {

  def userRepository: UserRepository

  trait UserRepository {
    def get(id: Int): User
    def find(username: String): User
  }
}

Then we declare the dependency in abstract classes using self-type declarations.

1
2
3
4
5
6
7
8
9
10
11
trait Users {
  this: UserRepositoryComponent =>

  def getUser(id: Int): User = {
    userRepository.get(id)
  }

  def findUser(username: String): User = {
    userRepository.find(username)
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
trait UserInfo extends Users {
  this: UserRepositoryComponent =>

  def userEmail(id: Int): String = {
    getUser(id).email
  }

  def userInfo(username: String): Map[String, String] = {
    val user = findUser(username)
    val boss = getUser(user.supervisorId)
    Map(
      "fullName" -> s"${user.firstName} ${user.lastName}",
      "email" -> s"${user.email}",
      "boss" -> s"${boss.firstName} ${boss.lastName}"
    )
  }
}

We extend the component trait to provide concrete implementations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
trait UserRepositoryComponentImpl extends UserRepositoryComponent {

  def userRepository = new UserRepositoryImpl

  class UserRepositoryImpl extends UserRepository {

    def get(id: Int) = {
      ...
    }

    def find(username: String) = {
      ...
    }
  }
}

Finally we can create an instance of our class with the dependency by mixing in a concrete implementation.

1
2
3
object UserInfoImpl extends
  UserInfo with
  UserRepositoryComponentImpl

For testing we can create a test instance of our class and mix in a mock implementation.

1
2
3
4
5
6
object TestUserInfo extends
  UserInfo with
  UserRepositoryComponent {

  lazy val userRepository = mock[UserRepository]
}

Implicits

The cake pattern works really well but tends to require a lot of boilerplate. Another approach is to add implicit parameters to the methods that require the dependency.

1
2
3
4
5
6
7
8
9
10
trait Users {

  def getUser(id)(implicit userRepository: UserRepository) = {
    userRepository.get(id)
  }

  def findUser(username)(implicit userRepository: UserRepository) = {
    userRepository.find(username)
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
object UserInfo extends Users {

  def userEmail(id: Int)(implicit userRepository: UserRepository) = {
    getUser(id).email
  }

  def userInfo(username: String)(implicit userRepository: UserRepository) = {
    val user = findUser(username)
    val boss = getUser(user.supervisorId)
    Map(
      "fullName" -> s"${user.firstName} ${user.lastName}",
      "email" -> s"${user.email}",
      "boss" -> s"${boss.firstName} ${boss.lastName}"
    )
  }
}

This is a lot less code and has the advantage of allowing us to substitute a different UserRepository at any point either by defining our own implicit value or by passing it as an explicit argument.

A downside of this approach is that it clutters up the method signatures. Every method that depends on a UserRepository (userEmail and userInfo in this example) has to declare that implicit parameter.

The Reader Monad

In Scala, a unary function (a function with one parameter) is an object of type Function1. For example, we can define a function triple that takes an Int and triples it:

1
2
3
val triple = (i: Int) => i * 3

triple(3)   // => 9

The type of triple is Int => Int, which is just a fancy way of saying Function1[Int, Int].

Function1 lets us create new functions from existing ones using andThen:

1
2
3
val thricePlus2 = triple andThen (i => i + 2)

thricePlus2(3)  // => 11

The andThen method combines two unary functions into a third function that applies the first function, then applies the second function to the result. The parameter type of the second function has to match the result type of the first function, but the result type of the second function can be anything:

1
2
3
val f = thricePlus2 andThen (i => i.toString)

f(3)  // => "11"

The type of f here is Int => String. Using andThen we can change the result type as much as we want but the parameter type is always the same as in the initial function.

The Reader Monad is a monad defined for unary functions, using andThen as the map operation. A Reader, then, is just a Function1. We can wrap the function in a scalaz.Reader to get the map and flatMap methods:

1
2
3
4
5
6
7
8
9
import scalaz.Reader

val triple = Reader((i: Int) => i * 3)

triple(3)   // => 9

val thricePlus2 = triple map (i => i + 2)

thricePlus2(3)  // => 11

The map and flatMap methods let us use for comprehensions to define new Readers:

1
2
3
val f = for (i <- thricePlus2) yield i.toString

f(3)  // => "11"

If the above example looks strange, remember that it’s just a fancy way of writing:

1
2
3
val f = thricePlus2 map (i => i.toString)

f(3)  // => "11"

Dependency Injection with the Reader Monad

To use the Reader Monad for dependency injection we just need to define functions with a UserRepository parameter. We can wrap each of these functions in a scalaz.Reader to get the full monady goodness.

We define a “primitive” Reader for each operation defined in the UserRepository trait:

1
2
3
4
5
6
7
8
9
10
11
trait Users {
  import scalaz.Reader

  def getUser(id: Int) = Reader((userRepository: UserRepository) =>
    userRepository.get(id)
  )

  def findUser(username: String) = Reader((userRepository: UserRepository) =>
    userRepository.find(username)
  )
}

Note that these primitives return Reader[UserRepository, User] and not User. It’s a decorated UserRepository => User that you can use with for comprehensions. It’s a function that will eventually return a User when given a UserRepository. The actual injection of the dependency is deferred.

We can now define all the other operations (as Readers) in terms of the primitive Readers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
object UserInfo extends Users {

  def userEmail(id: Int) = {
    getUser(id) map (_.email)
  }

  def userInfo(username: String) =
    for {
      user <- findUser(username)
      boss <- getUser(user.supervisorId)
    } yield Map(
      "fullName" -> s"${user.firstName} ${user.lastName}",
      "email" -> s"${user.email}",
      "boss" -> s"${boss.firstName} ${boss.lastName}"
    )
}

Because these methods use map and flatMap on the primitives (for comprehensions use map and flatMap), the methods return new higher-level Readers. The userEmail method in this example returns Reader[UserRepository, String]. All of the higher-level methods dealing with users return higher-level Readers built directly or indirectly from the primitives.

Unlike in the implicits example, we don’t have UserRepository anywhere in the signatures of userEmail and userInfo. We don’t have to mention UserRepository anywhere other than our primitives. If we gain additional dependencies we can encapsulate them all in a single Config object and we only have to change the primitives.

For example, say we needed to add a mail service:

1
2
3
4
trait Config {
  def userRepository: UserRepository
  def mailService: MailService
}

We would only have to change our primitives to take Config instead of UserRepository:

1
2
3
4
5
6
7
8
9
10
11
trait Users {
  import scalaz.Reader

  def getUser(id: Int) = Reader((config: Config) =>
    config.userRepository.get(id)
  )

  def findUser(username: String) = Reader((config: Config) =>
    config.userRepository.find(username)
  )
}

Our UserInfo object doesn’t need to change. This may seem like a small win in this example but it’s a huge win in a large application that has many times more higher-level Readers than primitives.

Injecting the dependency

So where does the actual UserRepository get injected? All of these methods return Readers that can get User-related stuff out of a UserRepository. The actual dependency injection keeps getting deferred up to higher layers.

At some point we need to apply one of these Readers to a concrete UserRepository to actually get the stuff. Normally this would be at the outer edges of our application. In the case of a web application, this would be the controller actions.

Assuming we have a concrete implementation of UserRepository as an object called UserRepositoryImpl, we might define a controller like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
object Application extends Application(UserRepositoryImpl)

class Application(userRepository: UserRepository) extends Controller with Users {

  def userEmail(id: Int) = Action {
    Ok(run(UserInfo.userEmail(id)))
  }

  def userInfo(username: String) = Action {
    Ok(run(UserInfo.userInfo(username)))
  }

  private def run[A](reader: Reader[UserRepository, A]): JsValue = {
    Json.toJson(reader(userRepository))
  }
}

The object Application uses the default concrete implementation, and we can instantiate a test version using the class Application with a mock repository for testing. In this example we’ve also defined a convenience method run that injects the UserRepository into a Reader and converts the result to JSON.

But I can’t choose!

There’s no rule that says we have to use just one dependency injection strategy. Why not use the Reader Monad throughout our application’s core, and the cake pattern at the outer edge?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
object Application extends Application with UserRepositoryComponentImpl

trait Application extends Controller with Users {
  this: UserRepositoryComponent =>

  def userEmail(id: Int) = Action {
    Ok(run(UserInfo.userEmail(id)))
  }

  def userInfo(username: String) = Action {
    Ok(run(UserInfo.userInfo(username)))
  }

  private def run[A](reader: Reader[UserRepository, A]): JsValue = {
    Json.toJson(reader(userRepository))
  }
}

This way we get the benefit of the cake pattern but we only have to apply it to our controllers. The Reader Monad lets us push the injection out to the edges of our application where it belongs.

Answers to Common Questions in Cocoa Development

I recently created my first Cocoa app, and it was a real pain. I wanted to add a bunch of common features I see in every app, and hunting down the right answers on stack overflow proved to be very difficult. The following is a list of answers to common problems I ran into.

Note that Apple’s new rules about app sandboxing makes some of these features unavailable, and I think that’s garbage. Also note that these are quick and dirty solutions, and in now way represent best practices. All of this was done in Xcode 5 with a target of 10.7.

Adding an Icon to the System Tray

Winning solution: Creating a Standalone StatusItem Menu

I’m not going to replicate the whole article, so you’ll have to go there to see how to rig up things in Interface Builder, but here are the relevant bits of code to insert:

1
2
3
4
5
6
7
// MyAppDelegate.h
@interface MyAppDelegate : NSObject <NSApplicationDelegate> {
    IBOutlet NSMenu *statusMenu;
    NSStatusItem *statusItem;
    NSImage *statusImage;
    NSImage *statusHighlightedImage;
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// MyAppDelegate.m
- (void) applicationDidFinishLaunching: (NSNotification *) aNotification {
  // Create the NSStatusBar and set its length
  statusItem = [[NSStatusBar systemStatusBar] statusItemWithLength: NSSquareStatusItemLength];

  // Allocates and loads the images into the application which will be used for our NSStatusItem
  NSBundle *appBundle = [NSBundle mainBundle];
  statusImage = [appBundle imageForResource:@"icon.grey"];
  statusHighlightedImage = [appBundle imageForResource:@"icon.highlighted"];

  // Sets the images in our NSStatusItem
  [statusItem setImage:statusImage];
  [statusItem setAlternateImage:statusHighlightedImage];

  // Tells the NSStatusItem what menu to load
  [statusItem setMenu:statusMenu];
  // Sets the tooptip for our item
  [statusItem setToolTip:@"My Custom Menu Item"];
  // Enables highlighting
  [statusItem setHighlightMode:YES];
}

Note that getting the sizing of the images right was obnoxious, and I couldn’t find any solid information about it in Apple branded documentation. I can at least point out that the retina image assets seem to work the way you would expect, so adding a icon.png and an icon@2x.png does the right thing. I had to mess with the sizes a bunch though, and found that a non-retina icon that is 21x21 pixels works. Continuing to mess with things… it seems like the retina version of the icon had to be 41x41 pixels (not exactly double… “it just works!”).

For more info, this is the best resource I found NSStatusItem – What Size Should Your Icon Be?.

Registering a Global Hotkey to Bring Your App Into Focus

From what I read, it seems like the way you do this in Cocoa is to use the Carbon api. Regardless, after enough digging, I finally found this: DDHotKey. It makes things incredibly easy:

1
2
3
4
5
6
7
8
9
10
11
12
13
// MyAppDelegate.m
#import "DDHotKeyCenter.h"

- (void) applicationDidFinishLaunching: (NSNotification *) aNotification {
  // register hotkey
  DDHotKeyCenter *c = [[DDHotKeyCenter alloc] init];
  if ([c registerHotKeyWithKeyCode:34 modifierFlags:(NSCommandKeyMask | NSShiftKeyMask) target:self action:@selector(hotkeyAction:withObject:) object:nil]) {
      NSLog(@"registered");
  }
  else {
      NSLog(@"not registered");
  }
}

This example will register command + shift + i as your hotkey. To ensure that your app pops up to the front, you can do:

1
2
3
4
5
- (void) hotkeyAction: (NSEvent *)hotKeyEvent withObject:(id)anObject {
  NSApplication *myApp = [NSApplication sharedApplication];
  [myApp activateIgnoringOtherApps:YES];
  [self.window orderFrontRegardless];
 }

Make App Hide When Losing Focus

1
2
3
4
// MyAppDelegate.m
- (void) applicationDidResignActive: (NSNotification *) notification {
    [[NSApplication sharedApplication] hide:self];
}

Prevent App From Showing In Dock/Alt+Tab

This one requires you to edit your MyApp-Info.plist file (usually under “Supporting Files” in the project). Add a new key for “Application is agent (UIElement)” and set it to YES.

Start App at Login

Winning solution: Stack Overflow – How do you make your app open at login (with a few changes for Xcode 5)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
// MyAppDelegate.m
+ (BOOL) willStartAtLogin: (NSURL *) itemURL {
  Boolean foundIt=false;
  LSSharedFileListRef loginItems = LSSharedFileListCreate(NULL, kLSSharedFileListSessionLoginItems, NULL);
  if (loginItems) {
    UInt32 seed = 0U;
    NSArray *currentLoginItems = (__bridge NSArray *)(LSSharedFileListCopySnapshot(loginItems, &seed));
    for (id itemObject in currentLoginItems) {
      LSSharedFileListItemRef item = (__bridge LSSharedFileListItemRef)itemObject;

      UInt32 resolutionFlags = kLSSharedFileListNoUserInteraction | kLSSharedFileListDoNotMountVolumes;
      CFURLRef URL = NULL;
            OSStatus err = LSSharedFileListItemResolve(item, resolutionFlags, &URL, /*outRef*/ NULL);
      if (err == noErr) {
        foundIt = CFEqual(URL, (__bridge CFTypeRef)(itemURL));
        CFRelease(URL);

        if (foundIt)
          break;
      }
    }
    CFRelease(loginItems);
  }
  return (BOOL)foundIt;
}

+ (void) setStartAtLogin: (NSURL *) itemURL enabled: (BOOL) enabled {
  OSStatus status;
  LSSharedFileListItemRef existingItem = NULL;

  LSSharedFileListRef loginItems = LSSharedFileListCreate(NULL, kLSSharedFileListSessionLoginItems, NULL);
  if (loginItems) {
    UInt32 seed = 0U;
    NSArray *currentLoginItems = (__bridge NSArray *)(LSSharedFileListCopySnapshot(loginItems, &seed));
    for (id itemObject in currentLoginItems) {
      LSSharedFileListItemRef item = (__bridge LSSharedFileListItemRef)itemObject;

      UInt32 resolutionFlags = kLSSharedFileListNoUserInteraction | kLSSharedFileListDoNotMountVolumes;
      CFURLRef URL = NULL;
            OSStatus err = LSSharedFileListItemResolve(item, resolutionFlags, &URL, /*outRef*/ NULL);
      if (err == noErr) {
        Boolean foundIt = CFEqual(URL, (__bridge CFTypeRef)(itemURL));
        CFRelease(URL);

        if (foundIt) {
          existingItem = item;
          break;
        }
      }
    }

    if (enabled && (existingItem == NULL)) {
      LSSharedFileListInsertItemURL(loginItems, kLSSharedFileListItemBeforeFirst,
        NULL, NULL, (__bridge CFURLRef)itemURL, NULL, NULL);

    } else if (!enabled && (existingItem != NULL))
    LSSharedFileListItemRemove(loginItems, existingItem);

    CFRelease(loginItems);
  }
}

If you’d like a checkbox that is bound to your app’s “start at login” state, you can create a property:

1
2
// MyAppDelegate.h
@property BOOL startAtLogin;

then use Interface Builder to bind a checkbox to this property, and implement the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// MyAppDelegate.m
- (NSURL *) appURL {
  return [NSURL fileURLWithPath:[[NSBundle mainBundle] bundlePath]];
}

- (BOOL) startAtLogin {
  return [ISPAppDelegate willStartAtLogin:[self appURL]];
}

- (void) setStartAtLogin: (BOOL) enabled {
  [self willChangeValueForKey:@"startAtLogin"];
  [ISPAppDelegate setStartAtLogin:[self appURL] enabled:enabled];
  [self didChangeValueForKey:@"startAtLogin"];
}

Conclusion

Now you have the basics of an app that can do whatever you’d like without having to figure out the boilerplate. If there’s some interest, I’d be willing to roll all of this up into a public github repo that people can check out and use as a starting place for utility apps. For those who are curious, the reason I had to figure all of this out was to build Insert Pic.

Using Play Framework’s in Memory Database to Improve Test Speed

I’ve been working with Play Framework 2.x in scala for the last few months and overall I’m pretty happy. One big issue I ran into recently was that if you want to use a real database, it becomes hard to convince Play Framework to run your tests against H2 (for the order of magnitude speed increase).

The Secret Sauce

The trick that finally made it work was found by digging through the Play source. You can tell your app to use a specific data source while running in test mode. The easiest way to set this up is to give yourself a trait that you will extend classes and objects with:

1
2
3
4
5
6
7
8
9
trait DatabaseClient {
  private def dbName = if (Mode.Test == current.mode) "test" else "default"

  def db = Database.forDataSource(DB.getDataSource(dbName))

  def withSession[A](f: => A): A = db.withSession(f)

  def withTransaction[A](f: => A): A = db.withTransaction(f)
}

This allows you to run a different DB config while in test mode.

Getting The Test Environment To Work

While the Play documentation tells you that you can just add multiple datasources to your application.conf file, what they fail to mention is that your app will try to use all specified DBs in all environments. Instead of placing the config for your test database in your config file, I inject it directly into the FakeApplication that I instantiate for my tests:

1
2
3
4
5
6
7
trait TestHelpers { self: Specification =>

  def fakeApplication = FakeApplication(additionalConfiguration = inMemoryDatabase("test") + (
    "applyEvolutions.test" -> "true"
  ))

}

It’s useful to have this in its own trait because you will likely want to use fakeApplication in a few instances (we have to reference it in our model tests as well as our controller tests). Now that you have a configuration for your “test” database, and due to the change above, when your tests run, they will use H2.

Note that if you are using Typesafe Slick, you’ll want to inject a "slick.test" -> "models.*" in there.

Providing your FakeApplication for your tests

In order to get my tests to run in the context of an H2 database, I use a trait that can wrap my specs for me:

1
2
3
4
5
6
7
8
9
trait TestSupport { self: Specification =>

  trait WithApplication extends Around {
    def around[T : AsResult](t: => T) = running(fakeApplication) {
      AsResult(t)
    }
  }

}

then in my spec:

1
2
3
4
5
6
7
8
9
10
11
class ModelSpec extends Specification with TestSupport {

  "Model" should {

    "be creatable" in new WithApplication {
      Model.create() must beSome[Model]
    }

  }

}

The nice thing about this setup is that it’s flexible to meet the demands of most apps. Adding changes to the config is always a black box, it’s hard to reason about how the framework will react to those changes. Moving this to actual code makes it more obvious what is actually happening, and lets you go even further, if, for example, you wanted to have your model tests run against MySql, but controller tests use H2 (not sure why you would want this…).

Caveat

As far as I can tell, Play will try to maintain a connection to your database regardless of whether or not you deliberately try to connect to it (if it’s in the config, it tries to connect). This means the tests ran fine on my machine, but were failing on our CI (circleci.com). The fix is to make sure that your connection to the database is still valid, so all I had to do was update the user/pass/dbname in my CI config and I was all set. It feels a bit sloppy that Play connects to a database you aren’t even using, so if anyone has any suggestions on how to prevent this behavior, let me know and I’ll update this post.