Patterns to Isolate Code in Continuous Delivery
Continuous delivery is one of those goals that is very easy to understand but very difficult to implement. The gist of continuous delivery is that the development team delivers a feature as soon as it is ready. This reduces overhead and the lag time between the process of developing a feature and seeing it to the next stage. Overall, this is a good idea because it allows for features to be developed and deployed at a more rapid pace as well as enabling users to interact with the new functionality in as little time as possible.
This is a great and simple idea, in theory. In practice, there are some inevitable complications with that process. One of those difficulties is how to develop new features while keeping your application deployable at any given time.
Imagine this situation: you're doing a continuous delivery pipeline for your product. You've told business that they can deploy at any time that they wish; they have informed the team that release is to go out tomorrow night. You've just finished a feature and pushed; the pipeline went green. Now, you have 4 hours left to work on new features. However, the next story to be played is rather large and is going to take around two days to complete. Obviously, you cannot finish the feature before the next release. What is the best practice in continuous delivery going forward?
This is a question with a very complex answer. The one sentence answer is: you isolate your new code from the users in the simplest way possible. "The simplest way possible", though, is up to the developers of the story and they must use their judgement to implement one of the follow set of practices.
Don't push until finished
In this practice, you develop your feature locally until your are finished before you push any code. In a DVCS like git, this might mean that you are committing multiple times locally for a multi-day story but do not push until you are completely done. This is essentially a short-lived feature branch that is not visible to anyone else.
- You should use this practice for very small stories. If a story is going to take 1-2 hours to develop, go ahead and use this one. It is, by far, the easiest practice of any and it will work for smaller stories.
- While this practice says that you should not push, it does not mean that you should not pull. Continually pull and merge your changes with what is on the master branch.
- The downsides of this are the same of those of feature branches: the longer the local changes are not shared, the more divergent your code becomes to others. It becomes more and more a pain to refactor without significant merge conflicts and to understand the impact of any changes you make to code the longer the changes are not shared.
- For anything of medium-to-large size, I would recommend against this practice.
Change the view last
In this practice, you are modifying the underlying system to a degree that is not permeated to the client. Then, when the underlying system is ready, you use another strategy (normally #1 Don't push until finished) to do the "view" part. This can be useful for refactoring-heavy stories.
- For MVC web applications, a simple example is if you were adding a new endpoint that modified the controller and model as well as adding a new view. You can do the controller and model part first and then circle back to the view.
- This can apply to non-MVC contexts, too: imagine a service that was exposing a RESTful JSON API. The JSON is the "view" in this case and if you can keep that the same you can monkey about behind it at will without effecting your consumers.
- Personally, I like to work outside-in. I like to write ATDD tests that guide me through the entire story. Unfortunately, this practice is at odds with the working style. I would probably not employ this practice very often.
Short lived feature branches
Feature branches are always a controversial topic. Please see feature branches and continuous delivery are incompatible for more information on my take of that topic. However, for code isolation, short-lived ad-hoc feature branches can be useful. There are a slew of issues that make this an almost-last-resort type of thing for me.
- Short-lived ad-hoc feature branches have a life cycle the same as a story in development and are one off sorts of decisions that a developer must make.
- The can isolate code very neatly; the code is never seen on the master branch.
- The isolation is at cost: we are wanting to isolate features from users not code from developers. Developers need the code as soon as it is written so that they have context for a variety of circumstances. Having a pile of code being "dropped" into a code base at irregular intervals can wreak havoc on a developers ability to make changes.
- I would only use this strategy as a last resort for a really big, breaking change that we are consciously internalizing the cost of the Big Merge.
Short-lived ad-hoc feature toggles
Feature toggles are switches that exist inside of the production system which hide features from consumers. These switches can be implemented in a number of ways but all have the same effect: code is able to be continuously pushed without affected consumers.
- These are a bit more expensive to implement than other strategies. They also introduce technical debt and complexity into the code base.
- Feature toggles lifecycle should be the same as the story is "in development." They should be created at the start of the story and removed when the developer thinks the story is done.
- There is a risk that business will use the feature toggles, which is a developer tool, as a system feature. Push back on this very hard. If business wants the ability to turn features on and off, it should create requirements and go through the normal product analysis. Feature toggles can become crazily complex otherwise.
- There are multiple ways to implement feature toggles. I have heard or seen of the following methods: using a flat text file to set values true/false that gets read on launch, using Rails environments to set boolean configuration options, using URL parameters to turn a feature on or off, using server-level redirects to prevent new features from being accessed. Each has its strengths and weaknesses. Remember: just do what is the simplest way to isolate your code.
Build a new route
When developing an entirely new feature, it can sometimes be easier to build a new route (or endpoint) that is inaccessible in production. This is a relatively narrow use case: only when the new feature is entirely on its own is this useful. However, it is an important pattern to know because when it can be used, it is often the simplest method.
- Build the new route as you would any other feature. Leave off the linking until the end (and use strategy #1). If its a sensitive feature, consider turning off the route for production.
- You can use any development method for this (including outside-in TDD--yay!).
- You can also use this method if you're overhauling another view entirely. Build the new view on a new route that will be swapped in at the end.
- Use this whenever it makes sense.
Use existing feature turn on/turn off system
Some places have existing systems in place to do feature level system turn on and turn off in production. This is a product level system. It is, however, very close to what we want and surely we can reuse the system? Not from experience reports from friends on projects that are doing just this. It quickly becomes confusing what the job of the system is and whether a particular feature is a "development toggle" or a "product toggle".
- Do not do this.
Build a cool system like Facebook's
Facebook has a system that helps them develop features and roll them out independent of the codebase. This is a really cool idea (that I unfortunately have no more insight than others about). Maybe your team should be doing this?
- Maybe. Consider the engineering effort to build and maintain the system. Is it worth it?
What is the best practice to keep a system production-ready while still being able to continuous deliver? The answer is: it depends. You should probably be using a mixture of the seven patterns described above. There could even be other simple ways to isolate your code. The goal is to isolate your code from the users as simply as possible.