Have you ever written software that was perfect after release? Did you ship it to production and never push an update again? I haven’t. And APIs are no exception.
After you publish your API, you’ll probably receive a flood of requests to better support different use cases. Customers will ask for more fields in your responses, extra query parameters to filter results, methods that provide all the data they need in one request, etc. You’ll end up with two camps of users: those who don’t want your API to change and those who do.
Catch 22? Not necessarily. What if you could add as many new features to your API as you want without changing anything for existing users? It is possible. You can have your cake and eat it, too!
There are some great examples of APIs with powerful versioning mechanisms and one I find particularly impressive is Stripe’s. In fact, even though we planned to do it anyway, a Stripe engineering blog post inspired both our API versioning project and this post. I definitely recommend it for the great job it does explaining both the reasoning behind including versioning in your API and the value it brings.
I often see questions in places like StackOverflow along the lines of, “How do I build an API versioning system like Stripe’s?” Well, I did just that. I built an API versioning system like Stripe’s, in a Go backend, and then saw a post on StackOverflow where someone specifically asked how to do that.
I think it’s probably a little harder in Go, or other similar languages, because the examples given by Stripe use Ruby and rely on language features unavailable in Go. So this leaves the question: how to actually implement versioning like theirs using a language other than Ruby, and more specifically, using Go?
I have a Java background and I’m relatively new to Go, so I’ll admit right up front that I may have come at some of this a bit sideways. But regardless of how I got there, it works. And my Go-loving co-workers approve of the result.
When I started, I had to ask myself a few questions:
What will require a new version and how frequently will my API need to change?
- Will I need a new version for all changes or only for backwards-incompatible changes?
- Multiple releases per day?
What pieces do I want versioned?
- Headers?
- Response body?
- Request parameters?
- Request body?
In my mind, only the response body and possibly the headers need to be versioned. Your server will be wonderful and gracefully handle different request versions that use the same method. And of course, you have automated tests that regularly validate how each version is handled, right? So, that just leaves response data to be versioned. This is where your users may have code that is closely tied to the exact data in your response. Their code may not be able to handle a new or deleted data field or any number of other possible changes you might make.
A good versioning system should be able to shield users from pretty much any change. At Dyspatch, we decided that we wanted to version every piece of the response data. Any changes to the response body would be done in a new version with no impact on previous versions. We have a lot of automated testing in place for validation, which frees us up to change things regularly, to make incremental improvements and adapt to customer needs.
So how do we build an API with ‘versioning like Stripe’s’?
Well, there are a lot of ways it could be done but some depend heavily on cool language features that aren’t always available. What follows is how I did it in Go but it would probably work in a lot of other languages as well. If you work on a web API that serializes data into a format like JSON, you should be able to replicate what I did.
The basic idea is simple: encode everything you want versioned as a JSON object and then mutate that object using a series of migration functions until it matches the version your user wants.
Why use this approach? In Go, I don’t really have objects that can be mutated like they can be in Ruby. I have structs and structs don’t like having data types change or fields added/removed on the fly. I could use a map, but again, I run into issues if I have a version of the same field that needs to be a different data type. I could use empty interfaces and pointers but that’s a pretty low-level solution that’s just begging for bugs. So instead I use the JSON library I already use in the API to write JSON data. I can add and remove fields, as well as change field types. Perfect for mutating an object incrementally, which is exactly what I need to mutate my data from one version to the next!
How do you get the version from the user?
The first thing we need to do is find what version of the API is being called (we’ll call this the “target” version). There are a ton of ways this can be done, but probably one of the most idiomatic HTTP approaches is to use the `Accept` header. Be warned, though, it can be a bit tricky to parse the Accept header in a way that actually follows the RFC standard. I really banged my head against the wall trying to write a regex that could accommodate all the content-negotiation syntax that the header can include. If you don’t want to do this or don’t have a library available that does this for you, you should probably just use a custom header like `X-MyCoolApp-Version`. There are other approaches too, like using a request parameter or a path variable to send the version. Generally, it’s best to use a header unless you have a really good reason not to. Also consider what your response will be if no version is passed. You may want to return an error or default to the latest version.
Whatever approach you end up choosing, you will want some middleware that intercepts every API request and parses out the version value. You’ll want to validate this version and return an error in the event of an unrecognized version.
In Go, this means I have a version detection function that wraps an HTTP handler function. This is the version detection middleware. It looks for the version header, grabs the value and puts it in the request context. Now the downstream handlers are able to look into this context to check what version is requested later on.
return func(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
// Check requested API version
version, err := parseAcceptVersion(r.Header.Get(AcceptHeaderName), Versions)
// Ensure valid version requested
if err != nil {
badParam(r.Context(), "version", "No valid version found in 'Accept' header.", w)
return
}
// Set the detected version in our request context so we can use that later
r = r.WithContext(context.WithValue(r.Context(), config.APIVersion, version))
// invoke the next handler in the chain now that we have a validated version in the request context
h(w, r, ps)
}
Once you detect the version, you can already make decisions based on that version. There may be some useful things to do here, but generally you won’t need it until you output your response. This is easy if you visualize your request as a pipeline with distinct stages: parsing, processing, and marshalling. The parsing stage is where we get the version. The processing is a stage that doesn’t matter to the versioning system. Think of it in abstract terms, a black box that takes in request parameters and spits out internal data. This is really key. Thinking of processing as an abstract black box makes the business logic simpler, something that can be changed without having to understand versioning at the same time. Trying to deal with both simultaneously can be a sure route into spaghetti-code hell.
The real versioning magic happens in the final, marshalling stage. This is where we render our internal data to JSON. We start by rendering the current version of the API. This is done in GO using structs called data transfer objects (DTOs) which have the same fields and types as the current (AKA newest) version of the API. For Go, we use an open source JSON library* to construct a JSON builder instance seeded with the data from the DTO. If this was the version that your user wanted, you’d be done and could just write the contents of the builder to the response body.
*Full disclosure: I maintain this JSON library as a side project
Great, now I have JSON of one version. What about the older versions?
Now we have a mutable JSON object that represents the response of the current version. But how do we get an older version? The short answer is this: for every API method, we will write functions that transform one version of the response into the version that came before it.
Chaining incremental migration functions together like this is useful because we can build a chain of responsibility that ends at the most recent version. Each time you build a new version, you simply write a function that takes your new version back to the previous one, which already has a function that links it back to the one that came before it, and so on through all previous versions. Having a chain of responsibility like this reduces the cognitive load on developers. They can focus on implementing one migration without having to worry about the rest. It doesn’t get more complicated as more versions are added.
So to actually realize this, I built a migration function that takes:
- The current JSON builder instance (we mutate this during each migration)
- The target version (we need to know what version we reach)
- A map of versions and migration functions
- The original (internal) data (we need to have the original data at hand in case we need to add in a new field or recover some other original data)
func ExampleAPIMethod(ctx context.Context, data *InteralData) ([]byte, error) {
// convert DTO into a mutable JSON builder
j := jsonbuilder.FromMarshaller(dto, util.SerializeJSON)
// Build the version changes that apply to this method
changes := map[string]util.Downgrade{
// migration closure
util.Versions["2018.02.09"]: func () {
// remove a field that didn't exist in this version from some paginated results
d := j.Enter("data")
for i := 0; i < size; i++ {
d.Enter(i).Delete("newField")
}
return nil // no error
},
// migration closure
util.Versions["2018.02.10"]: func () {
// migration closure
},// etc...
}
// Run the migrations
target := util.GetVersion(ctx)
err := DowngradeDTO(target, changes)
if err != nil {
return nil, errors.Wrap(err, "Unable to migrate example")
}
// Voila! Out comes JSON for your response at the version requested!
return j.MarshalBytes(), nil // set this as your response body
}
Migrator code
// Downgrade functions will be closures over the data being migrated, so no args needed.
// Return an error in the event that the downgrade was unsuccessful. Do not continue.
type Downgrade func() error
// DowngradeDTO downgrades some DTO JSON to an earlier version
func DowngradeDTO(target string, versionChanges map[string]Downgrade) error {
versions := make([]string, 0, len(versionChanges))
migrationExists := false
for k := range versionChanges {
versions = append(versions, k)
if target <= k {
migrationExists = true
}
}
// Ensure the version we are migrating to exists
if !migrationExists {
return errors.New(“Version does not exist”)
}
// Need to sort the versions by date so that we apply the migrations in the right order
sort.Sort(sort.Reverse(sort.StringSlice(versions))) // lexicographical sorting happens to sort by date
// We iterate over all version changes until we hit the version we are targeting.
// Each time we hit a version that isn't our target, we apply all the changes and then repeat.
// Repeat until we hit our target version
for _, version := range versions {
// When we find the target, we are done
if version < target {
return nil
}
// Apply the downgrade (it will mutate the data we are processing)
change := versionChanges[version]
err := change()
if err != nil {
return errors.Wrap(err, "Encountered error while downgrading a DTO object.")
}
}
return nil
}
This is where your choice of a versioning strings will really matter. You should choose a version scheme that allows you to sort your strings in an order that matches their release date. Why not use a release date? Something like the string “2018.04.03”, for example, would work well. But if you choose not to use release date, make very sure the string format you choose sorts in the same order the versions were released. You may also want to consider accepting version qualifiers like “alpha” and “beta” at the end of your strings. Or you could write a custom sorting algorithm.
The migration function starts by sorting the mutate functions by the versions they are keyed to. This puts our migration functions in a time order so we can migrate backward through time. The migrator loops through each version, checking it against the target version until we reach our target. Once we encounter a version older than the one we want, we know no more migrations are required.
One really nice side effect of using this approach is that it covers “gap” versions. A gap version is a valid version of the API that had no change for a particular API method, leaving a gap in the list of migration functions. The algorithm illustrated above will migrate a chain of functions ordered by date, handling gaps by simply skipping them or stopping before them.
Available migrations for method /example "2018.03", // gap "2018.01", "2017.12", "2017.11", // gap “2017.09”, Request version “2018.02” of /example “2018.03” ← downgrade “2017.10” ← skipped (gap) “2017.09” ← downgrade DONE
In this example, version “2018.02” is requested, but the requested method “/example” didn’t have any changes, so that version doesn’t appear in the list of available migrations. The migrator simply skips over the missing version to the closest older version. In the example above, this meant version “2018.01” was the final target. This is a little counter-intuitive until you realize that the newer version “2018.02” didn’t have any changes, so it should have the same response as the “2018.01” version. If we didn’t continue to the next older version, this would leave the response at the “2018.03” version, which is different.
Finally, I output the result data in the response. If you only versioned the body then you can just serialize the JSON builder’s contents to bytes for your response body. If you also included headers in your versioning scheme, then you probably want to encode the entire response like this:
{
"headers": {
"X-MyHeader-1": "some value",
"X-MyHeader-2": "another value"
},
"body": {
"dtoField1": "value",
"dtoField2": "value"
}
}
You could then render just the “body” portion of the JSON builder to bytes in your response body and loop over each header in the “headers” portion of the builder, setting the corresponding headers in your response to those values. The migration functions can manipulate these headers, adding, removing and changing their values just like they can for the response body contents.
The magic sauce for building our API versioning system in Go is the JSON builder library. It allows me to easily mutate data, including the types and structure of the data, to incrementally take the current version back to the one that came before, and that version to the one before it, until the user’s version is reached. This way, we can update our API as often as we want, with zero impact on users of older versions. A win-win for both us and our customers.
