By Martin Nally, Programmer and API Designer, Apigee

There's a lot of advice on the web about API versioning, much of it contradictory and inconclusive: One expert says to put version identifiers in HTTP headers, another expert insists on version identifiers in URL paths, and a third says that versioning of APIs is not necessary at all. (For some examples of those divergent views, take a look at this blog post and its bibliography and this interview with the author of the original REST dissertation).

With all this information, who’s an API developer to believe?

Here on the Apigee team at Google Cloud, we design, use and see a lot of APIs, and have developed some strong opinions about the right way to version them. My aim is to bring some clarity and offer unequivocal practical advice.

A significant part of the confusion around API versioning stems from the fact that the word “versioning” is used to describe at least two fundamentally different techniques, each with different uses and consequences. Understanding the two different meanings of API versioning is a prerequisite to deciding how and when to use each.

Type 1 versioning: format versioning

Consider the example of a bank that maintains accounts for customers. These accounts have been around for a long time and customers can get information about these accounts through multiple channels (the postal service, the telephone or the bank website, for example).

In addition, the bank has a web API that allows access to accounts programmatically. At some point, the bank sees a way to improve the web API; to attract more developers to it they decide to introduce a new version. I don't think there's an API designer who hasn't one day wished they had organized the information in some request or response differently.

The important point in this example is that version 1 and version 2 of the API both allow access to the same bank accounts. The API change introduces no new entities; versions 1 and 2 simply provide two different "formats" [my word¹] for manipulating the same bank accounts.

Further, any change made using the version 2 API changes the underlying account entity in ways that are visible to clients of the version 1 API. In other words, each new API version defines a new format for viewing a common set of entities. It’s in this sense that I use the phrase "format versioning" in the rest of this post.

Format versioning seems straightforward, but it presents a technical challenge: how do you ensure that any changes made to an entity using one version of the API are seen correctly through all other versions of the API, and how do you ensure that changes made by older clients don't undo or corrupt changes made by newer clients?

In practice, this means that format versioning often can’t accommodate major functional changes. Another problem is that if a lot of format versions are introduced, the server code can become bloated and complex, raising costs for the API publisher.

Type 2 versioning: entity versioning

Given that format versioning can be complex to implement, and that it cannot be used for substantial changes that would cause older API clients to stop working, it’s a good thing that there’s another way to version APIs.

Consider how car manufacturers do versioning. Car manufacturers typically introduce a new generation of popular models every 4 or 5 years. Car model generations are sometimes also called marks; they're a form of versioning. When you buy a specific car of a particular model, it’s a generation 2 car or a generation 3 car, but not both. Car manufacturers will recall cars to fix faults, but they can’t make your generation 2 car look and behave like a generation 3 car on demand.

This model can work well for software too. Extending the bank example, imagine that the bank wants to introduce checking accounts based on blockchain technology, which requires the underlying data for the account to be organized quite differently. If the API that was previously exposed for accounts made assumptions that are simply not compatible with the new technology, it's not going to be possible to read and manipulate the blockchain accounts using the old API. The bank’s solution is the same as the car company’s: introduce "version 2" checking accounts. Each account is either a conventional account or a blockchain account, but not both at the same time. Each version has its own API that are the same where possible but different where necessary.

While "entity versioning" is attractive for its flexibility and simplicity, it also is not free; you still have to maintain the old versions for as long as people use them.

You could think of entity versioning as a limited form of software product lines (SPLs), where the product line evolves along a single time dimension. There's quite a bit of material on the general topic of SPLs on the internet.

New entity version or new entity?

From a technical perspective, introducing a new "entity version" is very similar to introducing a brand new entity type. The argument by Roy Fielding, the creator of REST, that you don't need to version APIs at all seems to be based on this idea.

Fielding makes an important point, but there are good reasons not to present a new API for similar concepts as something completely unrelated. We have all seen advertisements that say something like "Acme Motors introduces the all-new model Y," where model Y is an existing car model that Acme Motors has been selling for years. Not every statement made in car advertisements is accurate, but when car companies say "all-new," they're usually being truthful in the sense that there isn't a single part on the new model that is the same as on the old one.

So if the model is really "all-new," why is it still a model Y, and not some completely different model? The answer is that the manufacturer is choosing to emphasize brand continuity; although it's "all-new," the new model serves the same market segment in a similar way, and the manufacturer wants to leverage its existing investment in the model Y brand.

The bank has a similar choice to make when it introduces accounts based on blockchain technology. Although the new accounts are based on a different technology, they provide all of the basic capabilities of traditional accounts to the same customers. They may offer additional capabilities, or they may just hold the promise of greater security, or shorter delays in transferring money. The bank likely chooses to present them as a new version of their traditional bank accounts, rather than a completely different financial product.

One of the reasons why this form of versioning works well for car companies is that customers replace their cars every few years, so there's a natural migration from older to newer inventory over time. Banking customers may keep bank accounts for a long time, so the bank may want to offer to automatically migrate their customers to move them onto the new version more quickly. If your company's products are digital technology products, the natural inventory turnover may be even more rapid, which makes this versioning model even more attractive.

When is versioning required?

It turns out that in practice, many APIs don’t need versioning of either kind. We can say with certainty that some of our own APIs (Apigee’s management API is one example) have never had the need for a subsequent version or only in limited portions of the API.

One reason why many APIs never need versioning is that you can make many small enhancements to APIs in a backwards-compatible way, usually by adding new properties or new entities that older clients can safely ignore. Your first thought should always be to try to find a backwards-compatible way of introducing an API change without versioning; versioning of either sort should only be attempted if that fails. Fortunately, there are things you can do up front when designing your API to maximize the number of changes that can be made in a backwards-compatible way. One example is using PATCH instead of PUT for updates. You can also add a couple of random and meaningless properties to every entity representation; this will test the clients' ability to ignore future properties they haven't seen before.

One of the simplest changes that might motivate a version change is to change the name of a particular property or entity type. We all know that naming is hard, and finding the right names for concepts is difficult even when the concepts are clear. Names that were carefully chosen during development often become obsolete as concepts evolve and ideas become clearer. A format version can be an opportunity to clean up nomenclature without changing the fundamental meaning of entities and their properties. Format versioning lends itself to this sort of change—the same changes can be made by old and new clients using different names for the same things.

Another example that can motivate a version change is restructuring the representation of a resource. Here's an example of two ways of structuring the same information in JSON:

{"kind": "Book",
 "name": "The Adventures of Tom Sawyer",
 "characters": {
   "Tom Sawyer": {"address": "St Petersburg, Mo"},
   "Huckleberry Finn": {"father": "Pap Finn"}
  }
}

{"kind": "Book",
 "name": "The Adventures of Tom Sawyer",
 "characters": [
   {"name": "Tom Sawyer", "address": "St Petersburg, Mo"},
   {"name": "Huckleberry Finn", "father": "Pap Finn"}
  ]
}

One of these formats encodes the list of characters as a JSON object keyed by the characters' name, and the other encodes it as a JSON array. Neither are right or wrong. The first format is convenient for clients that always access the characters by name, but it requires clients to learn that the name of the character is to be found in the place that a property name is usually found in JSON, rather than as a property value. The second format does not favor one access pattern over another and is more self-describing; if in doubt, I recommend you use this one. This particular representation choice may not seem very important, but as an API designer you're faced with a large number of options, and you may sometimes wish you had chosen differently.

Sadly, there's no practical way to write API clients that are insensitive to name changes and changes in data representation like these. A version format allows you to make changes like this without breaking existing API clients.

Browsers are able to survive HTML webpage changes without versioning, but the techniques that make this work for browsers—e.g., the ability to download and execute client code that is specific to the current format of a particular resource, enormous investment in the technology of the browser itself, industry-level standardization of HTML, and the human user's ability to adapt to changes in the final outcome—are not available or practical for most API clients. An exception is when the API client runs in a web browser and is loaded on demand each time an API resource is accessed. Even then, you have to be willing to manage a tight coordination between the team producing the browser code and the team producing the API—this doesn't happen often, even for browser UI development within a single company.

A very common situation that usually requires an entity version change, rather than just a format version change, is when you split or merge entity hierarchies. In the bank example, imagine that Accounts belong to Customers, and each Account entity has a reference to the Customer it belongs to. Because some customers have many Accounts, the bank wants Accounts to be grouped into Portfolios. Now Accounts need to reference the Portfolio they belong to, not the Customer, and it's the Portfolio that references the Customer. Changes like this are hard to accommodate with format versions, because older clients will try to set a property linking an Account to a Customer and newer clients will try to set a property linking an Account to a Portfolio. You can sometimes find ways to make both sets of clients work in cases like this, but more often you are forced to introduce new entity versions, each of which is updated using only one API format.

The sort of structural changes that force a new entity version usually introduce new concepts and new capabilities that are visible to the user, whereas the changes handled by format version changes are more superficial.

In general, the more clients an API has, and the greater the independence of the clients from the API provider, the more careful the API provider has to be about API compatibility and versioning.

Providers of APIs sometimes make different choices if the consumers of the API are internal to the same company, or limited to a small number of partners. In that case they may be tempted to try to avoid versioning by coordinating with consumers of the API to introduce a breaking change. In our experience this approach has limited success; it typically causes disruption and a large coordination effort on both sides. Google uses this approach internally, but at considerable cost—this article describes some of Google's investments to make it work. It is usually much better for API providers to treat internal users and partners as if they were external consumers whose development process is independent.

Choosing the appropriate technique

You can probably see already that format version and entity versioning are fundamentally different techniques that solve different problems with different consequences, even though they both sail under the flag of versioning.

So when should you choose to do format versioning versus entity versioning? Usually the business requirements make the choice obvious.

In the case of the bank, it isn’t feasible to introduce a new entity version of an account in order to enable an API improvement. Accounts are stable and long-lived, and moving from old accounts to new ones is disruptive. A bank is unwilling to inconvenience its banking customers just to make life better for API developers. If the goal is just to improve the API, the bank should pick format versioning, which will limit the sort of changes that they make to superficial improvements.

The bank should consider introducing a new entity version if there's significant new value that it wants to expose to its banking customers, or if it's forced to do so for security or regulatory reasons. In the case of blockchain accounts, there may be publicity value as well as practical value. Entity version upgrades are less common than format versioning changes for established services, but they do happen; you may have received messages from your bank telling you about a significant technology upgrade to your accounts and alerting you to actions you need to take, or changes you will see.

Entity versioning puts an additional burden on API clients, because the older clients cannot work with the newer entities, even though they continue to work unchanged with the older ones. This puts pressure on client developers to produce a new client application or upgrade an existing one to work with the new API.

Entity versioning can work well for technology products, where the the users of the API and the core customers are often one and the same and rapid obsolescence is considered normal.

How do you implement the different versions of versioning?

On the web, you often see conflicting advice on whether or not a version number should appear in the URLs of a web API. The primary alternative is to put the version ID in an HTTP header. The better choice depends on whether you're doing format versioning or entity versioning.

For format versioning, put the version identifier in an HTTP header, not in the URL. Continuing the banking example, it’s conceptually simpler for each account to have a single URL, regardless of which format the API client wants to see it in. If you put a format version identifier in the URL, you are effectively making each format of each entity a separate web resource, with some behind-the-scenes magic that causes changes in one to be reflected in the other.

Not only is this a more complex conceptual model for users, it also creates problems with links. Suppose that in addition to having an API for accounts, the bank also has an API for customer records, and that each account contains a link to the record for the customer that owns it. If the developer asks for the version 2 format of the account, what version should be used in the link to the customer record? Should the server assume that the developer will also want to use the version 2 format of the customer record and provide that link? What if customer records don't even have a version 2 format?

Some APIs that put version identifiers in URLs (OpenStack, for example, and at least one bank we know) solve the link problem by having a “canonical” URL for each entity that's used in links, and a set of version-specific URLs for the same entity that are used to access the entity's format versions. Clients that want to follow a link have to convert a canonical URL in a link into a version-specific URL by following a documented formula. This is more complex for both the provider and the client; it's simpler to use a header.

The usual objection to putting format version identifiers in a header is that it's no longer possible to simply type a URL into a browser to test the result of a GET on a specific version. While this is true, it's not very hard to add headers in the browser using plugins like Postman, and you'll probably have to set headers anyway for the Authorization and Accept headers. If you'ew using the cURL shell command to test your API, adding headers is even simpler. You'll also need more than just the browser to create, update or delete requests to your API, so optimizing for GET only helps for one scenario. Your judgement may be different, but I have never found it very onerous to set a header.

There's no standard request header that's ideal for the client to say what format version it wants. The standard "Accept" header specifies which media types the client can accept (e.g., json, yaml, xml, html, plain text), and the standard "Accept-Language" header denotes which natural languages the client can accept (e.g., French, English, Spanish). Some API designers (e.g., the authors of the Restify framework) use a non-standard header called "Accept-Version". If you're doing format versioning, I recommend this header. The standard "Accept" headers allow the client to give a list of values they accept, and even provide a weighting for each. This level of complexity isn’t necessary for "Accept-Version"; a single value is enough. If you're meticulous, you should set a corresponding "Content-Version" header in the response. Further, it can be useful for clients if the server also puts the format version in the body of the response; in fact, if the representation of one resource is embedded in another, the body is the only place to put it. [This argument applies to a number of the standard headers too: e.g., Etag, Location, and Content-Location.]

By contrast, if you're doing entity versioning, the version identifier will appear somewhere in the URL of each entity—usually either in the domain name or the path. Users of the API do not have to be aware of this; for them, it's just the entity's URL. The version identifier will appear in the URL because the URL has to contain information for two different purposes: for routing requests to the correct part of the implementation for processing, and for identifying the entity within that implementation. Because requests for entities that belong to two different entity versions are almost always processed by a different part of the implementation or use different storage, the version identifier (or a proxy for it) must be somewhere in the URL for your routing infrastructure or implementation to use.

Coincidentally, banking provides a simple illustration of the principle that identifiers contain information for both routing and identification. If you have a checking account at a U.S. bank (the details are different in other countries, but the idea is similar), you'll find two numbers at the bottom of each check. The first is called the routing number. It identifies the institution that issued and can process this check. The second number identifies the check itself. Conceptually, entity URLs are like the numbers at the bottom of a check, though their formats may be different.

Do I have to define my versioning strategy up front?

You'll sometimes hear the advice that you must define a versioning strategy before you release your first version, or evolving your API will be impossible. This is not true.

You can always add a new versioning header later if you find the need to do format versioning and you can always add new URLs for new entities for a different entity version. Any requests that lack the format versioning header should be interpreted as meaning the first format version. Since instances of a new entity version get new URLs, you can easily introduce a version ID in those URLs without affecting the URLs of the entities of the first version. The new URLs may use a new hostname rather than adding path segments to URLs on the original hostname; whether or not you like that option will depend on your overall approach for managing hostnames.

Procrastination can be good

Laziness is not the only reason why you might not add versioning to the initial version of your API. If it turns out that versioning is never needed for your API, or for significant portions of your API, then the API will look better and be easier to use if it doesn’t include versioning in its initial release.

If you introduce an "Accept-Version" header in V1 of your API in anticipation of future "format versions" that never materialize, then you force your clients to set a header unnecessarily on every request.

Likewise, if you start all your URLs with the path prefix '/v1' in anticipation of future "entity version" introductions that never happen, then you make your URLs longer and uglier than they need to be.

More importantly, in both cases you introduce a complex topic to clients that you didn’t need to introduce.

Some more versioning tips

If you use versioning, make it clear what sort of versioning you use. If there'a a field in your HTTP requests and responses that says "version: V1," what does that mean? Does V1 apply to the persistent entity itself (entity versioning), or does it reflect the format in which the client asked to see the entity (format versioning)? Having a clear understanding of which versioning scheme or schemes you use helps your users understand how to use your API as it evolves.

If you're using format versioning and entity versioning together, signal them with different mechanisms. Format versions should go in headers—Accept-Version and Content-Version—in the request and response. Format versions can also be included in the bodies of responses and requests, for those requests that have them. Entity versions (which are really part of the entity type) belong in the request and response bodies; they're part of the representation of the entity.

Do not try to put versioning identifiers of either kind or entity type identifiers into the standard Accept or Content-Type headers; those headers should only include standard media types like text/html or application/json. Avoid using values like application/v2+json or application/customer+json; the media-type is not the place to try to encode version or type information. Unfortunately, even some of the web standards do this the wrong way, for example application/json-patch+json.

Don't put words like "beta" or "alpha" in version IDs for either format versioning or entity versioning. When you move from alpha to beta, or beta to general availability, you're making a statement about your level of support for the API, or its likely stability. You don't want to be in a position where the API version changes just because your level of support changes; you only want to change the version if there's a technical or functional reason for changing it. To illustrate this point, imagine I am a customer who develops a number of client applications that are using the V1beta4 version of an interface—a late-beta version. The API provider declares the product to be GA, and introduces the V1 version of the API, which is actually exactly the same as the V1beta4 API, since there were no breaking API changes between V1beta4 and GA. The V1Beta4 version of the API is still available, so my client applications don't break, but the language of the support agreement is clear—only users of the V1 version get full product support. The change to my client applications to upgrade to V1 is small—I only have to change the version number I'm using, which may even be as simple as recompiling with the latest release of the vendor-provided client libraries—but any change to my applications, no matter how small, needs to go through a full release process with QA testing, which costs me thousands of dollars. This is very annoying.

Hopefully this post helps bring a little more clarity to the topic of API versioning, and helps you with your design and implementation choices. Let us know what you think.

For more on API design, read the eBook, “Web API Design: The Missing Link” or check out more API design posts on the Apigee blog.

---

1 Representation would be another word for this concept that might be better aligned with REST terminology. For reasons I can't explain, the term "representation versioning" is not as appealing to me as "format versioning". 

By Seher Aylin Altiok, Product Manager

If you develop applications on Google Cloud Platform (GCP), you probably spend a lot of time in the GCP command line. But as we grow our GCP services, the number of commands and flags is growing by leaps and bounds. So today, we’re introducing a new command line interface (CLI) that lets you discover—and use—all these commands more efficiently: gcloud interactive.

The Google Cloud SDK offers a variety of command line tools to interact with GCP, namely:

• gcloud — GCP’s primary CLI 
• gsutil — CLI to interact with Google Cloud Storage 
• bq — CLI to interact with Google BigQuery 
• kubectl — Kubernetes Engine’s CLI

Currently in public alpha, the new interactive CLI environment provides auto-prompts and in-line help for gcloud, gsutil, bq and kubectl commands. No more context-switching as you search for command names, required flags or argument types in help pages. Now all of this information is included as part of the interactive environment as you type!
The interactive environment also supports standard bash features like:

• intermixing gcloud and standard bash commands 
• running commands like cd and pwd, and set/use shell variables across command executions 
• running and controlling background processes 
• TAB-completing shell variables, and much more!

For example, you can assign the result of the command to a variable and later call this variable as an input to a different command:

$ active_vms=$(gcloud compute instances list --format="value(NAME)" --filter="STATUS=RUNNING")
$ echo $active_vms

You can also create and run bash scripts while you're in the interactive environment.
For example, the following script iterates all compute instances and restarts the ones that have been TERMINATED.

#!/bin/bash
terminated_vms=$(gcloud compute instances list --format="value(NAME)" --filter="STATUS=terminated")
for name in $terminated_vms
do
  echo "Instance $name will restart."
  zone=$(gcloud compute instances list --format="value(ZONE)" --filter="NAME=$name")
  gcloud compute instances start $name --zone $zone 
done