HBase version number and compatibility

Aspirational Semantic Versioning

Starting with the 1.0.0 release, HBase is working towards Semantic Versioning for its release versioning. In summary:

Given a version number MAJOR.MINOR.PATCH, increment the:

• MAJOR version when you make incompatible API changes,
• MINOR version when you add functionality in a backwards-compatible manner, and
• PATCH version when you make backwards-compatible bug fixes.
• Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Compatibility Dimensions

In addition to the usual API versioning considerations HBase has other compatibility dimensions that we need to consider.

Client-Server wire protocol compatibility

• Allows updating client and server out of sync.
• We could only allow upgrading the server first. I.e. the server would be backward compatible to an old client, that way new APIs are OK.
• Example: A user should be able to use an old client to connect to an upgraded cluster.

Server-Server protocol compatibility

• Servers of different versions can co-exist in the same cluster.
• The wire protocol between servers is compatible.
• Workers for distributed tasks, such as replication and log splitting, can co-exist in the same cluster.
• Dependent protocols (such as using ZK for coordination) will also not be changed.
• Example: A user can perform a rolling upgrade.

File format compatibility

• Support file formats backward and forward compatible
• Example: File, ZK encoding, directory layout is upgraded automatically as part of an HBase upgrade. User can downgrade to the older version and everything will continue to work.

Client API compatibility

• Allow changing or removing existing client APIs.
• An API needs to be deprecated for a whole major version before we will change/remove it.
  • An example: An API was deprecated in 2.0.1 and will be marked for deletion in 4.0.0. On the other hand, an API deprecated in 2.0.0 can be removed in 3.0.0.
  • Occasionally mistakes are made and internal classes are marked with a higher access level than they should. In these rare circumstances, we will accelerate the deprecation schedule to the next major version (i.e., deprecated in 2.2.x, marked IA.Private 3.0.0). Such changes are communicated and explained via release note in Jira.
• APIs available in a patch version will be available in all later patch versions. However, new APIs may be added which will not be available in earlier patch versions.
• New APIs introduced in a patch version will only be added in a source compatible way: i.e. code that implements public APIs will continue to compile. ¹
  • Example: A user using a newly deprecated API does not need to modify application code with HBase API calls until the next major version. *

Client Binary compatibility

• Client code written to APIs available in a given patch release can run unchanged (no recompilation needed) against the new jars of later patch versions.
• Client code written to APIs available in a given patch release might not run against the old jars from an earlier patch version.
  • Example: Old compiled client code will work unchanged with the new jars.
• If a Client implements an HBase Interface, a recompile MAY be required upgrading to a newer minor version (See release notes for warning about incompatible changes). All effort will be made to provide a default implementation so this case should not arise.

Server-Side Limited API compatibility (taken from Hadoop)

• Internal APIs are marked as Stable, Evolving, or Unstable
• This implies binary compatibility for coprocessors and plugins (pluggable classes, including replication) as long as these are only using marked interfaces/classes.
• Example: Old compiled Coprocessor, Filter, or Plugin code will work unchanged with the new jars.

Dependency Compatibility

• An upgrade of HBase will not require an incompatible upgrade of a dependent project, except for Apache Hadoop.
• An upgrade of HBase will not require an incompatible upgrade of the Java runtime.
• Example: Upgrading HBase to a version that supports Dependency Compatibility won't require that you upgrade your Apache ZooKeeper service.
• Example: If your current version of HBase supported running on JDK 8, then an upgrade to a version that supports Dependency Compatibility will also run on JDK 8.

Operational Compatibility

• Metric changes
• Behavioral changes of services
• JMX APIs exposed via the /jmx/ endpoint

Summary

• A patch upgrade is a drop-in replacement. Any change that is not Java binary and source compatible would not be allowed. Downgrading versions within patch releases may not be compatible. ²
• A minor upgrade requires no application/client code modification. Ideally it would be a drop-in replacement but client code, coprocessors, filters, etc might have to be recompiled if new jars are used.
• A major upgrade allows the HBase community to make breaking changes.

Compatibility Matrix:

HBase API Surface

HBase has a lot of API points, but for the compatibility matrix above, we differentiate between Client API, Limited Private API, and Private API. HBase uses Apache Yetus Audience Annotations to guide downstream expectations for stability.

• InterfaceAudience (javadocs): captures the intended audience, possible values include:
  • Public: safe for end users and external projects
  • LimitedPrivate: used for internals we expect to be pluggable, such as coprocessors
  • Private: strictly for use within HBase itself Classes which are defined as IA.Private may be used as parameters or return values for interfaces which are declared IA.LimitedPrivate. Treat the IA.Private object as opaque; do not try to access its methods or fields directly.
• InterfaceStability (javadocs): describes what types of interface changes are permitted. Possible values include:
  • Stable: the interface is fixed and is not expected to change
  • Evolving: the interface may change in future minor versions
  • Unstable: the interface may change at any time

Please keep in mind the following interactions between the InterfaceAudience and InterfaceStability annotations within the HBase project:

• IA.Public classes are inherently stable and adhere to our stability guarantees relating to the type of upgrade (major, minor, or patch).
• IA.LimitedPrivate classes should always be annotated with one of the given InterfaceStability values. If they are not, you should presume they are IS.Unstable.
• IA.Private classes should be considered implicitly unstable, with no guarantee of stability between releases.

HBase Client API

HBase Client API consists of all the classes or methods that are marked with InterfaceAudience.Public interface. All main classes in hbase-client and dependent modules have either InterfaceAudience.Public, InterfaceAudience.LimitedPrivate, or InterfaceAudience.Private marker. Not all classes in other modules (hbase-server, etc) have the marker. If a class is not annotated with one of these, it is assumed to be a InterfaceAudience.Private class.

HBase LimitedPrivate API

LimitedPrivate annotation comes with a set of target consumers for the interfaces. Those consumers are coprocessors, phoenix, replication endpoint implementations or similar. At this point, HBase only guarantees source and binary compatibility for these interfaces between patch versions.

HBase Private API

All classes annotated with InterfaceAudience.Private or all classes that do not have the annotation are for HBase internal use only. The interfaces and method signatures can change at any point in time. If you are relying on a particular interface that is marked Private, you should open a jira to propose changing the interface to be Public or LimitedPrivate, or an interface exposed for this purpose.

Binary Compatibility

When we say two HBase versions are compatible, we mean that the versions are wire and binary compatible. Compatible HBase versions means that clients can talk to compatible but differently versioned servers. It means too that you can just swap out the jars of one version and replace them with the jars of another, compatible version and all will just work. Unless otherwise specified, HBase point versions are (mostly) binary compatible. You can safely do rolling upgrades between binary compatible versions; i.e. across maintenance releases: e.g. from 1.4.4 to 1.4.6. See "Does compatibility between versions also mean binary compatibility?" discussion on the HBase dev mailing list.

Footnotes

1. Running an offline upgrade tool without downgrade might be needed. We will typically only support migrating data from major version X to major version X+1. ↩ ↩²

2. See http://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html. ↩