The recent post by Jayush Luniya announced the community release of Apache Ambari 2.0. One of the three key Ambari features that Jayush discussed was Rolling Upgrades, enabling Hadoop operators to upgrade from one version of HDP to the next, with minimal disruption to the cluster.
The Hortonworks development team worked long and hard to make the Hadoop platform "rolling upgradeable". That groundwork was available in Hortonworks Data Platform 2.2 as described in this previous post. That breakthrough made rolling upgrades possible, but they were not automated; users still had to follow a set of manual steps to upgrade the cluster in a rolling fashion.
But now with Apache Ambari 2.0, you can automate rolling upgrades, and this post walks through how that works.
Overview
HDP 2.2 takes care of cluster API compatibility, seamless job execution, component high availability and software packaging. The "automation of steps" was the final piece to the puzzle, which just fell into place with Ambari 2.0.
HDP has a certified process for executing the upgrade in a certain order so component interdependencies are handled correctly and the software versions can be switched in a rolling fashion. The process follows this order:
-
Cross-cutting components (such as ZooKeeper and Ranger)
-
Core master components (such as HDFS NameNode and YARN ResourceManager)
-
Cluster slave components (such as HDFS DataNodes and YARN NodeManagers)
-
Ancillary components and clients
The Four-Step Process
Ambari 2.0 automates the process defined by HDP to expose a method for automating a cluster rolling upgrade through the Ambari Web UI.
Step 1: Register the New Version
The first part of a rolling upgrade is to register the new version with Ambari. This lets Ambari know that there is new software available and where to get it, which is particularly important in the case of installs when there is no Internet access.
Step 2: Install the software on all hosts
Second, the operator instructs Ambari to install the software on all the hosts in the cluster. This step is deliberately called out because with any cluster install, the most time-consuming task is getting the software on the machines. This install happens outside of the range of normal cluster operations, in order to prevent downtime. The end result is that the new software is placed side-by-side the currently running software (of an older version).
Step 3: Perform the upgrade
Once the software is installed, Ambari performs the upgrade by orchestrating the process defined by HDP to roll through the cluster and switch to the new software version. This experience is wizard-driven and shows the user each step in the process as it goes from ZooKeeper Servers and Core Master Components all the way through Client Components.
Ambari has built-in guardrails. At periodic stages of the upgrade, Ambari will automatically run Service checks to validate and verify expected functionality. As an added precaution, there are also points where Ambari will prompt (and encourage) you to confirm operations that are indeed running smoothly. If it at any point it seems that things have gone awry, you have the option to Downgrade, which orders Ambari to reverse the operations already performed to get you safely to the original state.
Step 4: Finalize the upgrade
Once the upgrade process is complete, the operator is prompted to finalize. This completes the landing process on the new version. Alternatively, you are given a final chance to Downgrade and remain on the original version.
The automated rolling upgrade feature of Ambari 2.0 also introduces a new concept of an Upgrade Pack to Ambari Stacks. Check out the upgrade pack, and you can see that it defines the steps, tasks and rules to guide the automation. This approach was important to make sure we have future ability to extend Ambari as more ecosystem components come under Ambari management.
Download Ambari and Learn More
We think you'll agree this capability is a huge step forward for Ambari. We encourage you to tryout Ambari 2.0 today. Checkout these resources for more information: