Hortonworks Data Platform Apache Ambari User Guide (November 28, 2016) docs.cloudera.com hdp-ambari-user-guide November 28, 2016 Hortonworks Data Platform: Apache Ambari User Guide Copyright © 2012-2016 Hortonworks, Inc. Some rights reserved. The Hortonworks Data Platform, powered by Apache Hadoop, is a massively scalable and 100% open source platform for storing, processing and analyzing large volumes of data. It is designed to deal with data from many sources and formats in a very quick, easy and cost-effective manner. The Hortonworks Data Platform consists of the essential set of Apache Hadoop projects including MapReduce, Hadoop Distributed File System (HDFS), HCatalog, Pig, Hive, HBase, ZooKeeper and Ambari. Hortonworks is the major contributor of code and patches to many of these projects. These projects have been integrated and tested as part of the Hortonworks Data Platform release process and installation and configuration tools have also been included. Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of our code back to the Apache Software Foundation. The Hortonworks Data Platform is Apache-licensed and completely open source. We sell only expert technical support, training and partner-enablement services. All of our technology is, and will remain free and open source. Please visit the Hortonworks Data Platform page for more information on Hortonworks technology. For more information on Hortonworks services, please visit either the Support or Training page. Feel free to Contact Us directly to discuss your specific needs. Except where otherwise noted, this document is licensed under Creative Commons Attribution ShareAlike 4.0 License. http://creativecommons.org/licenses/by-sa/4.0/legalcode ii hdp-ambari-user-guide November 28, 2016 Table of Contents 1. Overview: Ambari User Guide ...................................................................................... 1 1.1. Architecture ...................................................................................................... 1 1.1.1. Sessions .................................................................................................. 2 1.2. Accessing Ambari Web ...................................................................................... 2 2. Viewing the Ambari Dashboards .................................................................................. 3 2.1. Viewing the Cluster Dashboard ......................................................................... 3 2.1.1. Scanning Service Status .......................................................................... 4 2.1.2. Widget Descriptions ............................................................................... 4 2.1.3. Widget Details ....................................................................................... 5 2.1.4. Linking to Service UIs ............................................................................. 5 2.1.5. Viewing Cluster-Wide Metrics ................................................................. 6 2.2. Modifying the Cluster Dashboard ...................................................................... 7 2.2.1. Adding a Widget to the Dashboard ....................................................... 8 2.2.2. Resetting the Dashboard ........................................................................ 8 2.2.3. Customizing Widget Display ................................................................... 9 2.3. Viewing Cluster Heatmaps ................................................................................ 9 3. Managing Hosts ......................................................................................................... 11 3.1. Working with Hosts ........................................................................................ 11 3.2. Filtering the Hosts List ..................................................................................... 11 3.3. Performing Host-Level Actions ......................................................................... 12 3.4. Viewing Components on a Host ...................................................................... 13 3.5. Decommissioning Masters and Slaves .............................................................. 15 3.5.1. How to Decommission a Component .................................................... 16 3.6. How to Delete a Component .......................................................................... 16 3.7. Deleting a Host from a Cluster ........................................................................ 17 3.7.1. How to Delete a Host from a Cluster .................................................... 17 3.8. Setting Maintenance Mode ............................................................................. 18 3.8.1. Setting Maintenance Mode for Services, Components, and Hosts .......... 18 3.8.2. How to Turn On Maintenance Mode for a Service ................................ 18 3.8.3. How to Turn On Maintenance Mode for a Host .................................... 18 3.8.4. How to Turn On Maintenance Mode for a Host (alternative using filtering for hosts) .......................................................................................... 19 3.8.5. Maintenance Mode Use Cases .............................................................. 19 3.9. Adding Hosts to a Cluster ............................................................................... 21 3.10. Rack Awareness ............................................................................................ 21 4. Managing Services ..................................................................................................... 23 4.1. Starting and Stopping All Services ................................................................... 24 4.2. Selecting a Service ........................................................................................... 24 4.3. Adding a Service ............................................................................................. 24 4.3.1. Adding a Service to your Hadoop cluster .............................................. 24 4.4. Editing Service Config Properties ..................................................................... 28 4.5. Viewing Service Summary and Alerts ............................................................... 29 4.5.1. Alerts and Health Checks ...................................................................... 29 4.5.2. Modifying the Service Dashboard ......................................................... 29 4.6. Performing Service Actions .............................................................................. 32 4.7. Removing A Service ......................................................................................... 32 4.8. Monitoring Background Operations ................................................................ 32 4.9. Operations Audit ............................................................................................ 34 iii hdp-ambari-user-guide November 28, 2016 4.10. Using Quick Links .......................................................................................... 35 4.11. Rolling Restarts ............................................................................................. 35 4.11.1. Setting Rolling Restart Parameters ...................................................... 35 4.11.2. Aborting a Rolling Restart .................................................................. 36 4.12. Managing YARN ........................................................................................... 36 4.12.1. Refreshing YARN Capacity Scheduler .................................................. 36 4.13. Managing HDFS ............................................................................................ 37 4.13.1. Rebalancing HDFS ............................................................................... 37 4.13.2. Tuning Garbage Collection ................................................................. 37 4.14. Managing Storm ........................................................................................... 38 4.15. Managing Apache Atlas ................................................................................ 38 5. Managing Service High Availability ............................................................................. 39 5.1. NameNode High Availability ............................................................................ 39 5.1.1. How To Configure NameNode High Availability .................................... 39 5.1.2. How to Roll Back NameNode HA ......................................................... 44 5.2. ResourceManager High Availability ................................................................. 52 5.2.1. How to Configure ResourceManager High Availability ........................... 52 5.2.2. How to Disable ResourceManager High Availability ............................... 53 5.3. HBase High Availability .................................................................................... 55 5.3.1. Adding an HBase Master Component ................................................... 55 5.4. Hive High Availability ...................................................................................... 55 5.4.1. Adding a Hive Metastore Component .................................................. 56 5.4.2. Adding a HiveServer2 Component ........................................................ 56 5.4.3. Adding a WebHCat Component ........................................................... 56 5.5. Storm High Availability .................................................................................... 57 5.5.1. Adding a Nimbus Component .............................................................. 57 5.6. Oozie High Availability .................................................................................... 57 5.6.1. Adding an Oozie Server Component ..................................................... 57 5.7. Apache Atlas High Availability ......................................................................... 58 6. Managing Configurations ........................................................................................... 60 6.1. Configuring Services ........................................................................................ 60 6.1.1. Updating Service Properties .................................................................. 60 6.1.2. Restarting Components ........................................................................ 60 6.2. Using Host Config Groups ............................................................................... 60 6.3. Customizing Log Settings ................................................................................ 62 6.4. Downloading Client Configs ............................................................................ 63 6.5. Service Configuration Versions ........................................................................ 63 6.5.1. Basic Concepts ...................................................................................... 63 6.5.2. Terminology ......................................................................................... 64 6.5.3. Saving a Change ................................................................................... 64 6.5.4. Viewing History .................................................................................... 65 6.5.5. Comparing Versions .............................................................................. 66 6.5.6. Reverting a Change .............................................................................. 67 6.5.7. Versioning and Host Config Groups ...................................................... 67 7. Administering the Cluster ........................................................................................... 69 7.1. Stack and Versions .......................................................................................... 69 7.2. Service Accounts ............................................................................................. 69 7.3. Kerberos ......................................................................................................... 70 7.3.1. How To Regenerate Keytabs ................................................................ 71 7.3.2. How To Disable Kerberos ..................................................................... 71 8. Monitoring and Alerts ............................................................................................... 72 iv hdp-ambari-user-guide November 28, 2016 8.1. Managing Alerts ............................................................................................. 72 8.1.1. Alert Types ........................................................................................... 73 8.1.2. Alert Check Counts ............................................................................... 74 8.2. Configuring Notifications ................................................................................. 75 8.2.1. Customizing Notification Templates ...................................................... 77 8.3. List of Predefined Alerts .................................................................................. 79 8.3.1. HDFS Service Alerts .............................................................................. 80 8.3.2. HDFS HA Alerts .................................................................................... 83 8.3.3. NameNode HA Alerts ........................................................................... 84 8.3.4. YARN Alerts ......................................................................................... 84 8.3.5. MapReduce2 Alerts .............................................................................. 85 8.3.6. HBase Service Alerts ............................................................................. 86 8.3.7. Hive Alerts ............................................................................................ 87 8.3.8. Oozie Alerts ......................................................................................... 87 8.3.9. ZooKeeper Alerts ................................................................................. 87 8.3.10. Ambari Alerts ..................................................................................... 88 8.3.11. Ambari Metrics Alerts ......................................................................... 88 9. Using Ambari Core Services ........................................................................................ 90 9.1. Ambari Metrics ................................................................................................ 90 9.1.1. AMS Architecture ................................................................................. 90 9.1.2. Using Grafana ...................................................................................... 91 9.1.3. Performance Tuning ........................................................................... 123 9.1.4. Moving the Metrics Collector .............................................................. 127 9.1.5. (Optional) Enabling Individual Region, Table, and User Metrics for HBase ........................................................................................................... 127 9.2. Ambari Log Search (Technical Preview) .......................................................... 128 9.2.1. Log Search Architecture ...................................................................... 128 9.2.2. Installing Log Search ........................................................................... 129 9.2.3. Using Log Search ................................................................................ 130 9.3. Ambari Infra ................................................................................................. 133 v hdp-ambari-user-guide November 28, 2016 1. Overview: Ambari User Guide Hadoop is a large-scale, distributed data storage and processing infrastructure using clusters of commodity hosts networked together. Monitoring and managing such complex distributed systems is a non-trivial task. To help you manage the complexity, Apache Ambari collects a wide range of information from the cluster's nodes and services and presents it to you in an easy-to-read and use, centralized web interface, Ambari Web. Ambari Web displays information such as service-specific summaries, graphs, and alerts. You use Ambari Web to create and manage your HDP cluster and to perform basic operational tasks such as starting and stopping services, adding hosts to your cluster, and updating service configurations. You also can use Ambari Web to perform administrative tasks for your cluster such as enabling Kerberos security and performing Stack upgrades. For more information on administering Ambari users, groups and views, refer to Hortonworks Data Platform Apache Ambari Administration. 1.1. Architecture The Ambari Server serves as the collection point for data from across your cluster. Each host has a copy of the Ambari Agent - either installed automatically by the Install wizard or manually - which allows the Ambari Server to control each host. Figure - Ambari Server Architecture 1 hdp-ambari-user-guide November 28, 2016 1.1.1. Sessions Ambari Web is a client-side JavaScript application, which calls the Ambari REST API (accessible from the Ambari Server) to access cluster information and perform cluster operations. After authenticating to Ambari Web, the application authenticates to the Ambari Server. Communication between the browser and server occurs asynchronously via the REST API. The Ambari Web UI periodically accesses the Ambari REST API, which resets the session timeout. Therefore, by default, Ambari Web sessions do not timeout automatically. You can configure Ambari to timeout after a period of inactivity. Refer to Ambari Web Inactivity Timeout in Hortonworks Data Platform Apache Ambari Security for more information. 1.2. Accessing Ambari Web Typically, you start the Ambari Server and Ambari Web as part of the installation process. If Ambari Server is stopped, you can start it using a command line editor on the Ambari Server host machine. Enter the following command: ambari-server start To access Ambari Web, open a supported browser and enter the Ambari Web URL: http://<your.ambari.server>:8080 Enter your user name and password. If this is the first time Ambari Web is accessed, use the default values, admin/admin. These values can be changed, and new users provisioned, using the Manage Ambari option. For more information about managing users and other administrative tasks, see Administering Ambari. 2 hdp-ambari-user-guide November 28, 2016 2. Viewing the Ambari Dashboards This topic describes how to use Ambari Dashboards to monitor your HDP cluster. To see the Ambari Dashboards, select Dasboard, located at the top of the Ambari main window. • Viewing the Cluster Dashboard [3] • Modifying the Cluster Dashboard [7] • Viewing Cluster Heatmaps [9] 2.1. Viewing the Cluster Dashboard Ambari Web displays the Dashboard page as the home page. Use the Dashboard to view the operating status of your cluster. Each metrics widget displays status information for a single service in your HDP cluster. The Dashboard displays all metrics for the HDFS, YARN, HBase, and Storm services, and cluster-wide metrics by default. You can add and remove individual widgets, and rearrange the Dashboard by dragging and dropping each widget to a new location in the dashboard. Status information appears as simple pie and bar charts, more complex charts showing usage and load, sets of links to additional data sources, and values for operating parameters such as uptime and average RPC queue wait times. Most widgets display a single fact by default. For example, HDFS Disk Usage displays a load chart and a percentage figure. Note Each Service installed in your cluster also has a Service-specific dashboard. Refer to Modifying the Service Dashboard for more information. 3 hdp-ambari-user-guide November 28, 2016 2.1.1. Scanning Service Status Notice the color of the dot appearing next to each component name in a list of components, services or hosts. The dot color and blinking action indicates operating status of each component, service, or host. For example, in the Summary View, notice green dot next to each service name. The following colors and actions indicate service status: Status Indicators Color Status Solid Green All masters are running Blinking Green Starting up Solid Red At least one master is down Blinking Red Stopping Click the service name to open the Services screen, where you can see more detailed information on each service. 2.1.2. Widget Descriptions The Dashboard includes metrics for the following services: View Metrics that indicate the operating status of your cluster on the Ambari Dashboard. Each metrics widget displays status information for a single service in your HDP cluster. The Ambari Dashboard displays all metrics for the HDFS, YARN, HBase, and Storm services, and cluster-wide metrics by default. Note Metrics data for Storm is buffered and sent as a batch to Ambari every five minutes. After adding the Storm service, anticipate a five-minute delay for Storm metrics to appear. You can add and remove individual widgets, and rearrange the dashboard by dragging and dropping each widget to a new location in the dashboard. Status information appears as simple pie and bar charts, more complex charts showing usage and load, sets of links to additional data sources, and values for operating parameters such as uptime and average RPC queue wait times. Most widgets display a single fact by default. For example, HDFS Disk Usage displays a load chart and a percentage figure. The Ambari Dashboard includes metrics for the following services: Ambari Service Metrics and Descriptions Metric: Description: HDFS HDFS Disk Usage The Percentage of DFS used, which is a combination of DFS and non-DFS used. 4 hdp-ambari-user-guide November 28, 2016 Metric: Description: Data Nodes Live The number of DataNodes live, as reported from the NameNode. NameNode Heap The percentage of NameNode JVM Heap used. NameNode RPC The average RPC queue latency. NameNode CPU WIO The percentage of CPU Wait I/O. NameNode Uptime The NameNode uptime calculation. YARN (HDP 2.1 or later Stacks) ResourceManager Heap The percentage of ResourceManager JVM Heap used. ResourceManager Uptime The ResourceManager uptime calculation. NodeManagers Live The number of DataNodes live, as reported from the ResourceManager. YARN Memory The percentage of available YARN memory (used vs. total available). HBase HBase Master Heap The percentage of NameNode JVM Heap used. HBase Ave Load The average load on the HBase server. HBase Master Uptime The HBase Master uptime calculation. Region in Transition The number of HBase regions in transition. Storm (HDP 2.1 or later Stacks) Supervisors Live The number of Supervisors live, as reported from the Nimbus server. 2.1.3. Widget Details To see more detailed information about a service, hover your cursor over a Metrics widget. More detailed information about the service displays, as shown in the following example: • To remove a widget from the mashup, click the white X. • To edit the display of information in a widget, click the pencil icon. For more information about editing a widget, see Customizing Metrics Display. 2.1.4. Linking to Service UIs The HDFS Links and HBase Links widgets list HDP components for which links to more metrics information, such as thread stacks, logs and native component UIs are available. For example, you can link to NameNode, Secondary NameNode, and DataNode components for HDFS, using the links shown in the following example: 5