Amazon Elasticsearch Service Developer Guide API Version 2015-01-01 Amazon Elasticsearch Service Developer Guide Amazon Elasticsearch Service: Developer Guide Copyright © 2018 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Amazon Elasticsearch Service Developer Guide Table of Contents What Is Amazon Elasticsearch Service? ................................................................................................. 1 Features of Amazon Elasticsearch Service ...................................................................................... 1 Supported Elasticsearch Versions ................................................................................................. 2 Pricing for Amazon ES ................................................................................................................ 3 Getting Started with Amazon Elasticsearch Service ......................................................................... 3 Related Services......................................................................................................................... 3 Getting Started with Amazon ES Domains ............................................................................................ 5 Step 1: Creating an Amazon ES Domain ........................................................................................ 5 Step 2: Uploading Data for Indexing............................................................................................ 7 Step 3: Searching Documents in an Amazon ES Domain .................................................................. 8 Step 4: Deleting an Amazon ES Domain ........................................................................................ 9 Creating and Configuring Amazon ES Domains .................................................................................... 10 Creating Amazon ES Domains................................................................................................... 10 Creating Amazon ES Domains (Console) .............................................................................. 10 Creating Amazon ES Domains (AWS CLI) ............................................................................. 13 Creating Amazon ES Domains (AWS SDKs) ........................................................................... 15 Configuring Amazon ES Domains ............................................................................................... 15 Configuring Amazon ES Domains (Console) .......................................................................... 16 Configuring Amazon ES Domains (AWS CLI) ......................................................................... 17 Configuring Amazon ES Domains (AWS SDKs) ...................................................................... 18 Configuring EBS-based Storage.................................................................................................. 18 Configuring EBS-based Storage (Console) ............................................................................ 18 Configuring EBS-based Storage (AWS CLI) ........................................................................... 19 Configuring EBS-based Storage (AWS SDKs) ........................................................................ 20 Modifying VPC Access Configuration ........................................................................................... 20 Configuring VPC Access (Console) ....................................................................................... 20 Configuring Amazon Cognito Authentication for Kibana ................................................................ 21 Configuring Access Policies........................................................................................................ 21 Configuring Access Policies (Console) .................................................................................. 21 Configuring Access Policies (AWS CLI) ................................................................................. 22 Configuring Access Policies (AWS SDKs) ............................................................................... 23 Configuring Automatic Snapshots............................................................................................... 23 Configuring Snapshots (Console) ........................................................................................ 23 Configuring Snapshots (AWS CLI) ....................................................................................... 23 Configuring Snapshots (AWS SDKs) ..................................................................................... 24 Configuring Advanced Options................................................................................................... 24 Configuring Advanced Options (Console) ............................................................................. 24 Configuring Advanced Options (AWS CLI) ............................................................................ 25 Configuring Advanced Options (AWS SDKs) ......................................................................... 25 Configuring Logs...................................................................................................................... 25 Enabling Log Publishing (Console) ...................................................................................... 26 Enabling Log Publishing (AWS CLI) ..................................................................................... 27 Enabling Log Publishing (AWS SDKs) .................................................................................. 28 Setting Elasticsearch Logging Thresholds for Slow Logs ........................................................ 28 Viewing Logs................................................................................................................... 29 Access Control................................................................................................................................. 30 Types of Policies ...................................................................................................................... 30 Resource-based Policies..................................................................................................... 30 Identity-based policies...................................................................................................... 32 IP-based Policies.............................................................................................................. 33 Making and Signing Amazon ES Requests .................................................................................... 33 When Policies Collide ................................................................................................................ 34 Policy Element Reference.......................................................................................................... 35 Advanced Options and API Considerations ................................................................................... 39 API Version 2015-01-01 iii Amazon Elasticsearch Service Developer Guide Configuring Access Policies........................................................................................................ 41 Additional Sample Policies......................................................................................................... 41 Managing Amazon ES Domains .......................................................................................................... 42 About Configuration Changes.................................................................................................... 42 Charges for Configuration Changes............................................................................................. 43 Enabling Zone Awareness.......................................................................................................... 43 Monitoring Cluster Metrics and Statistics with Amazon CloudWatch (Console) ................................... 46 Cluster Metrics................................................................................................................. 46 Dedicated Master Node Metrics.......................................................................................... 49 EBS Volume Metrics .......................................................................................................... 50 Instance Metrics............................................................................................................... 51 Logging Configuration API Calls with CloudTrail ........................................................................... 52 Amazon Elasticsearch Service Information in CloudTrail ......................................................... 53 Understanding Amazon Elasticsearch Service Log File Entries ................................................. 53 Tagging Amazon Elasticsearch Service Domains ........................................................................... 55 Working with Tags (Console) .............................................................................................. 56 Working with Tags (AWS CLI) ............................................................................................. 56 Working with Tags (AWS SDKs) .......................................................................................... 58 Indexing Data.................................................................................................................................. 59 Introduction to Indexing ............................................................................................................ 59 Signing HTTP Requests..................................................................................................................... 62 Java........................................................................................................................................ 62 Python.................................................................................................................................... 64 Ruby....................................................................................................................................... 66 Node....................................................................................................................................... 68 Loading Streaming Data into Amazon ES ............................................................................................ 70 Loading Streaming Data into Amazon ES from Amazon S3 ............................................................ 70 Prerequisites.................................................................................................................... 70 Creating the Lambda Deployment Package .......................................................................... 71 Creating the Lambda Function ........................................................................................... 72 Testing the Lambda Function ............................................................................................. 74 Loading Streaming Data into Amazon ES from Amazon Kinesis Data Streams ................................... 75 Prerequisites.................................................................................................................... 75 Creating the Lambda Function ........................................................................................... 76 Testing the Lambda Function ............................................................................................. 77 Loading Streaming Data into Amazon ES from Amazon DynamoDB ................................................ 78 Prerequisites.................................................................................................................... 78 Creating the Lambda Function ........................................................................................... 79 Testing the Lambda Function ............................................................................................. 80 Loading Streaming Data into Amazon ES from Amazon Kinesis Data Firehose ................................... 80 Loading Streaming Data into Amazon ES from Amazon CloudWatch ............................................... 81 Loading Data into Amazon ES from AWS IoT ............................................................................... 81 Searching Data................................................................................................................................. 82 URI Searches............................................................................................................................ 82 Request Body Searches ............................................................................................................. 83 Boosting Fields................................................................................................................ 84 Paginating Search Results .................................................................................................. 84 Search Result Highlighting ................................................................................................ 84 Count API........................................................................................................................ 86 Working with Index Snapshots ........................................................................................................... 87 Manual Snapshot Prerequisites................................................................................................... 88 Registering a Manual Snapshot Repository .................................................................................. 89 Sample Python Client ....................................................................................................... 90 Taking Manual Snapshots.......................................................................................................... 91 Restoring Snapshots................................................................................................................. 92 Upgrading Elasticsearch.................................................................................................................... 94 Troubleshooting an Upgrade...................................................................................................... 94 API Version 2015-01-01 iv Amazon Elasticsearch Service Developer Guide Starting an Upgrade................................................................................................................. 96 Using a Snapshot to Migrate Data.............................................................................................. 96 Kibana and Logstash........................................................................................................................ 99 Kibana.................................................................................................................................... 99 Controlling Access to Kibana .............................................................................................. 99 Configuring Kibana to Use a WMS Map Server ................................................................... 101 Connecting a Local Kibana Server to Amazon ES ................................................................ 102 Loading Bulk Data with the Logstash Plugin.............................................................................. 102 Authentication for Kibana................................................................................................................ 104 Prerequisites.......................................................................................................................... 104 About the User Pool ....................................................................................................... 105 About the Identity Pool ................................................................................................... 105 About the IAM Role ........................................................................................................ 105 Configuring an Amazon ES Domain .......................................................................................... 106 Configuring Amazon Cognito Authentication (Console) ........................................................ 106 Configuring Amazon Cognito Authentication (AWS CLI) ....................................................... 107 Configuring Amazon Cognito Authentication (AWS SDKs) ..................................................... 108 Allowing the Authenticated Role .............................................................................................. 108 Configuring Identity Providers .................................................................................................. 109 (Optional) Configuring Granular Access ..................................................................................... 110 User Groups and Tokens .................................................................................................. 112 Rules............................................................................................................................. 112 (Optional) Customizing the Login Page ..................................................................................... 112 (Optional) Configuring Advanced Security .................................................................................. 112 Testing.................................................................................................................................. 112 Limits.................................................................................................................................... 113 Common Configuration Issues.................................................................................................. 113 Disabling Amazon Cognito Authentication for Kibana .................................................................. 115 Deleting Domains that Use Amazon Cognito Authentication for Kibana ......................................... 116 VPC Support.................................................................................................................................. 117 Limitations............................................................................................................................. 119 About Access Policies on VPC Domains ...................................................................................... 119 Testing VPC Domains .............................................................................................................. 120 Prerequisites.......................................................................................................................... 121 Creating a VPC ....................................................................................................................... 121 Reserving IP Addresses in a VPC Subnet .................................................................................... 122 Service-Linked Role for VPC Access ........................................................................................... 123 Migrating from Public Access to VPC Access ............................................................................... 123 Amazon VPC Documentation ................................................................................................... 124 Best Practices................................................................................................................................. 125 Sizing Amazon ES Domains ..................................................................................................... 125 Calculating Storage Requirements .................................................................................... 125 Choosing the Number of Shards ....................................................................................... 126 Choosing Instance Types and Testing ................................................................................ 127 Dedicated Master Nodes.......................................................................................................... 128 Recommended CloudWatch Alarms ........................................................................................... 130 Petabyte Scale............................................................................................................................... 132 Encryption..................................................................................................................................... 134 Encryption at Rest .................................................................................................................. 134 Enabling Encryption of Data at Rest ................................................................................. 134 Disabling Encryption of Data at Rest ................................................................................. 135 Monitoring Domains That Encrypt Data at Rest .................................................................. 135 Other Considerations...................................................................................................... 136 Node-to-node Encryption........................................................................................................ 136 Enabling Node-to-node Encryption ................................................................................... 136 Disabling Node-to-node Encryption .................................................................................. 136 Other Considerations...................................................................................................... 137 API Version 2015-01-01 v Amazon Elasticsearch Service Developer Guide Using Curator to Rotate Data ........................................................................................................... 138 Sample Code......................................................................................................................... 138 Basic Settings......................................................................................................................... 141 Triggers................................................................................................................................. 141 Permissions............................................................................................................................ 141 Troubleshooting ............................................................................................................................. 143 Can't Access Kibana ................................................................................................................ 143 Can't Access VPC Domain ........................................................................................................ 143 Red Cluster Status .................................................................................................................. 143 Recovering from a Continuous Heavy Processing Load ......................................................... 144 Yellow Cluster Status .............................................................................................................. 145 ClusterBlockException............................................................................................................. 145 Lack of Available Storage Space ....................................................................................... 145 Block Disks Due to Low Memory ...................................................................................... 146 JVM OutOfMemoryError.......................................................................................................... 146 Failed Cluster Nodes ............................................................................................................... 146 Can't Close Index.................................................................................................................... 147 Can't SSH into Node ............................................................................................................... 147 "Not Valid for the Object's Storage Class" Snapshot Error ............................................................ 147 Invalid Host Header ................................................................................................................ 148 Browser Error When Using Kibana ............................................................................................ 148 Unauthorized Operation After Selecting VPC Access .................................................................... 148 Stuck at Loading After Creating VPC Domain ............................................................................. 148 Certificate Error When Using SDK ............................................................................................. 149 General Reference.......................................................................................................................... 150 Supported Instance Types ........................................................................................................ 150 Supported Elasticsearch Operations .......................................................................................... 151 Notable API Differences ................................................................................................... 151 Version 6.3.................................................................................................................... 152 Version 6.2.................................................................................................................... 153 Version 6.0.................................................................................................................... 154 Version 5.6.................................................................................................................... 155 Version 5.5.................................................................................................................... 155 Version 5.3.................................................................................................................... 156 Version 5.1.................................................................................................................... 157 Version 2.3.................................................................................................................... 158 Version 1.5.................................................................................................................... 158 Supported Plugins.................................................................................................................. 159 Output Plugins............................................................................................................... 161 Other Supported Resources ..................................................................................................... 161 Using the AWS SDKs ....................................................................................................................... 163 Java...................................................................................................................................... 163 Amazon ES Configuration API Reference ........................................................................................... 167 Actions.................................................................................................................................. 167 AddTags........................................................................................................................ 168 CreateElasticsearchDomain.............................................................................................. 169 DeleteElasticsearchDomain.............................................................................................. 174 DeleteElasticsearchServiceRole......................................................................................... 176 DescribeElasticsearchDomain............................................................................................ 176 DescribeElasticsearchDomainConfig................................................................................... 178 DescribeElasticsearchDomains.......................................................................................... 181 DescribeElasticsearchInstanceTypeLimits............................................................................ 184 DescribeReservedElasticsearchInstanceOfferings................................................................. 187 DescribeReservedElasticsearchInstances............................................................................. 188 GetCompatibleElasticsearchVersions.................................................................................. 190 GetUpgradeHistory......................................................................................................... 192 GetUpgradeStatus.......................................................................................................... 194 API Version 2015-01-01 vi Amazon Elasticsearch Service Developer Guide ListDomainNames........................................................................................................... 195 ListElasticsearchInstanceTypeDetails.................................................................................. 196 ListElasticsearchInstanceTypes (Deprecated)....................................................................... 198 ListElasticsearchVersions.................................................................................................. 199 ListTags......................................................................................................................... 201 PurchaseReservedElasticsearchInstance.............................................................................. 202 RemoveTags................................................................................................................... 203 UpdateElasticsearchDomainConfig .................................................................................... 204 UpgradeElasticsearchDomain............................................................................................ 209 Data Types............................................................................................................................ 210 AdvancedOptions............................................................................................................ 210 AdvancedOptionsStatus................................................................................................... 211 ARN.............................................................................................................................. 211 CognitoOptions.............................................................................................................. 212 CognitoOptionsStatus..................................................................................................... 212 CreateElasticsearchDomainRequest................................................................................... 212 DomainID...................................................................................................................... 213 DomainName................................................................................................................. 213 DomainNameList............................................................................................................ 213 EBSOptions.................................................................................................................... 214 ElasticsearchClusterConfig............................................................................................... 214 ElasticsearchDomainConfig .............................................................................................. 215 ElasticsearchDomainStatus............................................................................................... 215 ElasticsearchDomainStatusList.......................................................................................... 217 EncryptionAtRestOptions................................................................................................. 218 EncryptionAtRestOptionsStatus........................................................................................ 218 EndpointsMap................................................................................................................ 218 LogPublishingOptions..................................................................................................... 218 LogPublishingOptionsStatus............................................................................................. 219 NodeToNodeEncryptionOptions........................................................................................ 219 NodeToNodeEncryptionOptionsStatus............................................................................... 219 OptionState................................................................................................................... 220 OptionStatus.................................................................................................................. 220 ServiceURL..................................................................................................................... 220 SnapshotOptions............................................................................................................ 221 SnapshotOptionsStatus................................................................................................... 221 Tag............................................................................................................................... 221 TagKey.......................................................................................................................... 221 TagList.......................................................................................................................... 222 TagValue....................................................................................................................... 222 VPCDerivedInfo .............................................................................................................. 222 VPCDerivedInfoStatus..................................................................................................... 222 VPCOptions................................................................................................................... 223 VPCOptionsStatus........................................................................................................... 223 Errors.................................................................................................................................... 223 Limits............................................................................................................................................ 225 Cluster and Instance Limits ...................................................................................................... 225 EBS Volume Size Limits ........................................................................................................... 225 Network Limits....................................................................................................................... 227 Java Process Limit .................................................................................................................. 228 Reserved Instances......................................................................................................................... 229 Purchasing Reserved Instances (Console) ................................................................................... 229 Purchasing Reserved Instances (AWS CLI) ................................................................................... 230 Purchasing Reserved Instances (AWS SDKs) ................................................................................ 231 Examining Costs..................................................................................................................... 232 Tutorial: Creating a Search Application .............................................................................................. 233 Step 1: Index Sample Data...................................................................................................... 233 API Version 2015-01-01 vii Amazon Elasticsearch Service Developer Guide Step 2: Create the API ............................................................................................................ 233 Step 3: Create the Lambda Function ......................................................................................... 235 Step 4: Modify the Domain Access Policy ................................................................................... 236 Step 5: Test the Web Application ............................................................................................. 236 Next Steps............................................................................................................................. 238 Tutorial: Visualizing Support Calls ..................................................................................................... 239 Step 1: Configure Prerequisites ................................................................................................ 240 Step 2: Copy Sample Code ...................................................................................................... 240 (Optional) Step 3: Add Sample Data ......................................................................................... 243 Step 4: Analyze and Visualize Your Data .................................................................................... 244 Step 5: Clean Up Resources and Next Steps ............................................................................... 252 Document History.......................................................................................................................... 253 Using Service-Linked Roles .............................................................................................................. 256 Service-Linked Role Permissions for Amazon ES ......................................................................... 256 Creating a Service-Linked Role for Amazon ES ........................................................................... 256 Editing a Service-Linked Role for Amazon ES ............................................................................. 257 Deleting a Service-Linked Role for Amazon ES ........................................................................... 257 Cleaning Up a Service-Linked Role .................................................................................... 257 Manually Delete a Service-Linked Role .............................................................................. 257 AWS Glossary................................................................................................................................. 258 API Version 2015-01-01 viii Amazon Elasticsearch Service Developer Guide Features of Amazon Elasticsearch Service What Is Amazon Elasticsearch Service? Amazon Elasticsearch Service (Amazon ES) is a managed service that makes it easy to deploy, operate, and scale Elasticsearch clusters in the AWS Cloud. Elasticsearch is a popular open-source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. With Amazon ES, you get direct access to the Elasticsearch APIs; existing code and applications work seamlessly with the service. Amazon ES provisions all the resources for your Elasticsearch cluster and launches it. It also automatically detects and replaces failed Elasticsearch nodes, reducing the overhead associated with self-managed infrastructures. You can scale your cluster with a single API call or a few clicks in the console. To get started using Amazon ES, you create a domain. An Amazon ES domain is synonymous with an Elasticsearch cluster. Domains are clusters with the settings, instance types, instance counts, and storage resources that you specify. You can use the Amazon ES console to set up and configure a domain in minutes. If you prefer programmatic access, you can use the AWS CLI or the AWS SDKs. Topics • Features of Amazon Elasticsearch Service (p. 1) • Supported Elasticsearch Versions (p. 2) • Pricing for Amazon Elasticsearch Service (p. 3) • Getting Started with Amazon Elasticsearch Service (p. 3) • Related Services (p. 3) Features of Amazon Elasticsearch Service Amazon ES includes the following features: Scale • Numerous configurations of CPU, memory, and storage capacity, known as instance types • Up to 1.5 PB of instance storage • Amazon EBS storage volumes Security • AWS Identity and Access Management (IAM) access control • Easy integration with Amazon VPC and VPC security groups • Encryption of data at rest • Amazon Cognito authentication for Kibana Stability API Version 2015-01-01 1 Amazon Elasticsearch Service Developer Guide Supported Elasticsearch Versions • Multiple geographical locations for your resources, known as regions and Availability Zones • Dedicated master nodes to offload cluster management tasks • Automated snapshots to back up and restore Amazon ES domains • Cluster node allocation across two Availability Zones in the same region, known as zone awareness Integration with Popular Services • Data visualization using Kibana • Integration with Amazon CloudWatch for monitoring Amazon ES domain metrics and setting alarms • Integration with AWS CloudTrail for auditing configuration API calls to Amazon ES domains • Integration with Amazon S3, Amazon Kinesis, and Amazon DynamoDB for loading streaming data into Amazon ES Supported Elasticsearch Versions Amazon ES currently supports the following Elasticsearch versions: • 6.3 • 6.2 • 6.0 • 5.6 • 5.5 • 5.3 • 5.1 • 2.3 • 1.5 Compared to earlier versions of Elasticsearch, the 6.x versions offer powerful features that make them faster, more secure, and easier to use. Here are some highlights: • Index splitting – If an index outgrows its original number of shards, the _split API offers a convenient way to split each primary shard into two or more shards in a new index. • Vega visualizations – Kibana 6.2 and newer support the Vega visualization language, which lets you make context-aware Elasticsearch queries, combine multiple data sources into a single graph, add user interactivity to graphs, and much more. • Ranking evaluation – The _rank_eval API lets you measure and track how ranked search results perform against a set of queries to ensure that your searches perform as expected. • Composite aggregations – These aggregations build composite buckets from one or more fields and sort them in "natural order" (alphabetically for terms, numerically or by date for histograms). • Higher indexing performance – Newer versions of Elasticsearch provide superior indexing capabilities that significantly increase the throughput of data updates. • Better safeguards – The 6.x versions of Elasticsearch offer many safeguards that are designed to prevent overly broad or complex queries from negatively affecting the performance and stability of the cluster. • Kibana autocomplete – Kibana 6.3 and newer support autocomplete for queries, which greatly improves the day-to-day user experience. For more information about the differences between Elasticsearch versions and the APIs that Amazon ES supports, see the section called “Supported Elasticsearch Operations” (p. 151). API Version 2015-01-01 2