Amazon DynamoDB Developer Guide API Version 2012-08-10 Amazon DynamoDB Developer Guide Amazon DynamoDB: Developer Guide Copyright © 2023 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Amazon DynamoDB Developer Guide Table of Contents What is Amazon DynamoDB? ............................................................................................................... 1 High availability and durability .................................................................................................... 1 Getting started with DynamoDB.................................................................................................. 1 DynamoDB tutorials................................................................................................................... 2 How it works............................................................................................................................. 2 Cheat sheet....................................................................................................................... 3 Core components............................................................................................................... 6 DynamoDB API................................................................................................................ 12 Supported data types and naming rules .............................................................................. 15 Read consistency.............................................................................................................. 19 Read/write capacity mode ................................................................................................. 19 Table classes.................................................................................................................... 24 Partitions and data distribution .......................................................................................... 25 From SQL to NoSQL ................................................................................................................. 27 Relational or NoSQL? ........................................................................................................ 28 Characteristics of databases ............................................................................................... 29 Creating a table ............................................................................................................... 31 Getting information about a table ...................................................................................... 33 Writing data to a table ..................................................................................................... 34 Reading data from a table ................................................................................................. 37 Managing indexes............................................................................................................. 43 Modifying data in a table .................................................................................................. 46 Deleting data from a table ................................................................................................ 49 Removing a table ............................................................................................................. 50 Additional Amazon DynamoDB resources .................................................................................... 51 Blog posts, repositories, and guides .................................................................................... 51 Data modeling and design pattern presentations ................................................................. 52 Training courses............................................................................................................... 52 Tools for coding and visualization ...................................................................................... 52 Setting up DynamoDB...................................................................................................................... 53 Setting up DynamoDB local (downloadable version) ..................................................................... 53 Deploying ........................................................................................................................ 53 Usage notes..................................................................................................................... 57 Setting up DynamoDB (web service) ........................................................................................... 60 Signing up for AWS .......................................................................................................... 60 Getting an AWS access key ................................................................................................ 60 Configuring your credentials .............................................................................................. 61 Accessing DynamoDB........................................................................................................................ 62 Using the console ..................................................................................................................... 62 Using the AWS CLI ................................................................................................................... 63 Downloading and configuring the AWS CLI .......................................................................... 63 Using the AWS CLI with DynamoDB .................................................................................... 63 Using the AWS CLI with downloadable DynamoDB ............................................................... 64 Using the API.......................................................................................................................... 65 Using the NoSQL workbench ..................................................................................................... 65 IP address ranges..................................................................................................................... 65 Getting started with DynamoDB........................................................................................................ 66 Basic concepts......................................................................................................................... 66 Prerequisites............................................................................................................................ 66 Step 1: Create a table ............................................................................................................... 66 Step 2: Write data .................................................................................................................... 70 Step 3: Read data.................................................................................................................... 74 Step 4: Update data................................................................................................................. 76 Step 5: Query data ................................................................................................................... 78 API Version 2012-08-10 iii Amazon DynamoDB Developer Guide Step 6: Create a global secondary index ...................................................................................... 81 Step 7: Query the global secondary index ................................................................................... 84 Step 8: (Optional) clean up........................................................................................................ 87 Next steps............................................................................................................................... 87 Getting started with DynamoDB and the AWS SDKs ............................................................................. 88 Create a table.......................................................................................................................... 88 Create a DynamoDB table using an AWS SDK ...................................................................... 88 Write an item.......................................................................................................................... 99 Write an item to a DynamoDB table using an AWS SDK ......................................................... 99 Read an item ......................................................................................................................... 109 Read an item from a DynamoDB table using an AWS SDK .................................................... 109 Update an item...................................................................................................................... 118 Update an item in a DynamoDB table using an AWS SDK ..................................................... 118 Delete an item....................................................................................................................... 128 Delete an item in a DynamoDB table using an AWS SDK ...................................................... 128 Query a table ......................................................................................................................... 138 Query a DynamoDB table using an AWS SDK ..................................................................... 138 Scan a table.......................................................................................................................... 150 Scan a DynamoDB table using an AWS SDK ....................................................................... 138 Working with AWS SDKs ......................................................................................................... 159 Programming with DynamoDB......................................................................................................... 161 Overview of AWS SDK support for DynamoDB ........................................................................... 161 Programmatic interfaces.......................................................................................................... 163 Low-level interfaces........................................................................................................ 164 Document interfaces....................................................................................................... 165 Object persistence interface ............................................................................................. 165 Low-level API......................................................................................................................... 167 Request format.............................................................................................................. 169 Response format............................................................................................................ 169 Data type descriptors ...................................................................................................... 170 Numeric data................................................................................................................. 171 Binary data.................................................................................................................... 171 Error handling........................................................................................................................ 171 Error components........................................................................................................... 171 Transactional errors........................................................................................................ 172 Error messages and codes ............................................................................................... 172 Error handling in your application .................................................................................... 175 Error retries and exponential backoff ................................................................................ 176 Batch operations and error handling ................................................................................. 176 Higher-level programming interfaces ........................................................................................ 177 Java: DynamoDBMapper.................................................................................................. 178 .NET: Document model ................................................................................................... 225 .NET: Object persistence model ........................................................................................ 248 Running the code examples ..................................................................................................... 276 Load sample data........................................................................................................... 277 Java code examples ........................................................................................................ 282 .NET code examples ........................................................................................................ 284 Working with DynamoDB................................................................................................................ 287 Working with Tables ............................................................................................................... 287 Basic Operations on Tables .............................................................................................. 287 Considerations when changing read/write Capacity Mode .................................................... 293 Considerations when choosing a table class....................................................................... 293 Provisioned Capacity Tables ............................................................................................. 294 Item Sizes and Formats ................................................................................................... 300 Managing throughput capacity with auto scaling ................................................................ 301 Working with Global Tables ............................................................................................. 315 Tagging resources........................................................................................................... 339 API Version 2012-08-10 iv Amazon DynamoDB Developer Guide Working with tables: Java ............................................................................................... 342 Working with tables: .NET ................................................................................................ 347 Working with read and write operations .................................................................................... 354 DynamoDB API............................................................................................................... 354 PartiQL query language .................................................................................................. 476 Working with indexes .............................................................................................................. 507 Global secondary indexes ................................................................................................ 510 Local Secondary Indexes .................................................................................................. 550 Working with transactions ....................................................................................................... 588 How it works ................................................................................................................. 588 Using IAM with transactions ............................................................................................ 594 Example code................................................................................................................. 596 Working with streams ............................................................................................................. 599 Options......................................................................................................................... 599 Working with Kinesis Data Streams ................................................................................... 600 Working with DynamoDB Streams .................................................................................... 609 Working with On-Demand backup and restore ........................................................................... 650 Using AWS Backup ......................................................................................................... 650 Using DynamoDB backups............................................................................................... 657 Working with point-in-time recovery ......................................................................................... 670 How it works ................................................................................................................. 670 Before you begin ............................................................................................................ 672 Restoring a table to a point in time .................................................................................. 672 In-memory acceleration with DAX ..................................................................................................... 677 Use cases for DAX .................................................................................................................. 678 DAX usage notes .................................................................................................................... 678 How it works ......................................................................................................................... 679 How DAX processes requests ............................................................................................ 680 Item cache..................................................................................................................... 681 Query cache................................................................................................................... 682 DAX cluster components ......................................................................................................... 682 Nodes........................................................................................................................... 682 Clusters......................................................................................................................... 683 Regions and availability zones .......................................................................................... 683 Parameter groups........................................................................................................... 684 Security groups.............................................................................................................. 684 Cluster ARN................................................................................................................... 684 Cluster endpoint............................................................................................................. 684 Node endpoints.............................................................................................................. 685 Subnet groups............................................................................................................... 685 Events........................................................................................................................... 685 Maintenance window...................................................................................................... 685 Creating a DAX cluster ............................................................................................................ 686 Creating an IAM service role for DAX to access DynamoDB ................................................... 686 Using the AWS CLI .......................................................................................................... 688 Using the console ........................................................................................................... 692 Consistency models................................................................................................................ 694 Consistency among DAX cluster nodes .............................................................................. 695 DAX item cache behavior ................................................................................................. 695 DAX query cache behavior ............................................................................................... 697 Strongly consistent and transactional reads ....................................................................... 698 Negative caching ............................................................................................................ 698 Strategies for writes....................................................................................................... 698 Developing with the DAX client ................................................................................................ 701 Tutorial: Running a sample application .............................................................................. 701 Modifying an existing application to use DAX ..................................................................... 733 Managing DAX clusters ............................................................................................................ 734 API Version 2012-08-10 v Amazon DynamoDB Developer Guide IAM permissions for managing a DAX cluster ..................................................................... 734 Scaling a DAX cluster ...................................................................................................... 736 Customizing DAX cluster settings ..................................................................................... 737 Configuring TTL settings ................................................................................................. 737 Tagging support for DAX ................................................................................................. 738 AWS CloudTrail integration .............................................................................................. 739 Deleting a DAX cluster .................................................................................................... 739 Monitoring DAX...................................................................................................................... 740 Monitoring tools............................................................................................................. 740 Monitoring with CloudWatch ............................................................................................ 741 Logging DAX operations using AWS CloudTrail ................................................................... 755 DAX T3/T2 burstable instances ................................................................................................ 755 DAX T2 instance family ................................................................................................... 755 DAX T3 instance family ................................................................................................... 755 DAX access control ................................................................................................................. 756 IAM service role for DAX ................................................................................................. 756 IAM policy to allow DAX cluster access .............................................................................. 757 Case study: Accessing DynamoDB and DAX ........................................................................ 758 Access to DynamoDB, but no access with DAX .................................................................... 759 Access to DynamoDB and to DAX ..................................................................................... 761 Access to DynamoDB via DAX, but no direct access to DynamoDB ......................................... 764 DAX encryption at rest ............................................................................................................ 766 Enabling encryption at rest using the AWS Management Console .......................................... 767 DAX encryption in transit ........................................................................................................ 768 Using service-linked roles for DAX ............................................................................................ 769 Service-linked role permissions for DAX ............................................................................. 769 Creating a service-linked role for DAX ............................................................................... 770 Editing a service-linked role for DAX ................................................................................. 770 Deleting a service-linked role for DAX ............................................................................... 770 Accessing DAX across AWS accounts ......................................................................................... 771 Set up IAM.................................................................................................................... 772 Set up a VPC................................................................................................................. 773 Modify the DAX client to allow cross-account access ........................................................... 775 DAX cluster sizing guide .......................................................................................................... 778 Overview....................................................................................................................... 778 Estimating traffic............................................................................................................ 779 Load testing.................................................................................................................. 779 API reference......................................................................................................................... 780 NoSQL Workbench.......................................................................................................................... 781 Download.............................................................................................................................. 781 Install.................................................................................................................................... 782 Data modeler......................................................................................................................... 785 Creating a new model ..................................................................................................... 785 Importing an existing model ............................................................................................ 793 Exporting a model .......................................................................................................... 796 Editing an existing model ................................................................................................ 798 Data visualizer........................................................................................................................ 800 Adding sample data ........................................................................................................ 801 Importing from CSV ........................................................................................................ 802 Facets........................................................................................................................... 803 Aggregate view.............................................................................................................. 807 Committing a data model ................................................................................................ 808 Operation builder................................................................................................................... 813 Connecting to datasets ................................................................................................... 813 Building operations......................................................................................................... 817 Exporting to CSV ............................................................................................................ 834 Sample data models............................................................................................................... 835 API Version 2012-08-10 vi Amazon DynamoDB Developer Guide Employee data model ..................................................................................................... 835 Discussion forum data model.......................................................................................... 835 Music library data model ................................................................................................. 836 Ski resort data model ..................................................................................................... 836 Credit card offers data model .......................................................................................... 836 Bookmarks data model................................................................................................... 837 Release history....................................................................................................................... 837 Code examples............................................................................................................................... 839 Actions.................................................................................................................................. 840 Create a table ................................................................................................................ 840 Delete a table................................................................................................................ 851 Delete an item from a table ............................................................................................ 857 Get a batch of items....................................................................................................... 866 Get an item from a table ................................................................................................ 873 Get information about a table......................................................................................... 882 List tables...................................................................................................................... 888 Put an item in a table..................................................................................................... 894 Query a table ................................................................................................................. 904 Run a PartiQL statement ................................................................................................. 916 Run batches of PartiQL statements ................................................................................... 929 Scan a table.................................................................................................................. 945 Update an item in a table............................................................................................... 954 Write a batch of items .................................................................................................... 965 Scenarios............................................................................................................................... 976 Accelerate reads with DAX ............................................................................................... 976 Get started using tables, items, and queries ....................................................................... 981 Query a table by using batches of PartiQL statements ....................................................... 1049 Query a table using PartiQL........................................................................................... 1090 Use a document model................................................................................................. 1125 Use a high-level object persistence model ........................................................................ 1135 Cross-service examples.......................................................................................................... 1141 Build an app to submit data to a DynamoDB table............................................................ 1142 Create a REST API to track COVID-19 data ....................................................................... 1143 Create a messenger application ...................................................................................... 1144 Create a web application to track DynamoDB data ............................................................ 1144 Create a websocket chat application ............................................................................... 1146 Detect PPE in images.................................................................................................... 1146 Invoke a Lambda function from a browser ....................................................................... 1147 Save EXIF and other image information........................................................................... 1148 Use API Gateway to invoke a Lambda function ................................................................. 1149 Use Step Functions to invoke Lambda functions ............................................................... 1150 Use scheduled events to invoke a Lambda function ........................................................... 1151 Security....................................................................................................................................... 1152 Data protection.................................................................................................................... 1152 Encryption at rest ......................................................................................................... 1153 Data protection in DAX ................................................................................................. 1169 Internetwork traffic privacy............................................................................................ 1169 Identity and access management ............................................................................................ 1170 Identity and access management.................................................................................... 1171 Identity and access management in DAX ......................................................................... 1200 Compliance validation........................................................................................................... 1200 Resilience............................................................................................................................. 1201 Infrastructure security........................................................................................................... 1201 Using VPC endpoints..................................................................................................... 1202 Configuration and vulnerability analysis .................................................................................. 1208 Security best practices ........................................................................................................... 1208 Preventative security best practices................................................................................. 1208 API Version 2012-08-10 vii Amazon DynamoDB Developer Guide Detective security best practices ..................................................................................... 1210 Monitoring................................................................................................................................... 1212 Logging and monitoring in DynamoDB.................................................................................... 1212 Monitoring tools........................................................................................................... 1213 Monitoring with Amazon CloudWatch............................................................................. 1214 Logging DynamoDB operations by using AWS CloudTrail ................................................... 1233 Logging and monitoring in DAX ............................................................................................. 1249 Contributor Insights.............................................................................................................. 1249 How it works................................................................................................................ 1249 Getting started............................................................................................................. 1253 Using IAM.................................................................................................................... 1257 Best practices............................................................................................................................... 1262 NoSQL design...................................................................................................................... 1264 NoSQL vs. RDBMS ......................................................................................................... 1264 Two key concepts......................................................................................................... 1265 General approach......................................................................................................... 1265 Partition key design .............................................................................................................. 1266 Burst capacity.............................................................................................................. 1266 Adaptive capacity......................................................................................................... 1266 Distributing workloads.................................................................................................. 1268 Write sharding .............................................................................................................. 1268 Uploading data efficiently.............................................................................................. 1269 Sort key design .................................................................................................................... 1270 Version control............................................................................................................. 1271 Secondary indexes................................................................................................................ 1272 General guidelines........................................................................................................ 1272 Sparse indexes............................................................................................................. 1274 Aggregation................................................................................................................. 1275 GSI overloading............................................................................................................ 1276 GSI sharding................................................................................................................ 1277 Creating a replica ......................................................................................................... 1278 Large items.......................................................................................................................... 1279 Compression................................................................................................................ 1279 Using Amazon S3......................................................................................................... 1279 Time series data................................................................................................................... 1280 Design pattern for time series data ................................................................................. 1280 Time series table examples............................................................................................ 1280 Many-to-many relationships................................................................................................... 1281 Adjacency lists.............................................................................................................. 1281 Materialized graphs....................................................................................................... 1282 Hybrid DynamoDB–RDBMS..................................................................................................... 1285 Not migrating.............................................................................................................. 1285 Hybrid system implementation....................................................................................... 1286 Relational modeling .............................................................................................................. 1286 First steps.................................................................................................................... 1288 Example...................................................................................................................... 1289 Querying and scanning .......................................................................................................... 1292 Scan performance......................................................................................................... 1292 Avoid spikes................................................................................................................. 1292 Parallel scans............................................................................................................... 1295 Cost optimization................................................................................................................. 1296 Evaluate your costs at the table level .............................................................................. 1296 Evaluate your table capacity mode ................................................................................. 1297 Evaluate your table's auto scaling settings ....................................................................... 1301 Evaluate your table class selection .................................................................................. 1307 Identifying your unused resources ................................................................................... 1308 Evaluate your table usage patterns ................................................................................. 1314 API Version 2012-08-10 viii Amazon DynamoDB Developer Guide Evaluate your Streams usage......................................................................................... 1316 Evaluate your provisioned capacity for right-sized provisioning ........................................... 1319 Integrating with other AWS services ................................................................................................ 1329 Integrating with Amazon Cognito ........................................................................................... 1329 Integrating with Amazon Redshift ........................................................................................... 1331 Integrating with Amazon EMR ................................................................................................ 1332 Overview..................................................................................................................... 1332 Tutorial: Working with Amazon DynamoDB and Apache Hive .............................................. 1333 Creating an external table in Hive................................................................................... 1339 Processing HiveQL statements ........................................................................................ 1341 Querying data in DynamoDB .......................................................................................... 1342 Copying data to and from Amazon DynamoDB ................................................................. 1343 Performance tuning ...................................................................................................... 1353 Integrating with S3............................................................................................................... 1356 Import from Amazon S3 ................................................................................................ 1357 Export to Amazon S3 .................................................................................................... 1367 Quotas and limits......................................................................................................................... 1376 Read/write capacity mode and throughput .............................................................................. 1376 Capacity unit sizes (for provisioned tables)....................................................................... 1376 Request unit sizes (for on-demand tables) ....................................................................... 1377 Throughput default quotas............................................................................................ 1377 Increasing or decreasing throughput (for provisioned tables) .............................................. 1378 Tables................................................................................................................................. 1379 Table size.................................................................................................................... 1379 Maximum number of tables per account per region ........................................................... 1379 Global tables........................................................................................................................ 1379 Secondary indexes................................................................................................................ 1380 Secondary indexes per table.......................................................................................... 1380 Projected Secondary Index attributes per table ................................................................. 1380 Partition keys and sort keys ................................................................................................... 1380 Partition key length...................................................................................................... 1380 Partition key values...................................................................................................... 1380 Sort key length ............................................................................................................ 1380 Sort key values............................................................................................................. 1380 Naming rules........................................................................................................................ 1381 Table names and Secondary Index names ........................................................................ 1381 Attribute names........................................................................................................... 1381 Data types........................................................................................................................... 1381 String.......................................................................................................................... 1381 Number....................................................................................................................... 1381 Binary......................................................................................................................... 1382 Items................................................................................................................................... 1382 Item size...................................................................................................................... 1382 Item size for tables with Local Secondary Indexes ............................................................. 1382 Attributes............................................................................................................................ 1382 Attribute name-value pairs per item............................................................................... 1382 Number of values in list, map, or set............................................................................... 1382 Attribute values............................................................................................................ 1382 Nested attribute depth .................................................................................................. 1383 Expression parameters........................................................................................................... 1383 Lengths....................................................................................................................... 1383 Operators and operands................................................................................................ 1383 Reserved words............................................................................................................ 1383 DynamoDB transactions......................................................................................................... 1383 DynamoDB Streams.............................................................................................................. 1384 Simultaneous readers of a shard in DynamoDB Streams ..................................................... 1384 Maximum write capacity for a table with DynamoDB Streams enabled ................................. 1384 API Version 2012-08-10 ix Amazon DynamoDB Developer Guide DynamoDB Accelerator (DAX) ................................................................................................. 1384 AWS region availability .................................................................................................. 1384 Nodes.......................................................................................................................... 1385 Parameter groups......................................................................................................... 1385 Subnet groups.............................................................................................................. 1385 API-specific limits................................................................................................................. 1385 DynamoDB encryption at rest ................................................................................................ 1386 Table export to Amazon S3 ................................................................................................... 1386 Backup and restore ............................................................................................................... 1386 API reference............................................................................................................................... 1387 Appendix..................................................................................................................................... 1388 Troubleshooting SSL/TLS connection establishment issues ......................................................... 1388 Testing your application or service .................................................................................. 1388 Testing your client browser ............................................................................................ 1389 Updating your software application client ........................................................................ 1389 Updating your client browser ......................................................................................... 1389 Manually updating your certificate bundle ....................................................................... 1389 Example tables and data....................................................................................................... 1390 Sample data files.......................................................................................................... 1390 Creating example tables and uploading data ............................................................................ 1399 Creating example tables and uploading data - Java ........................................................... 1399 Creating example tables and uploading data - .NET ........................................................... 1406 Example application using AWS SDK for Python (Boto3) ............................................................ 1415 Step 1: Deploy and test locally ....................................................................................... 1415 Step 2: Examine the data model and implementation details.............................................. 1419 Step 3: Deploy in production ......................................................................................... 1427 Step 4: Clean up resources............................................................................................. 1433 Integrating with AWS Data Pipeline ........................................................................................ 1434 Prerequisites to export and import data .......................................................................... 1436 Exporting data from DynamoDB to Amazon S3 ................................................................ 1440 Importing data from Amazon S3 to DynamoDB ................................................................ 1441 Troubleshooting............................................................................................................ 1442 Predefined templates for AWS Data Pipeline and DynamoDB .............................................. 1443 Amazon DynamoDB Storage Backend for Titan ......................................................................... 1443 Reserved words in DynamoDB ................................................................................................ 1444 Legacy conditional parameters ............................................................................................... 1452 AttributesToGet............................................................................................................ 1453 AttributeUpdates.......................................................................................................... 1454 ConditionalOperator...................................................................................................... 1456 Expected...................................................................................................................... 1456 KeyConditions.............................................................................................................. 1460 QueryFilter................................................................................................................... 1462 ScanFilter..................................................................................................................... 1463 Writing conditions with legacy parameters ....................................................................... 1464 Previous low-level API version (2011-12-05) ............................................................................ 1470 BatchGetItem............................................................................................................... 1470 BatchWriteItem............................................................................................................. 1475 CreateTable.................................................................................................................. 1480 DeleteItem................................................................................................................... 1485 DeleteTable.................................................................................................................. 1489 DescribeTables.............................................................................................................. 1492 GetItem....................................................................................................................... 1495 ListTables..................................................................................................................... 1498 PutItem....................................................................................................................... 1500 Query.......................................................................................................................... 1504 Scan............................................................................................................................ 1513 UpdateItem.................................................................................................................. 1523 API Version 2012-08-10 x