Amazon Simple Storage Service Developer Guide API Version 2006-03-01 Amazon Simple Storage Service Developer Guide Amazon Simple Storage Service: Developer Guide Copyright © 2014 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. The following are trademarks of Amazon Web Services, Inc.: Amazon, Amazon Web Services Design, AWS, Amazon CloudFront, Cloudfront, Amazon DevPay, DynamoDB, ElastiCache, Amazon EC2, Amazon Elastic Compute Cloud, Amazon Glacier, Kindle, Kindle Fire, AWS Marketplace Design, Mechanical Turk, Amazon Redshift, Amazon Route 53, Amazon S3, Amazon VPC. In addition, Amazon.com graphics, logos, page headers, button icons, scripts, and service names are trademarks, or trade dress of Amazon in the U.S. and/or other countries. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Amazon Simple Storage Service Developer Guide Table of Contents What Is Amazon S3? ..................................................................................................................... 1 How Do I...? ......................................................................................................................... 1 Introduction .................................................................................................................................. 2 Overview of Amazon S3 and This Guide.................................................................................... 2 Advantages to Amazon S3...................................................................................................... 2 Amazon S3 Concepts ............................................................................................................ 3 Buckets ....................................................................................................................... 3 Objects........................................................................................................................ 3 Keys ........................................................................................................................... 4 Regions ....................................................................................................................... 4 Amazon S3 Data Consistency Model................................................................................ 5 Features .............................................................................................................................. 7 Reduced Redundancy Storage ........................................................................................ 7 Bucket Policies.............................................................................................................. 7 AWS Identity and Access Management............................................................................. 8 Access Control Lists ...................................................................................................... 8 Versioning .................................................................................................................... 8 Operations ................................................................................................................... 8 Amazon S3 Application Programming Interfaces (API)................................................................. 9 The REST Interface....................................................................................................... 9 The SOAP Interface....................................................................................................... 9 Paying for Amazon S3............................................................................................................ 9 Related Services ................................................................................................................ 10 Making Requests......................................................................................................................... 11 About Access Keys .............................................................................................................. 11 AWS Account Access Keys ........................................................................................... 11 IAM User Access Keys ................................................................................................. 12 Temporary Security Credentials...................................................................................... 12 Request Endpoints .............................................................................................................. 13 Making Requests Using the AWS SDKs................................................................................... 14 Using AWS Account or IAM User Credentials................................................................... 15 Using IAM User Temporary Credentials........................................................................... 20 Using Federated User Temporary Credentials................................................................... 31 Making Requests Using the REST API.................................................................................... 47 Virtual Hosting of Buckets ............................................................................................. 48 Request Redirection and the REST API........................................................................... 52 Buckets...................................................................................................................................... 55 Restrictions and Limitations................................................................................................... 56 Rules for Naming......................................................................................................... 56 Configuration Options........................................................................................................... 57 Buckets and Regions.................................................................................................... 57 Bucket Website Configuration ................................................................................................ 59 Using the AWS Management Console............................................................................. 60 Using the SDK for Java................................................................................................. 60 Using the SDK for .NET................................................................................................ 63 Using the SDK for PHP................................................................................................. 66 Using the REST API..................................................................................................... 68 Requester Pays Buckets ....................................................................................................... 69 Configure Requester Pays by Using the Amazon S3 Console.............................................. 69 Configure Requester Pays with the REST API.................................................................. 70 DevPay and Requester Pays.......................................................................................... 72 Charge Details ............................................................................................................ 72 Buckets and Access Control.................................................................................................. 73 Billing and Reporting of Buckets............................................................................................. 73 Cost Allocation Tagging................................................................................................. 73 API Version 2006-03-01 iii Amazon Simple Storage Service Developer Guide Configuration Errors ............................................................................................................. 74 Objects ...................................................................................................................................... 75 Object Key and Metadata...................................................................................................... 76 Object Keys ................................................................................................................ 76 Object Metadata.......................................................................................................... 78 Subresources ..................................................................................................................... 80 Versioning .......................................................................................................................... 80 Lifecycle Management.......................................................................................................... 83 Overview.................................................................................................................... 84 Lifecycle Configuration Elements.................................................................................... 85 Object Archival............................................................................................................ 93 Object Expiration ......................................................................................................... 95 Specifying a Lifecycle Configuration................................................................................ 95 Cross-Origin Resource Sharing............................................................................................ 107 Cross-Origin Resource Sharing: Examples..................................................................... 108 How Do I Enable CORS on My Bucket?......................................................................... 108 How Does Amazon S3 Evaluate the CORS Configuration On a Bucket?.............................. 110 Using the AWS Management Console........................................................................... 111 Using the SDK for Java............................................................................................... 111 Using the SDK for .NET .............................................................................................. 116 Using the REST API................................................................................................... 122 Troubleshooting CORS Issues...................................................................................... 122 Operations on Objects ........................................................................................................ 122 Multipart Upload Overview........................................................................................... 123 Getting Objects.................................................................................................................. 127 Related Resources..................................................................................................... 127 Using the SDK for Java............................................................................................... 128 Using the SDK for .NET .............................................................................................. 131 Using the SDK for PHP............................................................................................... 134 Using the REST API................................................................................................... 136 Share an Object with Others........................................................................................ 136 Uploading Objects.............................................................................................................. 142 Related Resources..................................................................................................... 142 Uploading Objects in a Single Operation........................................................................ 143 Uploading Objects Using Multipart Upload API................................................................ 151 Uploading Objects Using Pre-Signed URLs.................................................................... 187 Copying Objects ................................................................................................................ 193 Related Resources..................................................................................................... 194 Copying Objects in a Single Operation........................................................................... 195 Copying Objects Using the Multipart Upload API............................................................. 204 Listing Object Keys............................................................................................................. 212 Iterating Through Multi-Page Results............................................................................. 213 Using a Prefix and Delimiter......................................................................................... 213 Using the SDK for Java............................................................................................... 214 Using the SDK for .NET .............................................................................................. 217 Using the SDK for PHP............................................................................................... 219 Using the REST API................................................................................................... 222 Related Resources..................................................................................................... 222 Deleting Objects ................................................................................................................ 222 Deleting Objects from a Version-Enabled Bucket............................................................. 222 Deleting Objects from an MFA-Enabled Bucket................................................................ 223 Related Resources..................................................................................................... 223 Deleting One Object Per Request................................................................................. 223 Deleting Multiple Objects Per Request........................................................................... 233 Restoring Objects .............................................................................................................. 255 Using the Console...................................................................................................... 256 Using the SDK for Java............................................................................................... 257 Using the SDK for .NET .............................................................................................. 259 API Version 2006-03-01 iv Amazon Simple Storage Service Developer Guide Using the REST API................................................................................................... 262 Managing Access ...................................................................................................................... 263 Introduction....................................................................................................................... 263 Overview .................................................................................................................. 264 How Amazon S3 Authorizes a Request.......................................................................... 269 Guidelines for Using the Available Access Policy Options.................................................. 274 Example Walkthroughs: Managing Access...................................................................... 277 Using Bucket Policies and User Policies................................................................................. 305 Access Policy Language Overview................................................................................ 305 Bucket Policy Examples .............................................................................................. 329 User Policy Examples................................................................................................. 335 Managing Access with ACLs ............................................................................................... 356 Access Control List (ACL) Overview.............................................................................. 356 Managing ACLs ......................................................................................................... 361 Data Protection.......................................................................................................................... 373 Data Encryption................................................................................................................. 373 Server-Side Encryption ............................................................................................... 374 Client-Side Encryption ................................................................................................ 397 Reduced Redundancy Storage............................................................................................. 409 Setting the Storage Class of an Object You Upload.......................................................... 410 Changing the Storage Class of an Object in Amazon S3................................................... 410 Versioning ........................................................................................................................ 411 Enabling a Bucket's Versioning State............................................................................. 412 Adding Objects to Versioning-Enabled Buckets................................................................ 417 Listing the Objects in a Versioning-Enabled Bucket.......................................................... 417 Retrieving Object Versions........................................................................................... 419 Deleting Object Versions ............................................................................................. 421 Transitioning Object Versions ....................................................................................... 426 Restoring Previous Versions ........................................................................................ 426 Versioned Object Permissions and ACLs........................................................................ 427 Working with Versioning-Suspended Buckets.................................................................. 428 Hosting a Static Website ............................................................................................................. 433 Website Endpoints ............................................................................................................. 434 Key Differences Between the Amazon Website and the REST API Endpoint......................... 435 Configure a Bucket for Website Hosting................................................................................. 436 Overview .................................................................................................................. 436 Syntax for Specifying Routing Rules.............................................................................. 438 Index Document Support ............................................................................................ 442 Custom Error Document Support ................................................................................. 443 Configuring a Redirect ................................................................................................ 445 Permissions Required for Website Access...................................................................... 447 Example Walkthroughs ....................................................................................................... 447 Example: Setting Up a Static Website............................................................................ 448 Example: Setting Up a Static Website Using a Custom Domain.......................................... 449 Setting Up Notification of Bucket Events......................................................................................... 457 Request Routing........................................................................................................................ 461 Request Redirection and the REST API................................................................................. 461 Overview .................................................................................................................. 461 DNS Routing............................................................................................................. 461 Temporary Request Redirection.................................................................................... 462 Permanent Request Redirection ................................................................................... 464 DNS Considerations........................................................................................................... 465 Performance Optimization ........................................................................................................... 466 Request Rate and Performance Considerations ...................................................................... 466 Workloads with a Mix of Request Types ......................................................................... 467 GET-Intensive Workloads ............................................................................................ 469 TCP Window Scaling.......................................................................................................... 469 TCP Selective Acknowledgement.......................................................................................... 470 API Version 2006-03-01 v Amazon Simple Storage Service Developer Guide BitTorrent.................................................................................................................................. 471 How You are Charged for BitTorrent Delivery........................................................................... 471 Using BitTorrent to Retrieve Objects Stored in Amazon S3........................................................ 472 Publishing Content Using Amazon S3 and BitTorrent................................................................ 473 Amazon DevPay ........................................................................................................................ 474 Amazon S3 Customer Data Isolation..................................................................................... 474 Example................................................................................................................... 475 Amazon DevPay Token Mechanism....................................................................................... 475 Amazon S3 and Amazon DevPay Authentication..................................................................... 475 Amazon S3 Bucket Limitation............................................................................................... 476 Amazon S3 and Amazon DevPay Process.............................................................................. 477 Additional Information ......................................................................................................... 477 Error Handling........................................................................................................................... 478 The REST Error Response.................................................................................................. 478 Response Headers .................................................................................................... 479 Error Response ......................................................................................................... 479 The SOAP Error Response.................................................................................................. 480 Amazon S3 Error Best Practices........................................................................................... 480 Retry InternalErrors.................................................................................................... 480 Tune Application for Repeated SlowDown errors.............................................................. 480 Isolate Errors ............................................................................................................ 481 Server Access Logging ............................................................................................................... 482 Overview .......................................................................................................................... 482 Log Object Key Format ............................................................................................... 483 How are Logs Delivered? ............................................................................................ 483 Best Effort Server Log Delivery..................................................................................... 483 Bucket Logging Status Changes Take Effect Over Time.................................................... 484 Related Topics................................................................................................................... 484 Enabling Logging Using Console .......................................................................................... 484 Enabling Logging Programmatically....................................................................................... 486 Enabling logging ........................................................................................................ 486 Granting the Log Delivery Group Write Permission........................................................... 486 Example: AWS SDK for .NET....................................................................................... 487 Log Format ....................................................................................................................... 489 Custom Access Log Information................................................................................... 491 Programming Considerations for Extensible Server Access Log Format............................... 491 Additional Logging for Copy Operations ......................................................................... 491 Deleting Log Files .............................................................................................................. 493 AWS SDKs and Explorers ........................................................................................................... 494 Specifying Signature Version in Request Authentication............................................................ 495 Using the AWS SDK for Java............................................................................................... 496 The Java API Organization.......................................................................................... 497 Testing the Java Code Examples.................................................................................. 497 Using the AWS SDK for .NET .............................................................................................. 498 The .NET API Organization.......................................................................................... 498 Testing the .NET Code Examples.................................................................................. 499 Using the AWS SDK for PHP and Running PHP Examples........................................................ 499 AWS SDK for PHP Levels............................................................................................ 499 Running PHP Examples.............................................................................................. 500 Related Resources..................................................................................................... 501 Using the AWS SDK for Ruby............................................................................................... 501 The Ruby API Organization ......................................................................................... 501 Testing the Ruby Script Examples................................................................................. 501 Using the AWS SDK for Python (Boto)................................................................................... 502 Appendices............................................................................................................................... 503 Appendix A: Using the SOAP API......................................................................................... 503 Common SOAP API Elements ..................................................................................... 503 Authenticating SOAP Requests .................................................................................... 504 API Version 2006-03-01 vi Amazon Simple Storage Service Developer Guide Setting Access Policy with SOAP.................................................................................. 505 Appendix B: Authenticating Requests (AWS Signature Version 2)............................................... 506 Authenticating Requests Using the REST API................................................................. 508 Signing and Authenticating REST Requests ................................................................... 509 Browser-Based Uploads Using POST............................................................................ 520 Resources ................................................................................................................................ 537 Document History ...................................................................................................................... 539 AWS Glossary........................................................................................................................... 546 API Version 2006-03-01 vii Amazon Simple Storage Service Developer Guide How Do I...? What Is Amazon S3? Amazon Simple Storage Service is storage for the Internet. It is designed to make web-scale computing easier for developers. Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites.The service aims to maximize benefits of scale and to pass those benefits on to developers. This guide explains the core concepts of Amazon S3, such as buckets and objects, and how to work with these resources using the Amazon S3 application programming interface (API). How Do I...? Information Relevant Sections General product overview and pricing Amazon Simple Storage Service (Amazon S3) Get a quick hands-on introduction to Amazon Simple Storage Service Getting Started Guide Amazon S3 Learn about Amazon S3 key terminology Introduction to Amazon S3 (p.2) and concepts How do I work with buckets? Working with Amazon S3 Buckets (p.55) How do I work with objects? Working with Amazon S3 Objects (p.75) How do I make requests? Making Requests (p.11) How do I manage access to my Managing Access Permissions to Your Amazon S3 resources? Resources (p.263) API Version 2006-03-01 1 Amazon Simple Storage Service Developer Guide Overview of Amazon S3 and This Guide Introduction to Amazon S3 This introduction to Amazon Simple Storage Service is intended to give you a detailed summary of this web service. After reading this section, you should have a good idea of what it offers and how it can fit in with your business. Topics • Overview of Amazon S3 and This Guide (p.2) • Advantages to Amazon S3 (p.2) • Amazon S3 Concepts (p.3) • Features (p.7) • Amazon S3 Application Programming Interfaces (API) (p.9) • Paying for Amazon S3 (p.9) • Related Services (p.10) Overview of Amazon S3 and This Guide Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. This guide describes how you send requests to create buckets, store and retrieve your objects, and manage permissions on your resources.The guide also describes access control and the authentication process. Access control defines who can access objects and buckets within Amazon S3, and the type of access (e.g., READ and WRITE).The authentication process verifies the identity of a user who is trying to access Amazon Web Services (AWS). Advantages to Amazon S3 Amazon S3 is intentionally built with a minimal feature set that focuses on simplicity and robustness. Following are some of advantages of the Amazon S3 service: • Create Buckets – Create and name a bucket that stores data. Buckets are the fundamental container in Amazon S3 for data storage. API Version 2006-03-01 2 Amazon Simple Storage Service Developer Guide Amazon S3 Concepts • Store data in Buckets – Store an infinite amount of data in a bucket. Upload as many objects as you like into an Amazon S3 bucket. Each object can contain up to 5 TB of data. Each object is stored and retrieved using a unique developer-assigned key. • Download data – Download your data or enable others to do so. Download your data any time you like or allow others to do the same. • Permissions – Grant or deny access to others who want to upload or download data into your Amazon S3 bucket. Grant upload and download permissions to three types of users. Authentication mechanisms can help keep data secure from unauthorized access. • Standard interfaces – Use standards-based REST and SOAP interfaces designed to work with any Internet-development toolkit. Note SOAP support over HTTP is deprecated, but it is still available over HTTPS. New Amazon S3 features will not be supported for SOAP.We recommend that you use either the REST API or the AWS SDKs. Amazon S3 Concepts Topics • Buckets (p.3) • Objects (p.3) • Keys (p.4) • Regions (p.4) • Amazon S3 Data Consistency Model (p.5) This section describes key concepts and terminology you need to understand to use Amazon S3 effectively. They are presented in the order you will most likely encounter them. Buckets A bucket is a container for objects stored in Amazon S3. Every object is contained in a bucket. For example, if the object named photos/puppy.jpg is stored in the johnsmith bucket, then it is addressable using the URL http://johnsmith.s3.amazonaws.com/photos/puppy.jpg Buckets serve several purposes: they organize the Amazon S3 namespace at the highest level, they identify the account responsible for storage and data transfer charges, they play a role in access control, and they serve as the unit of aggregation for usage reporting. You can configure buckets so that they are created in a specific Region. For more information, see Buckets and Regions (p.57).You can also configure a bucket so that every time an object is added to it, Amazon S3 generates a unique version ID and assigns it to the object. For more information, see Versioning (p.411). For more information about buckets, see Working with Amazon S3 Buckets (p.55). Objects Objects are the fundamental entities stored in Amazon S3. Objects consist of object data and metadata. The data portion is opaque to Amazon S3.The metadata is a set of name-value pairs that describe the object.These include some default metadata, such as the date last modified, and standard HTTP metadata, such as Content-Type.You can also specify custom metadata at the time the object is stored. API Version 2006-03-01 3