AWS Import/Export Developer Guide API Version 2010-06-03 AWS Import/Export Developer Guide AWS Import/Export: Developer Guide Copyright © 2014 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. The following are trademarks of Amazon Web Services, Inc.: Amazon, Amazon Web Services Design, AWS, Amazon CloudFront, Cloudfront, Amazon DevPay, DynamoDB, ElastiCache, Amazon EC2, Amazon Elastic Compute Cloud, Amazon Glacier, Kindle, Kindle Fire, AWS Marketplace Design, Mechanical Turk, Amazon Redshift, Amazon Route 53, Amazon S3, Amazon VPC. In addition, Amazon.com graphics, logos, page headers, button icons, scripts, and service names are trademarks, or trade dress of Amazon in the U.S. and/or other countries. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. AWS Import/Export Developer Guide Table of Contents Welcome ..................................................................................................................................... 1 What Is AWS Import/Export?................................................................................................... 1 Overview of AWS Import/Export....................................................................................... 1 Advantages to AWS Import/Export ................................................................................... 2 Available Tools, Libraries, and Interfaces............................................................................ 2 AWS Import/Export Jobs ........................................................................................................ 2 Job ............................................................................................................................. 2 Job Type ...................................................................................................................... 3 Related Concepts.......................................................................................................... 4 Pricing................................................................................................................................. 4 Return Shipping ............................................................................................................ 5 Next Step............................................................................................................................. 5 Getting Started ............................................................................................................................. 6 Before You Use AWS Import/Export.......................................................................................... 6 Sign Up for AWS................................................................................................................... 7 Download the AWS Import/Export Web Service Tool.................................................................... 7 Save Your Credentials to a File................................................................................................ 8 Create an Import Job............................................................................................................. 9 Create Your First Amazon S3 Import Job........................................................................... 9 Create Your First Amazon EBS Import Job....................................................................... 13 Create Your First Amazon Glacier Import Job.................................................................... 17 Create an Export Job ........................................................................................................... 21 Create Your First Amazon S3 Export Job......................................................................... 21 Where Do I Go from Here?.................................................................................................... 25 Working with Import Jobs.............................................................................................. 25 Working with Export Jobs.............................................................................................. 25 Managing Your Jobs..................................................................................................... 25 Pricing Information....................................................................................................... 25 Using IAM with AWS Import/Export................................................................................................. 26 Controlling Access to Jobs .................................................................................................... 26 Example IAM User Policies for AWS Import/Export............................................................ 27 Granting Access to AWS Resources....................................................................................... 28 Working with Import Jobs.............................................................................................................. 30 Creating Amazon S3 Import Jobs........................................................................................... 30 Import to Amazon S3 Process........................................................................................ 30 Creating Amazon EBS Import Jobs......................................................................................... 32 Import to Amazon EBS Process..................................................................................... 33 Creating Amazon Glacier Import Jobs..................................................................................... 36 Import to Amazon Glacier Process.................................................................................. 36 Creating Import Manifests ..................................................................................................... 37 Import Manifest Examples............................................................................................. 38 Configuring Common Import Manifest Options.................................................................. 39 Working with Amazon S3 Import Manifests....................................................................... 40 Sending CreateJob Requests ................................................................................................ 46 Sending a CreateJob Request Using the AWS Import/Export Web Service Tool...................... 46 Sending a CreateJob Request Using the AWS SDK for Java............................................... 48 Sending a CreateJob Request Using the REST API........................................................... 51 Viewing Import Logs ............................................................................................................ 52 Log Files .................................................................................................................... 52 Amazon S3 and Amazon EBS Log File Fields................................................................... 53 Amazon Glacier Log File............................................................................................... 54 Status Codes .............................................................................................................. 54 Working with Export Jobs.............................................................................................................. 55 Creating Amazon S3 Export Jobs........................................................................................... 55 Amazon S3 Export Process........................................................................................... 56 API Version 2010-06-03 iii AWS Import/Export Developer Guide Creating Export Manifests .................................................................................................... 57 Export Manifest Example .............................................................................................. 57 Working with Amazon S3 Export Manifests....................................................................... 58 Managing File Exports.................................................................................................. 59 Collecting Files That Have Naming Errors........................................................................ 61 Viewing Import/Export Logs................................................................................................... 61 Sample Log File .......................................................................................................... 62 Shipping Your Storage Device........................................................................................................ 63 Storage Device Requirements................................................................................................ 63 Preparing Your Storage Device for Amazon S3 Import and Export Jobs......................................... 64 Copying the Signature Value.......................................................................................... 64 Filling Out the Packing Slip............................................................................................ 65 Preparing Your Storage Device for Amazon EBS and Amazon Glacier Import Jobs.......................... 65 Using TrueCrypt Encryption................................................................................................... 66 Encryption for Import to Amazon S3................................................................................ 66 Encryption with Export from Amazon S3.......................................................................... 67 Packing Your Storage Device ................................................................................................. 68 AWS Shipping Addresses ..................................................................................................... 68 Shipping Multiple Devices ..................................................................................................... 68 Mailing Your Storage Device .................................................................................................. 69 Managing Your Jobs..................................................................................................................... 71 Listing Your Jobs.................................................................................................................. 71 Updating a Job.................................................................................................................... 72 Getting Shipping Information.................................................................................................. 73 Getting the Status of Your Jobs.............................................................................................. 73 Canceling a Job .................................................................................................................. 74 Failed Jobs......................................................................................................................... 75 Manifest File Options Reference..................................................................................................... 76 Common Manifest File Options .............................................................................................. 76 Customs Manifest File Option ................................................................................................ 79 Import to Amazon S3 Manifest File Options.............................................................................. 84 Import to Amazon EBS Manifest File Options............................................................................ 91 Import to Amazon Glacier Manifest File Options........................................................................ 94 Export from Amazon S3 Manifest File Options.......................................................................... 97 API Reference........................................................................................................................... 103 Making Query Requests...................................................................................................... 103 Endpoints ................................................................................................................. 103 Structure of a POST Request....................................................................................... 103 Sample POST Request and Response.......................................................................... 104 Query Authentication .................................................................................................. 105 Actions............................................................................................................................. 106 CancelJob ................................................................................................................ 107 CreateJob................................................................................................................. 109 GetStatus ................................................................................................................. 113 ListJobs ................................................................................................................... 118 UpdateJob ................................................................................................................ 122 Common Query Parameters ................................................................................................ 124 Error and Status Codes....................................................................................................... 125 Error Response ......................................................................................................... 125 Error Codes .............................................................................................................. 126 Job and Device Status Codes...................................................................................... 127 AWS Import/Export Log Status Codes........................................................................... 128 Guidelines and Limitations........................................................................................................... 130 Document History ...................................................................................................................... 132 Appendices............................................................................................................................... 135 Appendix A: Other Tools and Services................................................................................... 135 Appendix B: Examples........................................................................................................ 135 Import Examples........................................................................................................ 135 API Version 2010-06-03 iv AWS Import/Export Developer Guide Export Examples ....................................................................................................... 140 Appendix C: File Extension to Mime Types.............................................................................. 143 Appendix D: AWS Import/Export Resources ........................................................................... 149 API Version 2010-06-03 v AWS Import/Export Developer Guide What Is AWS Import/Export? Welcome Topics • What Is AWS Import/Export? (p.1) • AWS Import/Export Jobs (p.2) • Pricing (p.4) • Next Step (p.5) This guide contains information for you to create import and export jobs quickly, and includes guidelines for shipping your storage devices to AWS.This guide also contains conceptual and reference information about manifest options, which allow you to configure your import and export jobs to meet your specific needs. What Is AWS Import/Export? Topics • Overview of AWS Import/Export (p.1) • Advantages to AWS Import/Export (p.2) • Available Tools, Libraries, and Interfaces (p.2) Overview of AWS Import/Export AWS Import/Export accelerates transferring large amounts of data between the AWS cloud and portable storage devices that you mail to us. AWS transfers data directly onto and off of your storage devices using Amazon’s high-speed internal network.Your data load typically begins the next business day after your storage device arrives at AWS. After the data export or import completes, we return your storage device. For large data sets, AWS Import/Export can be significantly faster than Internet transfer and more cost effective than upgrading your connectivity. AWS Import/Export supports: • Import to Amazon S3 • Export from Amazon S3 • Import to Amazon EBS API Version 2010-06-03 1 AWS Import/Export Developer Guide Advantages to AWS Import/Export • Import to Amazon Glacier AWS Import/Export does not currently support export from Amazon EBS or Amazon Glacier. Important This guide assumes that you are an Amazon S3 user, an Amazon Glacier user, or an Amazon Elastic Compute Cloud (EC2) user for Amazon EBS data. If that is not the case, go to the Amazon S3 Getting Started Guide, Amazon Glacier Developer Guide, or Amazon EC2 Getting Started Guide for Amazon EBS data. Advantages to AWS Import/Export Using AWS Import/Export provides the following major advantages: • Migrate Large Amounts of Data into the AWS Cloud—Mail us one or more storage devices with any amount of data.We load your data to the AWS cloud and return your storage device. • Content Distribution—We can export data from the AWS cloud onto one or more of your storage devices for delivery to your customers. • Direct Data Interchange—If you regularly receive content on portable storage devices from your business associates, you can have them send it directly to AWS for import into Amazon EBS, Amazon S3, or Amazon Glacier. • Off-Site Backup—Send full or incremental backups to Amazon S3 for reliable and redundant off-site storage. • Disaster Recovery—In the event you need to quickly retrieve a large backup stored in Amazon S3, use AWS Import/Export to transfer the data to a portable storage device and deliver it to your site. Available Tools, Libraries, and Interfaces AWS Import/Export provides a command line tool for creating import and export jobs, without writing any code. For applications requiring programming interface, AWS SDK provides Java, .NET and PHP libraries to create and manage import and export jobs.You can also optionally use the REST interface when working with AWS Import/Export. There are also third party tools that support AWS Import/Export. For more information, go to http://aws.amazon.com/importexport/tools. AWS Import/Export Jobs Topics • Job (p.2) • Job Type (p.3) • Related Concepts (p.4) This section describes AWS Import/Export jobs and job types. Job You create an AWS Import/Export job in two steps. First, you submit a job request to AWS. Each AWS Import/Export job corresponds to exactly one storage device.You submit your job request by using the AWS Import/Export command line tool, which requires no programming. If your application requires a programmatic interface, you can use the AWS SDK or the REST API. Second, you send your storage API Version 2010-06-03 2 AWS Import/Export Developer Guide Job Type device to AWS.We use our high bandwidth network to import your data into AWS or export data from AWS to your storage device, then return your device. You must submit a separate job request for each device. Note You can send multiple devices in the same shipment. If you do, however, there are specific guidelines and limitations that govern what devices you can ship and how your devices must be packaged. If your shipment is not prepared and packed correctly, AWS Import/Export cannot process your jobs. Regardless of how many devices you ship at one time, you must submit a separate job request for each device. For complete details about packaging requirements when shipping multiple devices, see Shipping Multiple Devices (p.68). Job Type For each job type, the following table summarizes the source of the data, either your device or an Amazon S3 bucket, where the data will be moved to, and the result you can expect when the job is complete. Import to Amazon S3 Source Target Result • Files on a device file system. • Objects in an existing • One object for each file. • One device per import job. Amazon S3 bucket. • Files and directories can be • We recommend encrypting • One bucket per import job. excluded or renamed using your data using TrueCrypt • If you encrypted your data manifest file options. open-source encryption using TrueCrypt, AWS • AWS erases your device after software. See Using decrypts the data before every import job prior to shipping. TrueCrypt Encryption (p.66). performing the import. Export from Amazon S3 Source Target Result • Objects in one or more • Files on your storage device. • One file for each object. Amazon S3 buckets. • One device per export job. • You can export from more than • You can export from multiple • AWS formats your device. one bucket and specify which files buckets for each export job. to export using manifest file • AWS copies your data to an • You must provide a encrypted file container on options. password that AWS will use your device. See Using • You will need use TrueCrypt to to encrypt your data. TrueCrypt Encryption (p.66). access the encrypted files. Import to Amazon Glacier Source Target Result • Entire device. • One archive in an existing • A device image stored as a single • One device per import job. Amazon Glacier vault. archive. • We recommend encrypting • One archive per import job. • The device's file system is never your device. • AWS does not decrypt your mounted. device. • AWS erases your device after every import job prior to shipping. API Version 2010-06-03 3 AWS Import/Export Developer Guide Related Concepts Import to Amazon EBS (Device capacity less than or equal to 1 TB) Important An import to Amazon EBS will have different results depending on whether the capacity of your storage device is less than or equal to 1 TB or greater than 1 TB.The maximum size of an Amazon EBS snapshot is 1 TB, so if the device image is larger than 1 TB, the image is chunked and stored on Amazon S3.The target location is determined based on the total capacity of the device, not the amount of data on the device. Source Target Result • Entire device (device • One Amazon EBS snapshot. • A device image stored as a single capacity less than or equal • One snapshot per import job. snapshot. to 1 TB). • AWS does not decrypt your • The device's file system is never • One device per import job. device. mounted. • We recommend encrypting • If the device was encrypted, the your device. device image is encrypted. • AWS erases your device after every import job prior to shipping. Import to Amazon EBS (Device capacity greater than 1 TB) Source Target Result • Entire device (device • Multiple objects in an existing • A device image chunked as a capacity greater than 1 TB). Amazon S3 bucket. series of 1 TB snapshots that are • One device per import job. • One device image, multiple stored as objects in the Amazon • We recommend encrypting Amazon S3 objects per S3 bucket specified in the your data. import job. logBucket manifest file option. • AWS does not decrypt your • The device's file system is never data. mounted. • If the device was encrypted, the device image is encrypted. • AWS erases your device after every import job prior to shipping. Related Concepts • To learn more about Amazon S3, see Introduction to Amazon S3. • To learn more about Amazon EBS, go to Amazon Elastic Block Store. • To learn more about Amazon Glacier, go to Amazon Glacier. • If you need assistance with your AWS Import/Export job, contact AWS Support Center. Pricing You are not charged for the time required to erase your device following an import, and you are not charged for the time required to decrypt or encrypt your device.You can approximate your costs by using the AWS Import/Export Calculator. API Version 2010-06-03 4 AWS Import/Export Developer Guide Return Shipping Return Shipping Return shipping charges are dependent on the location of your AWS storage and your return shipping destination. Any applicable return shipping expenses will be charged once your package is ready for shipment to your return shipping address.You can estimate your return shipping costs by using the AWS Import/Export Calculator. Next Step Your next step is to get set up and to create your first job. For more information, see Getting Started (p.6). API Version 2010-06-03 5