Amazon Rekognition Developer Guide Amazon Rekognition Developer Guide Amazon Rekognition: Developer Guide Copyright © 2022 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Amazon Rekognition Developer Guide Table of Contents What is Amazon Rekognition? ............................................................................................................. 1 Amazon Rekognition and HIPAA eligibility ..................................................................................... 3 Are you a first-time Amazon Rekognition user? .............................................................................. 3 How it works..................................................................................................................................... 4 Types of analysis........................................................................................................................ 4 Labels............................................................................................................................... 4 Custom labels.................................................................................................................... 5 Faces................................................................................................................................ 5 Face search........................................................................................................................ 5 People paths..................................................................................................................... 5 Personal Protective Equipment............................................................................................ 5 Celebrities......................................................................................................................... 6 Text detection................................................................................................................... 6 Inappropriate or offensive content....................................................................................... 6 Image and video operations........................................................................................................ 6 Amazon Rekognition Image operations ................................................................................. 6 Amazon Rekognition Video operations .................................................................................. 6 Non-storage and storage-based operations........................................................................... 7 Using the AWS SDK or HTTP to call Amazon Rekognition API operations ................................... 7 Non-storage and storage API operations....................................................................................... 7 Non-storage operations...................................................................................................... 7 Storage-based API operations.............................................................................................. 9 Model versioning ...................................................................................................................... 10 Getting started ................................................................................................................................ 11 Step 1: Set up an account ......................................................................................................... 11 Sign up for AWS .............................................................................................................. 11 Create an IAM user ........................................................................................................... 12 Next step........................................................................................................................ 13 Step 2: Set up the AWS CLI and AWS SDKs ................................................................................. 13 Next step........................................................................................................................ 14 Working with AWS SDKs ................................................................................................... 14 Step 3: Getting started using the AWS CLI and AWS SDK API ......................................................... 15 Formatting the AWS CLI examples ...................................................................................... 15 Next step........................................................................................................................ 15 Step 4: Getting started using the console .................................................................................... 15 Exercise 1: Detect objects and scenes (console) .................................................................... 16 Exercise 2: Analyze faces (console) ...................................................................................... 20 Exercise 3: Compare faces (console) .................................................................................... 23 Exercise 4: See aggregated metrics (console) ........................................................................ 25 Working with images and videos ........................................................................................................ 26 Working with images ................................................................................................................ 26 Image specifications......................................................................................................... 26 Analyzing images in an Amazon S3 bucket .......................................................................... 27 Using a local file system................................................................................................... 37 Displaying bounding boxes ................................................................................................ 46 Getting image orientation and bounding box coordinates ...................................................... 53 Working with stored video analysis ............................................................................................. 60 Types of analysis .............................................................................................................. 60 Amazon Rekognition Video API overview ............................................................................. 61 Calling Amazon Rekognition Video operations ..................................................................... 62 Configuring Amazon Rekognition Video .............................................................................. 66 Analyzing a stored video (SDK) .......................................................................................... 68 Analyzing a video (AWS CLI) .............................................................................................. 85 Reference: Video analysis results notification ....................................................................... 88 iii Amazon Rekognition Developer Guide Troubleshooting Amazon Rekognition Video ........................................................................ 89 Working with streaming video events ......................................................................................... 90 Overview of Amazon Rekognition Video stream processor operations ...................................... 91 Tagging the Amazon Rekognition Video stream processor ...................................................... 91 Error handling.......................................................................................................................... 93 Error components............................................................................................................. 93 Error messages and codes ................................................................................................. 93 Error handling in your application ...................................................................................... 97 Using Amazon Rekognition with FedRAMP .................................................................................. 98 Best practices for sensors, input images, and videos ............................................................................ 101 Amazon Rekognition Image operation latency ............................................................................ 101 Recommendations for facial comparison input images ................................................................ 101 Recommendations for camera setup (image and video) ............................................................... 102 Recommendations for camera setup (stored and streaming video) ................................................ 103 Recommendations for camera setup (streaming video) ................................................................ 103 Detecting labels............................................................................................................................. 105 Label Response Objects ........................................................................................................... 105 Bounding Boxes.............................................................................................................. 105 Confidence Score............................................................................................................ 106 Parents.......................................................................................................................... 106 Categories..................................................................................................................... 106 Aliases........................................................................................................................... 106 Image Properties............................................................................................................ 107 Model Version................................................................................................................ 108 Inclusion and Exclusion Filters .......................................................................................... 108 Detecting labels in an image .................................................................................................... 108 DetectLabels operation request ........................................................................................ 115 DetectLabels response..................................................................................................... 115 Transforming the DetectLabels response ........................................................................... 118 Detecting labels in a video ...................................................................................................... 121 StartLabelDetection Request............................................................................................ 121 GetLabelDetection Operation Response ............................................................................. 122 Transforming the GetLabelDetection Response ................................................................... 126 Detecting labels in streaming video events ................................................................................ 132 Setting up your Amazon Rekognition Video and Amazon Kinesis resources ............................. 132 Label detection operations for streaming video events ........................................................ 136 Detecting custom labels .......................................................................................................... 140 Detecting and analyzing faces .......................................................................................................... 141 Overview of face detection and face comparison ........................................................................ 142 Guidelines on face attributes ................................................................................................... 142 Detecting faces in an image ..................................................................................................... 143 DetectFaces operation request ......................................................................................... 150 DetectFaces operation response ....................................................................................... 150 Comparing faces in images ...................................................................................................... 154 CompareFaces operation request ...................................................................................... 161 CompareFaces operation response .................................................................................... 161 Detecting faces in a stored video .............................................................................................. 164 GetFaceDetection operation response ................................................................................ 168 Searching faces in a collection ......................................................................................................... 172 Managing collections.............................................................................................................. 172 Managing faces in a collection ................................................................................................. 172 Guidance for using IndexFaces ................................................................................................. 173 Critical or public safety applications .................................................................................. 173 Photo sharing and social media applications...................................................................... 173 General usage................................................................................................................ 173 Searching for faces within a collection ...................................................................................... 173 Using similarity thresholds to match faces ......................................................................... 174 iv Amazon Rekognition Developer Guide Use cases that involve public safety .......................................................................................... 174 Using Amazon Rekognition to help public safety ................................................................ 175 Creating a collection ............................................................................................................... 176 CreateCollection operation request ................................................................................... 179 CreateCollection operation response ................................................................................. 179 Tagging collections................................................................................................................. 179 Add tags to a new collection ........................................................................................... 180 Add tags to an existing collection ..................................................................................... 180 List tags in a collection ................................................................................................... 181 Delete tags from a collection ........................................................................................... 181 Listing collections................................................................................................................... 182 ListCollections operation request ...................................................................................... 186 ListCollections operation response .................................................................................... 186 Describing a collection ............................................................................................................ 186 DescribeCollection operation request ................................................................................ 190 DescribeCollection operation response .............................................................................. 190 Deleting a collection ............................................................................................................... 191 DeleteCollection operation request ................................................................................... 194 DeleteCollection operation response ................................................................................. 194 Adding faces to a collection ..................................................................................................... 194 Filtering faces................................................................................................................ 195 IndexFaces operation request ........................................................................................... 200 IndexFaces operation response ......................................................................................... 200 Listing faces in a collection ...................................................................................................... 206 ListFaces operation request .............................................................................................. 209 ListFaces operation response ............................................................................................ 210 Deleting faces from a collection ............................................................................................... 210 DeleteFaces operation request .......................................................................................... 213 DeleteFaces operation response ........................................................................................ 213 Searching for a face (face ID) ................................................................................................... 214 SearchFaces operation request ......................................................................................... 217 SearchFaces operation response ....................................................................................... 217 Searching for a face (image) .................................................................................................... 218 SearchFacesByImage operation request ............................................................................. 222 SearchFacesByImage operation response ........................................................................... 222 Searching stored videos for faces ............................................................................................ 223 GetFaceSearch operation response .................................................................................... 228 Searching faces in a collection in streaming video ....................................................................... 232 Setting up your Amazon Rekognition Video and Amazon Kinesis resources ............................. 232 Searching faces in a streaming video ................................................................................ 234 Streaming using a GStreamer plugin ................................................................................. 249 Troubleshooting streaming video ...................................................................................... 250 People pathing............................................................................................................................... 256 GetPersonTracking operation response ...................................................................................... 260 Detecting personal protective equipment .......................................................................................... 264 Types of PPE ......................................................................................................................... 265 Face cover..................................................................................................................... 265 Hand cover.................................................................................................................... 265 Head cover.................................................................................................................... 265 PPE detection confidence ........................................................................................................ 265 Summarizing PPE detected in an image .................................................................................... 265 Tutorial: Creating a AWS Lambda function that detects images with PPE ....................................... 266 Understanding the PPE detection API ....................................................................................... 266 Supplying an image........................................................................................................ 266 Understanding the DetectProtectiveEquipment response ..................................................... 267 Detecting PPE in an image ...................................................................................................... 271 Example: bounding boxes and face covers ................................................................................. 277 v Amazon Rekognition Developer Guide Recognizing celebrities.................................................................................................................... 286 Celebrity recognition compared to face search ........................................................................... 286 Recognizing celebrities in an image .......................................................................................... 287 Calling RecognizeCelebrities............................................................................................. 287 RecognizeCelebrities operation request ............................................................................. 292 RecognizeCelebrities operation response ........................................................................... 292 Recognizing celebrities in a stored video ................................................................................... 294 GetCelebrityRecognition operation response ...................................................................... 302 Getting celebrity information ................................................................................................... 304 Calling GetCelebrityInfo .................................................................................................. 304 GetCelebrityInfo operation request ................................................................................... 307 GetCelebrityInfo operation response ................................................................................. 307 Moderating content........................................................................................................................ 308 Using the image and video moderation APIs.............................................................................. 308 Detecting inappropriate images ................................................................................................ 310 Detecting inappropriate content in an image ..................................................................... 310 .................................................................................................................................... 310 DetectModerationLabels operation request ........................................................................ 314 DetectModerationLabels operation response ...................................................................... 314 Detecting inappropriate stored videos ...................................................................................... 315 GetContentModeration operation response ........................................................................ 319 Reviewing inappropriate content with Amazon A2I ..................................................................... 320 Detecting text................................................................................................................................ 324 Detecting text in an image ...................................................................................................... 325 DetectText operation request ........................................................................................... 329 DetectText operation response ......................................................................................... 330 Detecting text in a stored video ............................................................................................... 337 Filters............................................................................................................................ 341 GetTextDetection response.............................................................................................. 342 Detecting video segments............................................................................................................... 346 Technical cues........................................................................................................................ 346 Black frames.................................................................................................................. 347 Credits.......................................................................................................................... 347 Color bars...................................................................................................................... 347 Slates............................................................................................................................ 347 Studio logos.................................................................................................................. 347 Content......................................................................................................................... 347 Shot detection....................................................................................................................... 348 About the Amazon Rekognition Video Segment detection API ...................................................... 349 Using the Amazon Rekognition Segment API ............................................................................. 349 Starting segment analysis ................................................................................................ 349 Getting segment analysis results ...................................................................................... 350 Example: Detecting segments in a stored video .......................................................................... 353 Tutorials........................................................................................................................................ 361 Storing Amazon Rekognition Data with Amazon RDS and DynamoDB ............................................ 361 Prerequisites.................................................................................................................. 361 Getting Labels for Images in an Amazon S3 Bucket ............................................................. 362 Creating an Amazon DynamoDB Table .............................................................................. 363 Uploading Data to DynamoDB......................................................................................... 363 Creating a MySQL Database in Amazon RDS ...................................................................... 365 Uploading Data to a Amazon RDS MySQL Table ................................................................. 366 Using Amazon Rekognition and Lambda to tag assets in an Amazon S3 bucket ............................... 367 Prerequisites.................................................................................................................. 368 Configure the IAM Lambda role ........................................................................................ 368 Create the project ........................................................................................................... 369 Write the code ............................................................................................................... 371 Package the project ........................................................................................................ 377 vi Amazon Rekognition Developer Guide Deploy the Lambda function ............................................................................................ 378 Test the Lambda method ................................................................................................ 378 Creating AWS video analyzer applications .................................................................................. 379 Prerequisites.................................................................................................................. 379 Procedure...................................................................................................................... 380 Creating an Amazon Rekognition Lambda function ..................................................................... 380 Prerequisites.................................................................................................................. 381 Create the SNS topic ...................................................................................................... 381 Create the Lambda function ............................................................................................ 381 Configure the Lambda function ........................................................................................ 382 Configure the IAM Lambda role ........................................................................................ 382 Create the AWS Toolkit for Eclipse Lambda project ............................................................. 383 Test the Lambda function ................................................................................................ 385 Using Amazon Rekognition for Identity Verification .................................................................... 386 Prerequisites.................................................................................................................. 387 Creating a Collection ....................................................................................................... 387 New User Registration ..................................................................................................... 388 Existing User Login......................................................................................................... 394 Code examples............................................................................................................................... 396 Actions.................................................................................................................................. 397 Compare faces in an image against a reference image ......................................................... 397 Create a collection .......................................................................................................... 402 Delete a collection .......................................................................................................... 405 Delete faces from a collection .......................................................................................... 407 Describe a collection ....................................................................................................... 410 Detect faces in an image ................................................................................................. 413 Detect labels in an image ................................................................................................ 419 Detect moderation labels in an image ............................................................................... 423 Detect text in an image ................................................................................................... 427 Get information about celebrities ..................................................................................... 430 Index faces to a collection ............................................................................................... 431 List collections............................................................................................................... 435 List faces in a collection .................................................................................................. 438 Recognize celebrities in an image ..................................................................................... 441 Search for faces in a collection ......................................................................................... 445 Search for faces in a collection compared to a reference image ............................................. 448 Scenarios............................................................................................................................... 451 Build a collection and find faces in it ................................................................................ 451 Detect and display elements in images .............................................................................. 458 Detect information in videos ............................................................................................ 468 Cross-service examples............................................................................................................ 485 Detect PPE in images ...................................................................................................... 486 Detect faces in an image ................................................................................................. 487 Detect objects in images ................................................................................................. 487 Detect people and objects in a video ................................................................................ 490 Save EXIF and other image information ............................................................................. 491 API Reference................................................................................................................................. 492 Security......................................................................................................................................... 493 Identity and access management .............................................................................................. 493 Audience....................................................................................................................... 493 Authenticating with identities.......................................................................................... 494 Managing access using policies ......................................................................................... 496 How Amazon Rekognition works with IAM ......................................................................... 497 AWS managed policies .................................................................................................... 500 Using identity-based policy examples ................................................................................ 505 Resource-Based Policy Examples ....................................................................................... 508 Troubleshooting............................................................................................................. 508 vii Amazon Rekognition Developer Guide Data protection...................................................................................................................... 510 Data encryption............................................................................................................. 511 Internetwork traffic privacy .............................................................................................. 512 Monitoring............................................................................................................................. 512 Using CloudWatch metrics for Rekognition ........................................................................ 512 Access Rekognition metrics .............................................................................................. 513 Create an alarm ............................................................................................................. 514 CloudWatch metrics for Rekognition ................................................................................. 515 Logging Amazon Rekognition API calls with AWS CloudTrail ......................................................... 517 Amazon Rekognition information in CloudTrail ................................................................... 517 Understanding Amazon Rekognition log file entries ............................................................ 518 Using Amazon Rekognition with Amazon VPC endpoints ............................................................. 520 Creating Amazon VPC endpoints for Amazon Rekognition .................................................... 520 Create a VPC endpoint policy for Amazon Rekognition ........................................................ 521 Compliance validation............................................................................................................. 521 Resilience.............................................................................................................................. 522 Configuration and vulnerability analysis .................................................................................... 522 Cross-service confused deputy prevention ................................................................................. 522 Infrastructure security............................................................................................................. 524 Guidelines and quotas..................................................................................................................... 525 Supported regions.................................................................................................................. 525 Set quotas............................................................................................................................. 525 Amazon Rekognition Image ............................................................................................. 525 Amazon Rekognition Video stored video ............................................................................ 526 Amazon Rekognition Video streaming video ....................................................................... 526 Default quotas....................................................................................................................... 526 Calculate TPS quota change............................................................................................. 526 Best practices for TPS quotas ........................................................................................... 527 Create a case to change TPS quotas ................................................................................. 527 Document history........................................................................................................................... 529 AWS glossary................................................................................................................................. 534 viii Amazon Rekognition Developer Guide What is Amazon Rekognition? Amazon Rekognition makes it easy to add image and video analysis to your applications. You just provide an image or video to the Amazon Rekognition API, and the service can identify objects, people, text, scenes, and activities. It can detect any inappropriate content as well. Amazon Rekognition also provides highly accurate facial analysis, face comparison, and face search capabilities. You can detect, analyze, and compare faces for a wide variety of use cases, including user verification, cataloging, people counting, and public safety. Amazon Rekognition is based on the same proven, highly scalable, deep learning technology developed by Amazon’s computer vision scientists to analyze billions of images and videos daily. It requires no machine learning expertise to use. Amazon Rekognition includes a simple, easy-to-use API that can quickly analyze any image or video file that’s stored in Amazon S3. Amazon Rekognition is always learning from new data, and we’re continually adding new labels and facial comparison features to the service. For more information, see the Amazon Rekognition FAQs. Common use cases for using Amazon Rekognition include the following: • Searchable image and video libraries – Amazon Rekognition makes images and stored videos searchable so you can discover objects and scenes that appear within them.   • Face-based user verification – Amazon Rekognition enables your applications to confirm user identities by comparing their live image with a reference image.   • Detection of Personal Protective Equipment Amazon Rekognition detects Personal Protective Equipment (PPE) such as face covers, head covers, and hand covers on persons in images. You can use PPE detection where safety is the highest priority. For example, industries such as construction, manufacturing, healthcare, food processing, logistics, and retail. With PPE detection, you can automatically detect if a person is wearing a specific type of PPE. You can use the detection results to send a notification or to identify places where safety warnings or training practices can be improved.   • Sentiment and demographic analysis – Amazon Rekognition interprets emotional expressions such as happy, sad, or surprise, and demographic information such as gender from facial images. Amazon Rekognition can analyze images, and send the emotion and demographic attributes to Amazon Redshift for periodic reporting on trends such as in store locations and similar scenarios. Note that a prediction of an emotional expression is based on the physical appearance of a person's face only. It is not indicative of a person’s internal emotional state, and Rekognition should not be used to make such a determination.   • Facial Search – With Amazon Rekognition, you can search images, stored videos, and streaming videos for faces that match those stored in a container known as a face collection. A face collection is an index of faces that you own and manage. Searching for people based on their faces requires two major steps in Amazon Rekognition: 1. Index the faces. 2. Search the faces.   1 Amazon Rekognition Developer Guide • Unsafe content detection – Amazon Rekognition can detect adult and violent content in images and in stored videos. Developers can use the returned metadata to filter inappropriate content based on their business needs. Beyond flagging an image based on the presence of unsafe content, the API also returns a hierarchical list of labels with confidence scores. These labels indicate specific categories of unsafe content, which enables granular filtering and management of large volumes of user-generated content (UGC). Examples include social and dating sites, photo sharing platforms, blogs and forums, apps for children, ecommerce sites, entertainment, and online advertising services.   • Celebrity recognition – Amazon Rekognition can recognize celebrities within supplied images and in videos. Amazon Rekognition can recognize thousands of celebrities across a number of categories, such as politics, sports, business, entertainment, and media.   • Text detection – Amazon Rekognition Text in Image enables you to recognize and extract textual content from images. Text in Image supports most fonts, including highly stylized ones. It detects text and numbers in different orientations, such as those commonly found in banners and posters. In image sharing and social media applications, you can use it to enable visual search based on an index of images that contain the same keywords. In media and entertainment applications, you can catalog videos based on relevant text on screen, such as ads, news, sport scores, and captions. Finally, in public safety applications, you can identify vehicles based on license plate numbers from images taken by street cameras.   • Custom labels– With Amazon Rekognition Custom Labels, you can identify the objects and scenes in images that are specific to your business needs. For example, you can find your logo in social media posts, identify your products on store shelves, classify machine parts in an assembly line, distinguish healthy and infected plants, or detect animated characters in videos. For more information, see What is Amazon Rekognition Custom Labels? in the Amazon Rekognition Custom Labels Developer Guide. Some of the benefits of using Amazon Rekognition include: • Integrating powerful image and video analysis into your apps – You don’t need computer vision or deep learning expertise to take advantage of the reliable image and video analysis in Amazon Rekognition. With the API, you can build image and video analysis into any web, mobile, or connected device application.   • Deep learning-based image and video analysis – Amazon Rekognition uses deep-learning technology to accurately analyze images, find and compare faces in images, and detect objects and scenes within your images and videos. You can analyze images for the presence of many different labels and then filter the results to include and/or exclude sets of labels or label categories.   • Scalable image analysis – Amazon Rekognition enables you to analyze millions of images so you can curate and organize massive amounts of visual data.   • Analyze and filter images based on image properties – Amazon Rekognition lets you analyze image properties like quality or colors. You can determine the sharpness, brightness, and contrast of images. You can also detect dominant colors in the entire image, foreground, background, and objects with bounding boxes.   • Integration with other AWS services – Amazon Rekognition is designed to work seamlessly with other AWS services like Amazon S3 and AWS Lambda. You can call the Amazon Rekognition API directly from 2