About This E-Book EPUB is an open, industry-standard format for e-books. However, support for EPUB and its many features varies across reading devices and applications. Use your device or app settings to customize the presentation to your liking. Settings that you can customize often include font, font size, single or double column, landscape or portrait mode, and figures that you can click or tap to enlarge. For additional information about the settings and features on your reading device or app, visit the device manufacturer’s Web site. Many titles include programming code or configuration examples. To optimize the presentation of these elements, view the e-book in single-column, landscape mode and adjust the font size to the smallest setting. In addition to presenting code and configurations in the reflowable text format, we have included images of the code that mimic the presentation found in the print book; therefore, where the reflowable format may compromise the presentation of the code listing, you will see a “Click here to view code image” link. Click the link to view the print-fidelity code image. To return to the previous page viewed, click the Back button on your device or app. Sams Teach Yourself: Big Data Analytics with Microsoft HDInsight® in 24 Hours Arshad Ali Manpreet Singh 800 East 96th Street, Indianapolis, Indiana, 46240 USA Sams Teach Yourself Big Data Analytics with Microsoft HDInsight® in 24 Hours Copyright © 2016 by Pearson Education, Inc. All rights reserved. No part of this book shall be reproduced, stored in a retrieval system, or transmitted by any means, electronic, mechanical, photocopying, recording, or otherwise, without written permission from the publisher. No patent liability is assumed with respect to the use of the information contained herein. Although every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions. Nor is any liability assumed for damages resulting from the use of the information contained herein. ISBN-13: 978-0-672-33727-7 ISBN-10: 0-672-33727-4 Library of Congress Control Number: 2015914167 Printed in the United States of America First Printing November 2015 Editor-in-Chief Greg Wiegand Acquisitions Editor Joan Murray Development Editor Sondra Scott Managing Editor Sandra Schroeder Senior Project Editor Tonya Simpson Copy Editor Krista Hansing Editorial Services, Inc Senior Indexer Cheryl Lenser Proofreader Anne Goebel Technical Editors Shayne Burgess Ron Abellera Publishing Coordinator Cindy Teeter Cover Designer Mark Shirar Compositor codeMantra Trademarks All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized. Sams Publishing cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark. HDInsight is a registered trademark of Microsoft Corporation. Warning and Disclaimer Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness is implied. The information provided is on an “as is” basis. The authors and the publisher shall have neither liability nor responsibility to any person or entity with respect to any loss or damages arising from the information contained in this book. Special Sales For information about buying this title in bulk quantities, or for special sales opportunities (which may include electronic versions; custom cover designs; and content particular to your business, training goals, marketing focus, or branding interests), please contact our corporate sales department at [email protected] or (800) 382-3419. For government sales inquiries, please contact [email protected]. For questions about sales outside the U.S., please contact [email protected]. Contents at a Glance Introduction Part I: Understanding Big Data, Hadoop 1.0, and 2.0 HOUR 1 Introduction of Big Data, NoSQL, and Business Value Proposition 2 Introduction to Hadoop, Its Architecture, Ecosystem, and Microsoft Offerings 3 Hadoop Distributed File System Versions 1.0 and 2.0 4 The MapReduce Job Framework and Job Execution Pipeline 5 MapReduce—Advanced Concepts and YARN Part II: Getting Started with HDInsight and Understanding Its Different Components HOUR 6 Getting Started with HDInsight, Provisioning Your HDInsight Service Cluster, and Automating HDInsight Cluster Provisioning 7 Exploring Typical Components of HDFS Cluster 8 Storing Data in Microsoft Azure Storage Blob 9 Working with Microsoft Azure HDInsight Emulator Part III: Programming MapReduce and HDInsight Script Action HOUR 10 Programming MapReduce Jobs 11 Customizing the HDInsight Cluster with Script Action Part IV: Querying and Processing Big Data in HDInsight HOUR 12 Getting Started with Apache Hive and Apache Tez in HDInsight 13 Programming with Apache Hive, Apache Tez in HDInsight, and Apache HCatalog 14 Consuming HDInsight Data from Microsoft BI Tools over Hive ODBC Driver: Part 1 15 Consuming HDInsight Data from Microsoft BI Tools over Hive ODBC Driver: Part 2 16 Integrating HDInsight with SQL Server Integration Services 17 Using Pig for Data Processing 18 Using Sqoop for Data Movement Between RDBMS and HDInsight Part V: Managing Workflow and Performing Statistical Computing HOUR 19 Using Oozie Workflows and Job Orchestration with HDInsight 20 Performing Statistical Computing with R Part VI: Performing Interactive Analytics and Machine Learning HOUR 21 Performing Big Data Analytics with Spark 22 Microsoft Azure Machine Learning Part VII: Performing Real-time Analytics HOUR 23 Performing Stream Analytics with Storm 24 Introduction to Apache HBase on HDInsight Index Table of Contents Introduction Part I: Understanding Big Data, Hadoop 1.0, and 2.0 HOUR 1: Introduction of Big Data, NoSQL, and Business Value Proposition Types of Analysis Types of Data Big Data Managing Big Data NoSQL Systems Big Data, NoSQL Systems, and the Business Value Proposition Application of Big Data and Big Data Solutions Summary Q&A HOUR 2: Introduction to Hadoop, Its Architecture, Ecosystem, and Microsoft Offerings What Is Apache Hadoop? Architecture of Hadoop and Hadoop Ecosystems What’s New in Hadoop 2.0 Architecture of Hadoop 2.0 Tools and Technologies Needed with Big Data Analytics Major Players and Vendors for Hadoop Deployment Options for Microsoft Big Data Solutions Summary Q&A HOUR 3: Hadoop Distributed File System Versions 1.0 and 2.0 Introduction to HDFS HDFS Architecture Rack Awareness WebHDFS Accessing and Managing HDFS Data What’s New in HDFS 2.0 Summary Q&A HOUR 4: The MapReduce Job Framework and Job Execution Pipeline Introduction to MapReduce MapReduce Architecture MapReduce Job Execution Flow Summary Q&A HOUR 5: MapReduce—Advanced Concepts and YARN DistributedCache Hadoop Streaming MapReduce Joins Bloom Filter Performance Improvement Handling Failures Counter YARN Uber-Tasking Optimization Failures in YARN Resource Manager High Availability and Automatic Failover in YARN Summary Q&A Part II: Getting Started with HDInsight and Understanding Its Different Components HOUR 6: Getting Started with HDInsight, Provisioning Your HDInsight Service Cluster, and Automating HDInsight Cluster Provisioning Introduction to Microsoft Azure Understanding HDInsight Service Provisioning HDInsight on the Azure Management Portal Automating HDInsight Provisioning with PowerShell Managing and Monitoring HDInsight Cluster and Job Execution Summary Q&A Exercise HOUR 7: Exploring Typical Components of HDFS Cluster HDFS Cluster Components HDInsight Cluster Architecture High Availability in HDInsight Summary Q&A HOUR 8: Storing Data in Microsoft Azure Storage Blob Understanding Storage in Microsoft Azure Benefits of Azure Storage Blob over HDFS Azure Storage Explorer Tools Summary Q&A HOUR 9: Working with Microsoft Azure HDInsight Emulator Getting Started with HDInsight Emulator Setting Up Microsoft Azure Emulator for Storage Summary Q&A Part III: Programming MapReduce and HDInsight Script Action HOUR 10: Programming MapReduce Jobs MapReduce Hello World! Analyzing Flight Delays with MapReduce Serialization Frameworks for Hadoop Hadoop Streaming Summary Q&A HOUR 11: Customizing the HDInsight Cluster with Script Action Identifying the Need for Cluster Customization Developing Script Action Consuming Script Action Running a Giraph job on a Customized HDInsight Cluster Testing Script Action with HDInsight Emulator Summary