IN ACTION Peter Harrington M A N N I N G Machine Learning in Action Licensed to Brahim Chaibdraa <[email protected]> Licensed to Brahim Chaibdraa <[email protected]> Machine Learning in Action PETER HARRINGTON MANNING Shelter Island Licensed to Brahim Chaibdraa <[email protected]> For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 261 Shelter Island, NY 11964 Email: [email protected] ©2012 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co.Development editor:Jeff Bleiel 20 Baldwin Road Technical proofreaders: Tricia Hoffman, Alex Ott PO Box 261 Copyeditor: Linda Recktenwald Shelter Island, NY 11964 Proofreader: Maureen Spencer Typesetter: Gordan Salinovic Cover designer: Marija Tudor ISBN 9781617290183 Printed in the United States of America 1 2 3 4 5 6 7 8 9 10 – MAL – 17 16 15 14 13 12 Licensed to Brahim Chaibdraa <[email protected]> To Joseph and Milo Licensed to Brahim Chaibdraa <[email protected]> Licensed to Brahim Chaibdraa <[email protected]> brief contents P 1 C ...............................................................1 ART LASSIFICATION 1 ■ Machine learning basics 3 2 ■ Classifying with k-Nearest Neighbors 18 3 ■ Splitting datasets one feature at a time: decision trees 37 4 ■ Classifying with probability theory: naïve Bayes 61 5 ■ Logistic regression 83 6 ■ Support vector machines 101 7 ■ Improving classification with the AdaBoost meta-algorithm 129 P 2 F ..............151 ART ORECASTING NUMERIC VALUES WITH REGRESSION 8 ■ Predicting numeric values: regression 153 9 ■ Tree-based regression 179 P 3 U ...............................................205 ART NSUPERVISED LEARNING 10 ■ Grouping unlabeled items using k-means clustering 207 11 ■ Association analysis with the Apriori algorithm 224 12 ■ Efficiently finding frequent itemsets with FP-growth 248 vii Licensed to Brahim Chaibdraa <[email protected]> viii BRIEF CONTENTS P 4 A .......................................................267 ART DDITIONAL TOOLS 13 ■ Using principal component analysis to simplify data 269 14 ■ Simplifying data with the singular value decomposition 280 15 ■ Big data and MapReduce 299 Licensed to Brahim Chaibdraa <[email protected]> contents preface xvii acknowledgments xix about this book xxi about the author xxv about the cover illustration xxvi P 1 C ...................................................1 ART LASSIFICATION 1 Machine learning basics 3 1.1 What is machine learning? 5 Sensors and the data deluge 6 ■ Machine learning will be more important in the future 7 1.2 Key terminology 7 1.3 Key tasks of machine learning 10 1.4 How to choose the right algorithm 11 1.5 Steps in developing a machine learning application 11 1.6 Why Python? 13 Executable pseudo-code 13 ■ Python is popular 13 ■ What Python has that other languages don’t have 14 ■ Drawbacks 14 1.7 Getting started with the NumPy library 15 1.8 Summary 17 ix Licensed to Brahim Chaibdraa <[email protected]>