Data Mining Data Mining For Dummies® Published by: John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030-5774, www.wiley.com Copyright © 2014 by John Wiley & Sons, Inc., Hoboken, New Jersey Media and software compilation copyright © 2014 by John Wiley & Sons, Inc. All rights reserved. Published simultaneously in Canada No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without the prior written permission of the Publisher. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at www.wiley.com/go/permissions. Trademarks: Wiley, For Dummies, the Dummies Man logo, Dummies.com, Making Everything Easier, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and may not be used without written permission. Samsung and Galaxy S are registered trademarks of Samsung Electronics Co. Ltd. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book. LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY: THE PUBLISHER AND THE AUTHOR MAKE NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS WORK AND SPECIFICALLY DISCLAIM ALL WARRANTIES, INCLUDING WITH- OUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE. NO WARRANTY MAY BE CREATED OR EXTENDED BY SALES OR PROMOTIONAL MATERIALS. THE ADVICE AND STRATEGIES CONTAINED HEREIN MAY NOT BE SUITABLE FOR EVERY SITUATION. THIS WORK IS SOLD WITH THE UNDERSTANDING THAT THE PUBLISHER IS NOT ENGAGED IN RENDERING LEGAL, ACCOUNTING, OR OTHER PROFESSIONAL SERVICES. IF PROFESSIONAL ASSISTANCE IS REQUIRED, THE SERVICES OF A COMPETENT PROFESSIONAL PERSON SHOULD BE SOUGHT. NEITHER THE PUBLISHER NOR THE AUTHOR SHALL BE LIABLE FOR DAMAGES ARISING HEREFROM. THE FACT THAT AN ORGANIZATION OR WEBSITE IS REFERRED TO IN THIS WORK AS A CITATION AND/OR A POTENTIAL SOURCE OF FURTHER INFORMATION DOES NOT MEAN THAT THE AUTHOR OR THE PUBLISHER ENDORSES THE INFORMATION THE ORGANIZATION OR WEBSITE MAY PROVIDE OR RECOMMENDATIONS IT MAY MAKE. FURTHER, READERS SHOULD BE AWARE THAT INTERNET WEBSITES LISTED IN THIS WORK MAY HAVE CHANGED OR DISAPPEARED BETWEEN WHEN THIS WORK WAS WRITTEN AND WHEN IT IS READ. For general information on our other products and services, please contact our Customer Care Department within the U.S. at 877-762-2974, outside the U.S. at 317-572-3993, or fax 317-572-4002. For technical support, please visit www.wiley.com/techsupport. Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com. Library of Congress Control Number: 2014935519 ISBN 978-1-118-89317-3 (pbk); ISBN 978-1-118-89316-6 (ebk); ISBN 978-1-118-89319-7 (ebk) Manufactured in the United States of America 10 9 8 7 6 5 4 3 2 1 Contents at a Glance Introduction ................................................................ 1 Part I: Getting Started with Data Mining ....................... 5 Chapter 1: Catching the Data-Mining Train ....................................................................7 Chapter 2: A Day in Your Life as a Data Miner .............................................................17 Chapter 3: Teaming Up to Reach Your Goals ...............................................................49 Part II: Exploring Data-Mining Mantras and Methods ...... 61 Chapter 4: Learning the Laws of Data Mining ..............................................................63 Chapter 5: Embracing the Data-Mining Process ..........................................................73 Chapter 6: Planning for Data-Mining Success...............................................................89 Chapter 7: Gearing Up with the Right Sof tware ............................................................97 Part III: Gathering the Raw Materials ....................... 109 Chapter 8: Digging into Your Data ...............................................................................111 Chapter 9: Making New Data ........................................................................................119 Chapter 10: Ferreting Out Public Data Sources .........................................................141 Chapter 11: Buying Data ...............................................................................................163 Part IV: A Data Miner’s Survival Kit .......................... 171 Chapter 12: Getting Familiar with Your Data .............................................................173 Chapter 13: Dealing in Graphic Detail .........................................................................195 Chapter 14: Showing Your Data Who’s Boss ..............................................................219 Chapter 15: Your Exciting Career in Modeling ...........................................................245 Part V: More Data-Mining Methods ........................... 273 Chapter 16: Data Mining Using Classic Statistical Methods .....................................275 Chapter 17: Mining Data for Clues ...............................................................................295 Chapter 18: Expanding Your Horizons ........................................................................307 Part VI: The Part of Tens .......................................... 319 Chapter 19: Ten Great Resources for Data Miners ....................................................321 Chapter 20: Ten Useful Kinds of Analysis That Complement Data Mining ............325 Appendix A: Glossary ............................................... 333 Appendix B: Data-Mining Sof tware Sources ............... 339 Appendix C: Major Data Vendors ............................... 349 Appendix D: Sources and Citations ............................ 357 Index ...................................................................... 361 Table of Contents Introduction ................................................................. 1 About This Book ..............................................................................................1 Foolish Assumptions .......................................................................................2 Icons Used in This Book .................................................................................2 Beyond the Book .............................................................................................3 Where to Go from Here ...................................................................................3 Part I: Getting Started with Data Mining ....................... 5 Chapter 1: Catching the Data-Mining Train . . . . . . . . . . . . . . . . . . . . . . . 7 Getting Real about Data Mining .....................................................................7 Not your professor’s statistics .............................................................8 The value of data mining.......................................................................8 Working for it..........................................................................................9 Doing What Data Miners Do .........................................................................10 Focusing on the business....................................................................10 Understanding how data miners spend their time ..........................11 Getting to know the data-mining process .........................................11 Making models .....................................................................................12 Understanding mathematical models ...............................................12 Putting information into action ..........................................................13 Discovering Tools and Methods ..................................................................13 Visual programming ............................................................................14 Working quick and dirty .....................................................................15 Testing, testing, and testing some more ...........................................16 Chapter 2: A Day in Your Life as a Data Miner . . . . . . . . . . . . . . . . . . . . 17 Starting Your Day Off Right ..........................................................................17 Meeting the team .................................................................................18 Exploring with aim ...............................................................................18 Structuring time with the right process............................................20 Understanding Your Business Goals ...........................................................20 Understanding Your Data .............................................................................22 Describing data ....................................................................................22 Exploring data ......................................................................................23 Cleaning data ........................................................................................27 Preparing Your Data ......................................................................................28 Taking first steps with the property data .........................................28 Preparing the ownership change indicator ......................................32 Merging the datasets ...........................................................................32 Deriving new variables ........................................................................34 viii Data Mining For Dummies Modeling Your Data .......................................................................................40 Using balanced data ............................................................................40 Splitting data ........................................................................................41 Building a model ..................................................................................43 Evaluating Your Results ................................................................................44 Examining the decision tree ...............................................................44 Using a diagnostic chart .....................................................................46 Assessing the status of the model .....................................................47 Putting Your Results into Action .................................................................48 Chapter 3: Teaming Up to Reach Your Goals . . . . . . . . . . . . . . . . . . . . . 49 Nothing Could Be Finer Than to Be a Data Miner .....................................49 You can be a data miner .....................................................................50 Using the knowledge you have...........................................................51 Data Miners Play Nicely with Others ..........................................................51 Cooperation is a necessity..................................................................51 Oh, the people you’ll meet! .................................................................53 Working with Executives ..............................................................................56 Greetings and elicitations ...................................................................57 Lining up your priorities .....................................................................58 Talking data mining with executives .................................................58 Part II: Exploring Data-Mining Mantras and Methods .... 61 Chapter 4: Learning the Laws of Data Mining . . . . . . . . . . . . . . . . . . . . 63 1st Law: Business Goals ................................................................................63 2nd Law: Business Knowledge .....................................................................64 3rd Law: Data Preparation ............................................................................65 4th Law: Right Model ....................................................................................66 5th Law: Pattern .............................................................................................67 6th Law: Amplification ..................................................................................68 7th Law: Prediction .......................................................................................69 8th Law: Value ................................................................................................70 9th Law: Change .............................................................................................70 Chapter 5: Embracing the Data-Mining Process . . . . . . . . . . . . . . . . . . 73 Whose Standard Is It, Anyway? ....................................................................73 Approaching the process in phases ..................................................74 Cycling through phases and projects ...............................................74 Documenting your work .....................................................................75 Business Understanding ...............................................................................76 Data Understanding .......................................................................................79 Data Preparation ............................................................................................82 Modeling .........................................................................................................84 Evaluation .......................................................................................................86 Deployment ....................................................................................................87