R in a Nutshell Joseph Adler Published by O’Reilly Media Beijing ⋅ Cambridge ⋅ Farnham ⋅ Köln ⋅ Sebastopol ⋅ Tokyo Preface It’s been over 10 years since I was first introduced to R. Back then, I was a young product development manager at DoubleClick, a company that sold advertising software for managing online ad sales. I was working on inventory prediction: estimating the number of ad impressions that could be sold for a given search term, web page, or demographic characteristic. I wanted to play with the data myself, but we couldn’t afford a piece of expensive software like SAS or MATLAB. I looked around for a little while, trying to find an open-source statistics package, and stumbled on R. Back then, R was a bit rough around the edges and was missing a lot of the features it has today (like fancy graphics and statistics functions). But R was intuitive and easy to use; I was hooked. Since that time, I’ve used R to do many different things: estimate credit risk, analyze baseball statistics, and look for Internet security threats. I’ve learned a lot about data and matured a lot as a data analyst. R, too, has matured a great deal over the past decade. R is used at the world’s largest technology companies (including Google, Microsoft, and Facebook), the largest pharmaceutical companies (including Johnson & Johnson, Merck, and Pfizer), and at hundreds of other companies. It’s used in statistics classes at universities around the world and by statistics researchers to try new techniques and algorithms. Why I Wrote This Book This book is designed to be a concise guide to R. It’s not intended to be a book about statistics or an exhaustive guide to R. In this book, I tried to show all the things that R can do and to give examples showing how to do them. This book is designed to be a good desktop reference. I wrote this book because I like R. R is fun and intuitive in ways that other solutions are not. You can do things in a few lines of R that could take hours of struggling in a spreadsheet. Similarly, you can do things in a few lines of R that could take pages of Java code (and hours of Java coding). There are some excellent books on R, but I couldn’t find an inexpensive book that gave an overview of everything you could do in R. I hope this book helps you use R. When Should You Use R? I think R is a great piece of software, but it isn’t the right tool for every problem. Clearly, it would be ridiculous to write a video game in R, but it’s not even the best tool for all data problems. R is very good at plotting graphics, analyzing data, and fitting statistical models using data that fits in the computer’s memory. It’s not as good at storing data in complicated structures, efficiently querying data, or working with data that doesn’t fit in the computer’s memory. Typically, I use a scripting language like Perl, Python, or Ruby to preprocess files before using them in R. (If the files are really big, I’ll use Pig.) It’s technically possible to use R for these problems (by reading files one line at a time and using R’s regular expression support), but it’s pretty awkward. To hold large data files, I usually use Hadoop. Sometimes I use a database like MySQL, PostgreSQL, SQLite, or Oracle (when someone else is paying the license fee). What’s New in the Second Edition? This edition isn’t a total rewrite of the first book. But I have tried to improve the book in a few significant ways: There are new chapters on ggplot2 and using R with Hadoop. Formatting changes should make code examples easier to read. I’ve changed the order of the book slightly, grouping the plotting chapters together. I’ve made some minor updates to reflect changes in R 2.14 and R 2.15. There are some new sections on useful tools for manipulating data in R, such as plyr and reshape. I’ve corrected dozens of errors. R License Terms R is an open-source software package, licensed under the GNU General Public License (GPL).[1] This means that you can install R for free on most desktop and server machines. (Comparable commercial software packages sell for hundreds or thousands of dollars. If R were a poor substitute for the commercial software packages, they might have limited appeal. However, I think R is better than its commercial counterparts in many respects.) Capability You can find implementations for hundreds (maybe thousands) of statistical and data analysis algorithms in R. No commercial package offers anywhere near the scope of functionality available through the Comprehensive R Archive Network (CRAN). Community There are now hundreds of thousands (if not millions) of R users worldwide. By using R, you can be sure that you’re using the same software your colleagues are using. Performance R’s performance is comparable, or superior, to most commercial analysis packages. R requires you to load data sets into memory before processing. If you have enough memory to hold the data, R can run very quickly. Luckily, memory is cheap. You can buy 32 GB of server RAM for less than the cost of a single desktop license of a comparable piece of commercial statistical software. [1] There is some controversy about GPL licensed software and what it means to you as a corporate user. Some users are afraid that any code they write in R will be bound by the GPL. If you are not writing extensions to R, you do not need to worry about this issue. R is an interpreter, and the GPL does not apply to a program just because it is executed on a GPL-licensed interpreter. If you are writing extensions to R, they might be bound by the GPL. For more information, see the GNU foundation’s FAQ on the GPL: http://www.gnu.org/licenses/gplfaq. However, for a definite answer, see an attorney. If you are worried about a specific application, see an attorney. Examples In this book, I have tried to provide many working examples of R code. I deliberately decided to use new and original examples, instead of relying on the data sets included with R. I am not implying that the included examples are not good; they are good. I just wanted to give readers a second set of examples. In most cases, the examples are short and simple and I have not provided them in a downloadable form. However, I have included example data and a few of the longer examples in the nutshell R package, available through CRAN. To install the nutshell package, type the following command on the R console: > install.packages("nutshell")