Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Getting to Know Your Data with R

Próximo SlideShare
Chapter1 introduction
Chapter1 introduction
Cargando en…3

Eche un vistazo a continuación

1 de 121 Anuncio

Más Contenido Relacionado

Presentaciones para usted (20)

Similares a Getting to Know Your Data with R (20)


Más reciente (20)

Getting to Know Your Data with R

  1. 1. dev.Objective() ~ Thursday, 14 May 2015 Getting to KnowYour Data with R Steve Withington
  2. 2. 01 Steve Withington ✤ Blue River / Mura CMS Application Engineer & Instructor! ✤ dev.Objective() Steering Committee Member since 2010! ✤ Author! ✤ Marathoner & Triathlete! ✤ Amateur Data Enthusiast
  3. 3. Steps in a Data Analysis ✤ Define the question! ✤ Define the ideal data set! ✤ Determine what data you can access! ✤ Obtain the data! ✤ Clean the data! ✤ Exploratory data analysis! ✤ Statistical prediction/modeling! ✤ Interpret results! ✤ Challenge results! ✤ Synthesize/write up results! ✤ Create reproducible code
  4. 4. Steps in a Data Analysis ✤ Define the question! ✤ Define the ideal data set! ✤ Determine what data you can access! ✤ Obtain the data! ✤ Clean the data! ✤ Exploratory data analysis! ✤ Statistical prediction/modeling! ✤ Interpret results! ✤ Challenge results! ✤ Synthesize/write up results! ✤ Create reproducible code
  5. 5. Types of Data Science Questions ✤ In approximate order of difficulty! ✤ Descriptive! ✤ Exploratory! ✤ Inferential! ✤ Predictive! ✤ Causal! ✤ Mechanistic
  6. 6. Descriptive Analysis ✤ Goal: Describe a set of data! ✤ The first kind of data analysis performed! ✤ Commonly applied to census data! ✤ The description and interpretation are different steps! ✤ Descriptions can usually not be generalized without additional statistical modeling
  7. 7. Descriptive Analysis –
  8. 8. Exploratory Analysis ✤ Goal: Find relationships you didn’t know about! ✤ Exploratory models are good for discovering new connections! ✤ They are also useful for defining future studies! ✤ Exploratory analyses are usually not the final say! ✤ Exploratory analyses alone should not be used for generalizing/predicting! ✤ Correlation does not imply causation
  9. 9. Correlation is Not Causation ✤ Even if you observe two variables are correlated with each other, you have to prove to yourself that they’re not related because of some other variable you didn’t measure! ✤ Example: shoe size and literacy
  10. 10. Exploratory Analysis –
  11. 11. Inferential Analysis ✤ Goal: Use a relatively small sample of data to say something about a bigger population! ✤ Inference is commonly the goal of statistical models! ✤ Inference involves estimating both the quantity you care about and your uncertainty about your estimate! ✤ Inference depends heavily on both the population and the sampling scheme
  12. 12. Inferential Analysis –
  13. 13. Predictive Analysis ✤ Goal: To use the data on some objects to predict the values for another object! ✤ If X predicts Y it does not mean that X causes Y! ✤ Accurate prediction depends heavily on measuring the right variables! ✤ Although there are better and worse prediction models, more data and a simple model works really well! ✤ Prediction is very hard, especially about the future references
  14. 14. Predictive Analysis –
  15. 15. Predictive Analysis –
  16. 16. Causal Analysis ✤ Goal: To find out what happens to one variable when you make another variable change! ✤ Usually randomized studies are required to identify causation! ✤ There are approaches to inferring causation in non-randomized studies, but they are complicated and sensitive to assumptions! ✤ Causal relationships are usually identified as average effects, but may not apply to every individual! ✤ Causal models are usually the “gold standard” for data analysis
  17. 17. Causal Analysis –
  18. 18. Mechanistic Analysis ✤ Goal: Understand the exact changes in variables that lead to changes in other variables for individual objects! ✤ Incredibly hard to infer, except in simple situations! ✤ Usually modeled by a deterministic set of equations (physical/ engineering science)! ✤ Generally the random component of the data is measurement error! ✤ If the equations are known but the parameters are not, they may be inferred with data analysis
  19. 19. Mechanistic Analysis –
  20. 20. “You can have data without information, 
 but you cannot have information without data.” –Daniel Keys Moran
  21. 21. 01
  22. 22. NewTerminology –
  23. 23. Yottabyte ✤ 1000 kB kilobyte
 1000 2 MB megabyte
 1000 3 GB gigabyte
 1000 4 TB terabyte
 1000 5 PB petabyte
 1000 6 EB exabyte
 1000 7 ZB zettabyte
 1000 8 YB yottabyte! ✤ 1 YB 
 = 1000 8 bytes 
 = 10 24 bytes 
 = 1 000 000 000 000 000 000 000 000 bytes 
 = 1000 zettabytes 
 = 1 trillion terabytes
  24. 24. NSA’s Bumblehive – (Capable of storing one thousand trillion gigabytes!)
  25. 25. Data ✤ Companies have terabytes of data about the consumers they interact with! ✤ Governmental, academic, and private research institutions have extensive archival and survey data on every manner of research topic
  26. 26. The Challenge ✤ Presenting information in easily accessible and digestible ways
  27. 27. Visualizations ✤ We’re really good at gleaning useful information from visual presentations! ✤ Graphical presentations are an excellent way to convey results and uncover meaning
  28. 28. Visualizing Friendships –
  29. 29. Visualizing Friendships –
  30. 30. “Data is a set of values of 
 qualitative or quantitative variables; restated, 
 pieces of data are individual pieces of information.” –
  31. 31. Definition of Data “Data is a set of values of qualitative or quantitative variables; restated, pieces of data are individual pieces of information.” ! ✤ Qualitative: properties that are observed and can generally not be measured with a numerical result.
 (Sex, country of origin, etc.)! ✤ Quantitative: properties that can exist as a magnitude or multitude. 
 (Height, weight, blood pressure, etc.)
  32. 32. Definition of Data “Data is a set of values of qualitative or quantitative variables; restated, pieces of data are individual pieces of information.” ! ✤ Variable: a logical set of attributes! ✤ Attribute: a characteristic of an object (person, thing, etc.)
  33. 33. Definition of Data “Data is a set of values of qualitative or quantitative variables; restated, pieces of data are individual pieces of information.” ! ✤ Information: an answer to a question, as well as that from which knowledge and data can be derived (as data represents values attributed to parameters, and knowledge signifies understanding of real things or abstract concepts). As it regards data, the information’s existence is not necessarily coupled to an observer (it exists beyond an event horizon, for example), while in the case of knowledge, information requires a cognitive observer.
  34. 34. “The data may not contain the answer. 
 The combination of some data and an aching desire 
 for an answer does not ensure that a reasonable answer 
 can be extracted from a given body of data.” –John Wilder Tukey
  35. 35. What Data Looks Like –
  36. 36. What Data Looks Like –
  37. 37. What Data Looks Like –
  38. 38. What Data Looks Like –
  39. 39. Data is Second Most Important ✤ The most important thing in data science is the question! ✤ The second most important is the data! ✤ Often the data will limit or enable the questions! ✤ But having data can’t save you if you don’t have a question
  40. 40. 01
  41. 41. “We wanted users to be able to begin in an interactive environment, where they did not consciously think of themselves as programming. Then as their needs became clearer and their sophistication increased, they should be able to slide gradually into programming, when the language and system aspects would become more important.” –John Chambers, “Stages in the Evolution of S”
  42. 42. R ✤ Comprehensive statistical platform (just about any type of data analysis can be done in R)! ✤ Useful for interactive work, and also contains a powerful programming language for developing new tools (user -> programmer)! ✤ Open source & free (see! ✤ State of the art graphics! ✤ Easily import data from a wide variety of sources! ✤ RStudio IDE! ✤ Developer Ecosystem and active community (mailing lists, StackOverflow, etc.)! ✤ Can be integrated into other languages, including C++, Java, Python, PHP, etc.
  43. 43. Design of the R System ✤ The “base” R system that you download from CRAN! ✤ Everything else
  44. 44. The R System ✤ R functionality is divided into a number of packages! ✤ The “base” R system contains, among other things, the base package which is required to run R and contains the most fundamental functions! ✤ There are several other packages contained in the “base” system as well as several “recommended” packages! ✤ Over 6,000 other packages are available on CRAN developed by users and programmers around the world! ✤ In addition, sites such as (for genomic and biological data analysis) host a plethora of R packages
  45. 45. Where to Get R ✤ Comprehensive R Archive Network (CRAN)! ✤ Linux, Mac OS X, and Windows
  46. 46. Getting Started ✤ If you think of R as having a programming language instead of being one, you’ll be better off in the long run! ✤ It’s a case-sensitive, interpreted language! ✤ Enter commands one at a time via the command line, or sets of commands from a source file! ✤ Wide variety of data types, including vectors, matrices, data frames (similar to recordsets), and lists (collections of objects)! ✤ An object is basically anything that can be assigned a value
  47. 47. demo()
  48. 48. RStudio
  49. 49. RStudio’s Interface
  50. 50. text-file.R
  51. 51. Some Important R Functions ✤ Access help file
  52. 52. Some Important R Functions ✤ Search help files
  53. 53. Some Important R Functions ✤ Get arguments
 ✤ See code
  54. 54. R Reference Card –
  55. 55. Entering Input ✤ At the R prompt we type expressions. The <- symbol is the assignment operator
 ✤ The grammar of the language determines whether an expression is complete or not
 ✤ The # character indicates a comment. Anything to the right of the # (including the # itself) is ignored
  56. 56. Evaluation ✤ When a complete expression is entered at the prompt, it is evaluated and the result of the evaluated expression is returned. The result may be auto-printed.
 ✤ The [1] indicates that x is a vector and 5 is the first element
  57. 57. Printing ✤ The : operator is used to create integer sequences
  58. 58. Objects ✤ R has five basic or “atomic” classes of objects:! ✤ character! ✤ numeric (real numbers)! ✤ integer! ✤ complex! ✤ logical (True/False)
  59. 59. Objects ✤ The most basic object is a vector! ✤ Vectors are one-dimensional arrays that can hold data! ✤ A vector can only contain objects of the same class! ✤ The one exception is a list, which is represented as a vector but can contain objects of different classes (that’s usually why they’re used)! ✤ Empty vectors can be created with the vector() function
  60. 60. Numbers ✤ Numbers in R are generally treated as numeric objects (i.e., double precision real numbers)! ✤ If you explicitly want an integer, you need to specify the L suffix! ✤ For example, entering 1 gives you a numeric object; entering 1L explicitly gives you an integer! ✤ There is also a special number Inf which represents infinity; e.g., 1 / 0; Inf can be used in ordinary calculations; e.g., 1 / Inf is 0! ✤ The value NaN represents an undefined value (“not a number”); e.g., 0 / 0; NaN can also be thought of as a missing value
  61. 61. Attributes ✤ R objects can have attributes! ✤ names, dimnames! ✤ dimensions (e.g., matrices, arrays)! ✤ class! ✤ length! ✤ other user-defined attributes/metadata! ✤ Attributes of an object can be accessed using the attributes() function
  62. 62. Vectors ✤ Vectors are one-dimensional arrays that can hold numeric data, character data, or logical data. The combine function c() is used to form the vector.! ! ! ✤ You can also use the vector() function

  63. 63. Vectors ✤ You can refer to elements of a vector using a numeric vector of positions within brackets. For example, a[c(2, 4)] refers to the second and fourth elements of vector a
  64. 64. Mixing Objects ✤ What about the following?
 ✤ When different objects are mixed in a vector, coercion occurs so that every element in the vector is of the same class
  65. 65. Explicit Coercion ✤ Objects can be explicitly coerced from one class to another using the as.* functions, if available
  66. 66. Explicit Coercion ✤ Nonsensical coercion results in NAs
  67. 67. Matrices ✤ A matrix is a two-dimensional array. It has a dimension attribute which is an integer vector of length 2 (nrow, ncol)
  68. 68. Matrices ✤ Matrices are constructed column-wise, so entries can be thought of starting in the “upper left” corner an running down the columns
  69. 69. Matrices ✤ Matrices can also be created directly from vectors by adding a dimension attribute
  70. 70. cbind-ing and rbind-ing ✤ Matrices can also be created by column-binding or row- binding with cbind() and rbind()
  71. 71. Arrays ✤ Arrays are similar to matrices, however they can have more than two dimensions
  72. 72. Factors ✤ Variables can be described as nominal, ordinal, or continuous! ✤ Nominal, categorical, without an implied order (e.g., blood type)! ✤ Ordinal, also categorical and imply order but not amount (e.g., grades)! ✤ Continuous, take on any value within some range, and both order and amount are implied (e.g., age)! ✤ Categorical (nominal) and ordered categorical (ordinal) variables in R are called factors! ✤ Factors are crucial in R because they determine how data is analyzed and presented visually
  73. 73. Factors ✤ The factor() function stores the categorical values as a vector of integers in the range [1… k], (where k is the number of unique values in the nominal variable) and an internal vector of character strings (the original values) mapped to these integers
  74. 74. Factors ✤ By default, factor levels for character vectors are created in alphabetical order and can be overridden by specifying a levels option
  75. 75. Lists ✤ Lists are the most complex of the R data types. Basically, a list is an ordered collection of objects. A list allows you to gather a variety of (possibly unrelated) objects under one name. For example, a list may contain a combination of vectors, matrices, data frames, and even other lists.
  76. 76. Data Frames ✤ A data frame is more general than a matrix in that columns can contain different modes of data (numeric, character, etc.). They store tabular data. These are the most common data types in R.! ✤ They are represented as a special type of list where every element of the list has to have the same length! ✤ Each element of the list can be thought of as a column and the length of each element of the list is the number of rows! ✤ Unlike matrices, data frames can store different classes of objects in each column (just like lists); matrices must have every element be the same class! ✤ Data frames also have a special attribute called row.names! ✤ Data frames are usually created by calling read.table() or read.csv()! ✤ Can be converted to a matrix by calling data.matrix()
  77. 77. Data Frames ✤ Each column must have only one mode, but you can put columns of different modes together
  78. 78. Names ✤ R objects can also have names, which is very useful for writing readable code and self-describing objects
  79. 79. Names ✤ Lists can also have names
  80. 80. Names ✤ And matrices
  81. 81. MissingValues ✤ Missing values are denote by NA or NaN for undefined mathematical operations! ✤ is used to test objects if they are NA! ✤ is.nan() is used to test objects for NaN! ✤ NA values have a class also, so there are integer NA, character NA, etc.! ✤ A NaN value is also NA but the converse is not true
  82. 82. MissingValues
  83. 83. DataTypes Summary ✤ atomic classes: numeric, logical, character, integer, complex! ✤ vectors! ✤ matrices! ✤ arrays! ✤ lists! ✤ data frames! ✤ names! ✤ missing values
  84. 84. DataTypes Summary
  85. 85. Oddities ✤ The period (.) has no special significant in object names. The dollar sign ($) has a somewhat similar meaning to the period in other object-oriented languages and can be used to identify the parts of a data frame or list (e.g., mydata $columnname)! ✤ R don’t have multiline or block comments (must use the #)! ✤ Assigning a value to a nonexistent element of a vector, matrix, array, or list expands that structure to accommodate the new value! ✤ R doesn’t have scalar values, they’re represented as one-element vectors! ✤ Indices start at 1, not 0! ✤ Variables can’t be declared, they come into existence on the first assignment
  86. 86. Helpful Methods ✤ getwd()! ✤ setwd(“some/directory”)! ✤ list.files()! ✤ ls() or objects()! ✤ rm(object/objectlist)! ✤ rm(list=ls())! ✤ edit(objectname)! ✤ source(“filename”)! ✤ sink(“filename.ext”, append=TRUE, split=TRUE)! ✤ data() ~ list available data sets! ✤ search() ~ list loaded packages! ✤ q()
  87. 87. Reading Data ✤ There are a few principal functions reading data into R! ✤ read.table, read.csv, for reading tabular data! ✤ readLines, for reading lines of a text file! ✤ source, for reading in R code files (inverse of dump)! ✤ dget, for reading in R code files (inverse of dput)! ✤ load, for reading in saved workspaces! ✤ unserialize, for reading single R objects in binary form
  88. 88. Writing Data ✤ There are similar functions for writing data to files! ✤ write.table! ✤ writeLines! ✤ dump! ✤ dput! ✤ save! ✤ serialize
  89. 89. read.table ✤ The read.table function is one of the most commonly used functions for reading data! ✤ It has a few important arguments:! ✤ file, the name of the file, or a connection! ✤ header, logical indicating if the file has a header line! ✤ sep, a string indicating how the columns are separated! ✤ colClasses, a character vector indicating the class of each column in the dataset! ✤ nrows, the number of rows in the dataset! ✤ comment.char, a character string indicating the comment character! ✤ skip, the number of lines to skip from the beginning! ✤ stringsAsFactors, should character variables be coded as factors?
  90. 90. read.table ✤ For small to moderately sized datasets, you can usually call read.table without specifying any other arguments
 ✤ R will automatically! ✤ skip lines that begin with #! ✤ figure out how many rows there are (and how much memory needs to be allocated)! ✤ figure what time of variable is in each column of the table (telling R all these things directly makes R run much faster and more efficiently)! ✤ read.csv is identical to read.table except that the default separator is a comma
  91. 91. Larger Datasets ✤ With much larger datasets, doing the following things will make your life easier and will prevent R from choking! ✤ Read the help page for read.table, which contains many hints! ✤ Make a rough calculation of the memory required to store your dataset! ✤ If the dataset is larger than the amount of RAM on your computer, you can probably stop right here! ✤ Set comment.char = “” if there are no commented lines in your file
  92. 92. Larger Datasets ✤ Use the colClasses argument! ✤ Specifying this option instead of using the default can make read.table run much faster, often twice as fast! ✤ In order to use this option, you have to know the class of each column in your data farm! ✤ If all of the columns are numeric, for example, then you can just set colClasses = “numeric”! ✤ Set nrows! ✤ This doesn’t make R run faster but it helps with memory usage
  93. 93. KnowThy System ✤ In general, when using R with larger datasets, it’s useful to know a few things about your system! ✤ How much memory is available?! ✤ What other applications are in use?! ✤ Are there other users logged into the same system?! ✤ What is the Operating System?! ✤ Is the OS 32 or 64 bit?
  94. 94. Calculating Memory Requirements ✤ I have a data frame with 1,500,000 rows and 120 columns, all of which are numeric data. Roughly, how much memory is required to store this data frame? Note: each alphanumeric character requires 8 bits (1 byte) of memory to store.! ✤ 1,500,000 rows * 120 columns = 180,000,000 alphanumeric characters! ✤ 180,000,000 * 8 bits = 1,440,000,000 bytes! ✤ 1,440,000,000 bytes / 1024 = 1,406,250 kb! ✤ 1,406,250 kb / 1024 = 1,373.29 MB! ✤ 1,373.29 MB / 1024 = 1.34 GB! ✤ If you only have 2 GB of RAM, you may want to think twice before doing it. If you've got 4, 6, or 8+ GB of RAM, you should be fine.
  95. 95. Textual Formats ✤ dumping and dputing are useful because the resulting textual format is editable, and in the case of corruption, potentially recoverable! ✤ Unlike writing out a table or csv file, dump and dput preserve the metadata (sacrificing some readability), so that another user doesn't have to specify it all over again! ✤ Can work much better with version control programs like SVN or Git which can only track changes meaningfully in text files! ✤ Can be longer-lived; if there is corruption somewhere in the file, it can be easier to fix the problem! ✤ Downside: Format is not very space-efficient
  96. 96. dput-ting R Objects ✤ Another way to pass data around is by deparsing the R object with dput and reading it back in using dget
  97. 97. Dumping R Objects ✤ Multiple objects can be deparsed using the dump function and read back in using source
  98. 98. Interfaces to the OutsideWorld ✤ Data are read in using connection interfaces! ✤ Connections can be made to files (most common) or to other more exotic things! ✤ file, opens a connection to a file! ✤ gzfile, opens a connection to a file compressed with gzip! ✤ bzfile, opens a connection to a file compressed with bzip2! ✤ url, opens a connection to a webpage
  99. 99. File Connections ✤ description is the name of the file! ✤ open is a code indicating! ✤ “r” read only! ✤ “w” writing (and initializing a new file)! ✤ “a” appending! ✤ “rb”, “wb”, “ab” reading, writing, or appending in binary mode (Windows)
  100. 100. Connections ✤ In general, connections are powerful tools that let you navigate files or other external objects! ✤ In practice, we often don’t need to deal with the connection interface directly
  101. 101. Reading Lines of aText File ✤ readLines can be useful for reading in lines of webpages
  102. 102. Connecting and Listing Databases
  103. 103. Connecting and Listing Databases
  104. 104. R Packages ✤ When you download R from the Comprehensive Archive Network (CRAN), you get the “base” R system! ✤ The base R system comes with basic functionality and implements the R language! ✤ One reason R is so useful is the large collection of packages that extend the basic functionality of R! ✤ R packages are developed and published by the larger R community
  105. 105. Obtaining R Packages ✤ The primary location for obtaining R packages is CRAN! ✤ For biological applications, many packages are available from the Bioconductor Project! ✤ You can obtain information about the available packages on CRAN with the available.packages() function
  106. 106. Obtaining R Packages ✤ There are over 6,000 packages on CRAN covering a wide range of topics! ✤ A list of some topics is available through the Task Views link (, which groups together many R packages related to a given topic
  107. 107. Installing an R Package ✤ Packages can be installed with the install.packages() function in R! ✤ To install a single package, pass the name of the package to the install.packages() function as the first argument! ✤ The following code installs the zipcode package from CRAN ✤ This command downloads the zipcode package from CRAN and installs it on your computer! ✤ Any packages on which this package depends will also be downloaded and installed
  108. 108. Installing Multiple R Packages ✤ You can install multiple R packages at once with a single call to install.packages()! ✤ Place the names of the R packages in a character vector
  109. 109. Installing Packages via RStudio
  110. 110. Installing Packages via RStudio
  111. 111. Loading R Packages ✤ Installing a package does not make it immediately available to your in R; you must load the package! ✤ The library() function is used to load packages into R! ✤ The following code is used to load the ggplot2 package into R
  112. 112. Loading R Packages ✤ Any packages that need to be loaded as dependencies will be loaded first, before the named package is loaded! ✤ Note: Do not put the package name in quotes!! ✤ Some packages produce messages when they are loaded (but some don’t)
  113. 113. Loading R Packages ✤ After loading a package, the functions exported by that packages will be attached to the top of the search() list (after the workspace)
  114. 114. example()
  115. 115. R Packages Summary ✤ R packages provide a powerful mechanism for extending the functionality of R! ✤ R packages can be obtained from CRAN or other repositories! ✤ The install.packages() can be used to install packages at the R console! ✤ The library() function loads packages that have been installed so that you may access the functionality in the packages
  116. 116. Learning Resources ✤ The R Manuals! ✤ QuickR ! ✤ R Mailing Lists! ✤ SwiRl! ✤ Introduction to Data Science! ✤ Johns Hopkins Data University Data Science Specialization
  117. 117. More Learning Resources ✤ R By Example! ✤ R Data Import/Export Manual! ✤ Data Analysis with R: Visually Analyze and Summarize Data Sets! ✤ Intro to Descriptive Statistics: Mathematics for Understanding Data! ✤ Intro to Inferential Statistics: Making Predictions from Data
  118. 118. Good Stuff ✤ FiveThirtyEight ! ✤ Kaggle! ✤ Figshare! ✤ OpenCPU! ✤ RPy! ✤ rCharts! ✤ Facebook Data Science Research Service! ✤ Google’s R Style Guide
  119. 119. Books ✤ Chambers. Software for Data Analysis. 
 Springer, 2008.! ✤ Matloff. The Art of R Programming: A Tour of Statistical Software Design.
 No Starch Press, 2011.! ✤ Kabacoff. R in Action: Data analysis and graphics with R. 
 Second Edition. Manning Publications, 2015.! ✤ Cielen & Meysman. Introducing Data Science. 
 Manning Publications, 2015.! ✤ Tuckey. Exploratory Data Analysis. 
 Pearson, 1977.
  120. 120. Books ✤ Manning Publications Discount Code
 ctwdevob! ✤ Packt Discount Code
  121. 121. 01