Data manipulation and visualization in r 20190711 myanmarucsy

1. Data Manipulation and Visualization in R Hyebong Choi School of ICT and Global Entrepreneurship Handong Global University

2. Big Data: Why it matters 2  In the past, data(record) are only for important people and events  e.g. royal family, war records, …

3. Big Data: Why it matters 3  In the past, data(record) are expensive and for very important event or a few privileged people  e.g. royal family, war records, …  Now data is for everyone and every moment

4. Data easily go beyond Human and … 4

5. Data easily go beyond Human and computer 5 Moore’s Law is a computing term which originated around 1970; the simplified version of this law states that processor speeds, or overall processing power for computers will double every two years.

7. What we call Big Data 7 3V definition

8. Data Science  Data Science aims to derive knowledge fr om big data, efficiently and intelligently  Data Science encompasses the set of acti vities, tools, and methods that enable da ta-driven activities in science, business, medicine, and government  Machine Learning(or Data Mining) is agai n one of the core technologies that enabl es Data Science http://www.oreilly.com/data/free/what-is-data-science.csp 8 Data Science is process to get Actionable Insight from Big DATA

9. Night time Bus by Seoul Local Government  Effective Bus Route Design with Big Data  5 Million Records of Taxi Take on/off  3 Billion Night time call records(location) from telco company

10. Singapore Traffic visualization  Data from LTA  Subway, bus, taxi traffic info.  By data analytics team in Institute for Infocomm Research

11. Big data and Data Science

12. What We Cover Here • Data Manipulation and Visualization in R 12

13. R • Most common tools for data scientist other than DBMS • Cover wide range of data scientist – Data engineer, Statistician, Data main expert, … rather than just computer engineer • Providing thousands of ready-to-use powerful packages for data scientist • Well documented 13 http://blog.revolutionanalytics.com/2014/01/in-data-scientist-survey-r-is-the-most-used-tool-other-than-databases.html

14. R How to Start • Explanation of windows in R studio 14

15. R • Getting help ▫ help(command) or ?command ▫ example(command) to see examples 15

16. Package Installation and loading • install.packages(“package name”) • to load package ▫ library(“package name”) ▫ or require(“package name”) 16

17. Variable • Variable is a container to hold data (or information) that we want to work with • Variable can hold ▫ a single value: 10, 10.5, “abc”, factor, NA, NULL ▫ multiple values: vector, matrix, list ▫ specially formatted data (values): data.frame 17

18. How you assign a value to variable • var <- value 18 my_first_variable <- 35.121 New variable is now assigned and available in working environment

19. Operators 19 a <- 10.5 b <- 20 c <- 4 a + b ## addition ## [1] 30.5 a - c ## substraction ## [1] 6.5 a * c ## mulitiplication ## [1] 42 b / c ## division ## [1] 5 a %% c ## remainder ## [1] 2.5 a > b ## inequality ## [1] FALSE a*2 == b ## equality ## [1] FALSE !(a > b) ## negation ## [1] TRUE (b > a) & (b > c) ## logical AND ## [1] TRUE (a > b) | (a > c) ## logical OR ## [1] TRUE

20. Data Type – Missing Value (NA) • Sometimes values are missing, and R represent the missing values as NAs 20

21. Vector • A vector is a sequence of data elements of the same basic type. • All members should be of same data type 21 numeric_vector <- c(1, 10, 49) character_vector <- c("a", "b", "c") boolean_vector <- c(TRUE, FALSE, TRUE) typeof(numeric_vector) ## [1] "double" typeof(character_vector) ## [1] "character" typeof(boolean_vector) ## [1] "logical" length(numeric_vector) ## number of members in the vector ## [1] 3 new_vector <- c(numeric_vector, 50) new_vector ## [1] 1 10 49 50

22. Vector • R’s vector index starts from 1 ▫ 1,2,3,4, … • Minus Index means “except for” 22

23. Vector with named elements • We can give name to each element of vector • and we can use the name instead of index number 23 some_vector <- c("John Doe", "poker player") names(some_vector) <- c("Name", "Profession") some_vector ## Name Profession ## "John Doe" "poker player" some_vector['Name'] ## Name ## "John Doe" some_vector['Profession'] ## Profession ## "poker player" some_vector[1] ## Name ## "John Doe"

24. Vector with named elements • We can give name to each element of vector • and we can use the name instead of index number 24 weather_vector <- c("Mon" = "Sunny", "Tues" = "Rainy", "Wed" = "Cloudy", "Thur" = "Foggy", "Fri" = "Sunny", "Sat" = "Sunny", "Sun" = "Cloudy") weather_vector ## Mon Tues Wed Thur Fri Sat Sun ## "Sunny" "Rainy" "Cloudy" "Foggy" "Sunny" "Sunny" "Cloudy“ names(weather_vector) ## [1] "Mon" "Tues" "Wed" "Thur" "Fri" "Sat" "Sun"

25. Short-cut to make numeric vector 25 a_vector <- 1:10 ## numbers from 1 to 10 b_vector <- seq(1, 10, 2) ## numbers from 1 to 10 increasing by 2 a_vector ## [1] 1 2 3 4 5 6 7 8 9 10 b_vector ## [1] 1 3 5 7 9 c_vector <- rep(1:3, 3) d_vector <- rep(1:3, each = 3) c_vector ## [1] 1 2 3 1 2 3 1 2 3 d_vector ## [1] 1 1 1 2 2 2 3 3 3 c(a_vector, b_vector) ## combine vectors to single vector ## [1] 1 2 3 4 5 6 7 8 9 10 1 3 5 7 9

26. Basic Vector operations 26 a_vector <- c(1,5,2,7,8) b_vector <- seq(1, 10, 2) sum(a_vector) ## summation ## [1] 23 mean(a_vector) ## average ## [1] 4.6 # operation of Vector and Scala a_vector + 10 ## [1] 11 15 12 17 18 a_vector > 4 ## [1] FALSE TRUE FALSE TRUE TRUE sum(a_vector > 4) ## what does this mean? ## [1] 3 # operation of Vector and Vector a_vector - b_vector ## [1] 0 2 -3 0 -1 a_vector == b_vector ## [1] TRUE FALSE FALSE TRUE FALSE sum(a_vector == b_vector) ## what does this mean? ## [1] 2

27. Vector Indexing (Selection) 27 sample_vector <- c(1, 4, NA, 2, 1, NA, 4, NA) ## vector with some missing values sample_vector[1:5] ## [1] 1 4 NA 2 1 sample_vector[c(1,3,5)] ## [1] 1 NA 1 sample_vector[-1] ## [1] 4 NA 2 1 NA 4 NA sample_vector[c(-1, -3, -5)] ## [1] 4 2 NA 4 NA sample_vector[c(T, T, F, T, F, T, F, T)] ## [1] 1 4 2 NA NA is.na(sample_vector) ## [1] FALSE FALSE TRUE FALSE FALSE TRUE FALSE TRUE sum(is.na(sample_vector)) ## [1] 3 ## can you select non-NA elements from the vector? Selection by numeric vector Selection by logical vector

28. Data Frame • Very commonly datasets contains variables of different kinds ▫ e.g. student dataset may contain name(character), age(integer), major(factor), gpa(numeric, real number)… • Vector and metric can have values of same data type • A data frame has the variables of a data set as columns and the observations as rows. 28 mtcars ## mpg cyl disp hp drat wt qsec vs am gear carb ## Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 ## Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 ## Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1 ## Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 ## Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 ## Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1 ## Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4 ## Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2 ## Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2 ## Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4 ## Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4 ## Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3

29. Overviewing of Data frame • head functions shows the first n (6 by default) observation of dataframe • tail functions shows the last n (6 by default) observation of dataframe 29 head(mtcars) ## mpg cyl disp hp drat wt qsec vs am gear carb ## Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4 ## Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4 ## Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1 ## Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1 ## Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2 ## Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1 head(mtcars, 10) ## try to see what happens tail(mtcars) ## mpg cyl disp hp drat wt qsec vs am gear carb ## Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.7 0 1 5 2 ## Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.9 1 1 5 2 ## Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.5 0 1 5 4 ## Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.5 0 1 5 6 ## Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.6 0 1 5 8 ## Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.6 1 1 4 2 tail(mtcars, 10) ## try to see what happens

30. Overviewing of Data frame • str function shows the structure of your data set, it tells you ▫ The total number of observations (e.g. 32 car types) ▫ The total number of variables (e.g. 11 car features) ▫ A full list of the variables names (e.g. mpg, cyl ... ) ▫ The data type of each variable (e.g. num) ▫ The first few observations 30 str(mtcars) ## 'data.frame': 32 obs. of 11 variables: ## $ mpg : num 21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ... ## $ cyl : num 6 6 4 6 8 6 8 4 4 6 ... ## $ disp: num 160 160 108 258 360 ... ## $ hp : num 110 110 93 110 175 105 245 62 95 123 ... ## $ drat: num 3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ... ## $ wt : num 2.62 2.88 2.32 3.21 3.44 ... ## $ qsec: num 16.5 17 18.6 19.4 17 ... ## $ vs : num 0 0 1 1 0 1 0 1 1 1 ... ## $ am : num 1 1 1 0 0 0 0 0 0 0 ... ## $ gear: num 4 4 4 3 3 3 3 4 4 4 ... ## $ carb: num 4 4 1 1 2 1 4 2 2 4 ...

31. Creating Data frame • data.frame function with vectors (of same length and possibly different type) makes you a data frame 31 # Definition of vectors name <- c("Mercury", "Venus", "Earth", "Mars", "Jupiter", "Saturn", "Uranus", "Neptune") type <- c("Terrestrial planet", "Terrestrial planet", "Terrestrial planet", "Terrestrial planet", "Gas giant", "Gas giant", "Gas giant", "Gas giant") diameter <- c(0.382, 0.949, 1, 0.532, 11.209, 9.449, 4.007, 3.883) rotation <- c(58.64, -243.02, 1, 1.03, 0.41, 0.43, -0.72, 0.67) rings <- c(FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE) # Create a data frame from the vectors planets_df <- data.frame(name, type, diameter, rotation, rings) planets_df ## name type diameter rotation rings ## 1 Mercury Terrestrial planet 0.382 58.64 FALSE ## 2 Venus Terrestrial planet 0.949 -243.02 FALSE ## 3 Earth Terrestrial planet 1.000 1.00 FALSE ## 4 Mars Terrestrial planet 0.532 1.03 FALSE ## 5 Jupiter Gas giant 11.209 0.41 TRUE ## 6 Saturn Gas giant 9.449 0.43 TRUE ## 7 Uranus Gas giant 4.007 -0.72 TRUE ## 8 Neptune Gas giant 3.883 0.67 TRUE

32. Creating Data frame • you may specify the variables as parameters 32 my.df <- data.frame(name = c('John', 'Kim', 'Kaith'), job = c('Teacher', 'Policeman', 'Secertary'), age = c(32, 25, 28)) my.df ## name job age ## 1 John Teacher 32 ## 2 Kim Policeman 25 ## 3 Keith Secretary 28

33. Selection of data frame elements Similar to vectors and matrices, you select elements from a data frame with the help of square brackets [ ]. By using a comma, you can indicate what to select from the rows and the columns respectively. 33 # Print out diameter of Mercury (row 1, column 3) planets_df[1,3] ## [1] 0.382 # Print out data for Mars (entire fourth row) planets_df[4, ] ## name type diameter rotation rings ## 4 Mars Terrestrial planet 0.532 1.03 FALSE # you can use of directly variable name # Select first 5 values of diameter column planets_df[1:5, 'diameter'] ## [1] 0.382 0.949 1.000 0.532 11.209

34. Selection of data frame elements You will often want to select an entire column, namely one specific variable from a data frame. If you want to select all elements of the variable diameter, for example, both of these will do the trick: planets_df[,3] planets_df[,"diameter"] However, there is a short-cut. If your columns have names, you can use the $ sign: planets_df$diameter 34

35. Selection of data frame elements - a tricky part • You can use a logical vector to select from data frame 35 ## find planets with rings planets_df[planets_df$rings, ] ## name type diameter rotation rings ## 5 Jupiter Gas giant 11.209 0.41 TRUE ## 6 Saturn Gas giant 9.449 0.43 TRUE ## 7 Uranus Gas giant 4.007 -0.72 TRUE ## 8 Neptune Gas giant 3.883 0.67 TRUE ## select names of planets with rings planets_df[planets_df$rings, 'name'] ## [1] Jupiter Saturn Uranus Neptune ## Levels: Earth Jupiter Mars Mercury Neptune Saturn Uranus Venus ## find planets with larger diameter than earth planets_df$diameter > 1 ## [1] FALSE FALSE FALSE FALSE TRUE TRUE TRUE TRUE planets_df[planets_df$diameter > 1, ] ## name type diameter rotation rings ## 5 Jupiter Gas giant 11.209 0.41 TRUE ## 6 Saturn Gas giant 9.449 0.43 TRUE ## 7 Uranus Gas giant 4.007 -0.72 TRUE ## 8 Neptune Gas giant 3.883 0.67 TRUE

36. Handling Big Data using R <2st> Package “dplyr”, essential for data preprocessing in R 36 HyunJae Jo School of ICT and Global Entrepreneurship Handong Global University

37. Contents • Data preprocessing • dplyr packages • Lab example 37

38. Why we have to do “Data preprocessing” • Data preprocessing is the process of making raw data suitable for data analysis. 38 • Data Cleaning  Correction of missing value, outlier and noisy data • Data Integration  Combining data from multiple sources • Data Reduction  Reduce only the data needed for analysis • Data Transformation  Data transformation to maximize efficiency of data analytics

39. Data Example of “Data Integration” 39 Missing value Noisy data Outlier 1. Data Integration  Integrating “Female Population” Column “Data preprocessing”

40. Example of “Data Cleaning” 40 Missing value Noisy data Outlier “Data preprocessing” 2. Data Cleaning  Correction “Male Pop.” column  Remove missing value(“others” column)  Correction outliers

41. Example of “Data Reduce” 41 Missing value Noisy data Outlier “Data preprocessing” 3. Data Reduce  Top 5 Population Regions by Total Population

42. Example of “Data Transformation” 42 Missing value Noisy data Outlier “Data preprocessing” 4. Data Transfromation  Ratio region population to total population

43. What is “dplyr” • "dplyr" is an R package that specializes in data processing. • Functions of “dplyr” ▫ mutate() adds new variables that are functions of existing variables ▫ select() picks variables based on their names. ▫ filter() picks cases based on their values. ▫ summarise() reduces multiple values down to a single summary. ▫ arrange() changes the ordering of the rows. ▫ group_by() allows to perform any operation “by group” ▫ %>%(chain) connect each operation and perform it at once 43

44. How we can use “dplyr” • Install packages and load packages. • If you want to see package manual in R, you can use “help” function 44

45. imdb dataset • IMDb (Internet Movie Database) is an online database of information related to films, television programs, home videos and video games, and streaming content online -- including cast, production crew and personnel biographies, plot summaries, trivia, and fan reviews and ratings. As of October 2018, IMDb has approximately 5.3 million titles (including episodes) and 9.3 million personalities in its database, as well as 83 million registered users. (Wikipedia) • We use reduced imdb dataset and you can load imdb dataset using read.csv() function with URL link: • https://raw.githubusercontent.com/myhan0710/prac_dplyr/master/imdb.csv 45

46. Overviewing: imdb dataset 46 (e.g. 15,190 movies) (e.g. 44 related variable) (e.g. fn, tid, title,wordsInTitle, …) (e.g. chr, num, int)

47. Lab Example • We will now look at the functions of the dplyr library through the imdb dataset. • Familiarize yourself with the roles of the given functions, and just enter the code in your RStudio. • Through the Lab, we can finally obtain data from the imdb dataset that meet the following conditions: 47

48. dplyr: filter 48 • Use filter() function to extract specific row that fits the condition. • First, we can find recent, high rating and • many rating counts movie like this:

49. dplyr: mutate 49 • Use mutate() function to make new column. • Second, we can make column that indicates whether a movie contains action and adventure genres: • If a certain movie’s Action column value is 1 and Adventure column value is 0, it means that movie includes action genre, but not Advenutre genre. • So, ActionAdven column value 1 means a movie includes both action and adventure genres.

50. dplyr: select • Use select() function to extract specific column that fits the condition. • We only need a few columns: - wordsInTitle - imdbRating - ratingCount - Year - ActionAven 50

51. dplyr: arrange 51 • Use the arrange() function to sort data from small to large values based on a specified column. • Fourth, we would like to sort the year in ascending order and then grade in descending order. • desc() function allows it to be sorted in descending order.

52. dplyr: summarise • Use the summarise() function to obtain basic statistics by specifying functions such as mean(), var(), and median() • Now, we want to know average rating, average year, and number of Action-Adventure genre. 52

53. dplyr: group_by • Use the group_by() function, then you can group data by level in the specified column. • After group_by() function, You can associate the results with summarise() to view the results of each level. 53

54. dplyr: %>% • Use the chain function(%>%) to write a code with all functions at once. • Then, the results are same. (imdb_chain, imdb7) 54

55. All contents 55

56. Excercise ① Extract movies from the original imdb dataset between 1970 and 2000. ② Using chain function, extract movies that has both drama and family genre. Then, columns are only year, duration, nrOfphotos. • Please replace <Fill In> with your solution. • After replacing your solution, check your solution is right on the next page. 56

57. Exercise key ① Key ② Key 57

58. 58

59. Data Visualization • Essential component of skill set as a data scientist • With ever increasing volume of data, it is impossible to tell stories without visualizations. • Data visualization is an art of how to turn “Big Data” into useful knowledge 59

60. Data Visualization • Data Visualization 60 Statistics Design Graphical Data Analysis Communication & Perception

61. Data Visualization • Exploratory Visualization ▫ Help you see what is in the data • Explanatory Visualization ▫ Shows others what you’ve found in your data • R supports both types of visualizations 61

62. Data Visualization • Exploratory Visualization ▫ Help you see what is in the data ▫ Keep as much as detail as possible ▫ Practical Limit: how much can you see and interpret • Explanatory Visualization ▫ Help us share our understanding with others ▫ Shows others what you’ve found in your data ▫ Requires editorial decisions: ▫ Highlight the key features you want to emphasize ▫ Eliminate extraneous details 62

63. Exploratory Visualzation 63

64. Explanatory Visualization 64

65. ggplot2 • Author: Hadley Wickham • Open Source implementation of the layered grammar of graphics • High-level R package for creating publication-quality statistical graphics ▫ Carefully chosen defaults following basic graphical design rules • Flexible set of components for creating any type of graphics • Things you cannot do With ggplot2 ▫ 3-dimensional graphics ▫ Graph-theory type graphs (nodes/edges layout) 65

66. ggplot2 installation • In R console: install.packages("ggplot2") library(ggplot2) 66

67. Toy examples 67

68. Grammar of Graphics 68

69. The quick brown fox jumps over the lazy dog 69

70. Grammar of Graphics 70 • Plotting Framework • Leland Wilkinson, Grammar of Graphics, 1999 • 2 principles ▫ Graphics = distinct layers of grammatical elements ▫ Meaningful plots through aesthetic mapping

71. Essential Grammatical Elements 71 Element Description Data The dataset being plotted. Aesthetics The scales onto which we map our data. Geometries The visual elements used for our data.

72. All Grammatical Elements 72 Element Description Data The dataset being plotted. Aesthetics The scales onto which we map our data. Geometries The visual elements used for our data. Facets Plotting small multiples Statistics Representations of our data to aid understanding. Coordinates The space on which the data will be plotted. Themes All non-data ink.

73. Diagram 73

74. Grammar of Graphics • Building blocks • Solid, creative, meaningful visualizations • Essential Layers: Data, Aesthetics, Geometries • For Enhancement: Facets, Statistics, Coordinates, Themes 74

75. Example – First trial of ggplot • To get a first feel for ggplot2, let's try to run some basic ggplot2 commands. Together, they build a plot of the mtcars dataset that contains information about 32 cars from a 1973 Motor Trend magazine. This dataset is small, intuitive, and contains a variety of continuous and categorical variables. 75

76. Example – First trial of ggplot • The plot from the previous exercise wasn't really satisfying. Although cyl (the number of cylinders) is categorical, it is classified as numeric in mtcars. You'll have to explicitly tell ggplot2 that cyl is a categorical variable. 76

77. more examples 77

86. Practice 86 https://gist.githubusercontent.com/tiangechen/b68782efa49a1 6edaf07dc2cdaa855ea/raw/0c794a9717f18b094eabab2cd6a6b9a2269 03577/movies.csv

87. Practice 87

88. Practice 88

89. Practice 89

90. Practice 90 https://raw.githubusercontent.com/meKIDO/MyanmarData/master/MichelinNY.csv'

91. Practice 91

92. Practice 92

93. Practice 93

94. Practice 94

95. Practice 95 https://raw.githubusercontent.com/meKIDO/MyanmarData/master/M yanmarDB.csv

96. Practice 96

97. Thank you 97

Data manipulation and visualization in r 20190711 myanmarucsy

Estás leyendo una previsualización.

Eche un vistazo a continuación

Data manipulation and visualization in r 20190711 myanmarucsy

Más Contenido Relacionado

Presentaciones para usted (19)

Similares a Data manipulation and visualization in r 20190711 myanmarucsy (20)

Más reciente (20)