Aims

  • To learn about the categorical data class - factors
  • To understand more complex R data structures like data frames, lists etc
  • To demonstrate the R markdown

Factors

R has a special vector class for dealing with catergorical data - the factor. Categorical data might be something such as sex - “Male”, “Female” - or tumour stage - “Stage1”, “Stage2”, “Stage3”, etc. Critically, the possible entries, or levels, of a factor are limited.

Factors are particularly useful when generating plots or running statistical analyses.

While factors look and behave like character vectors, they are actually stored by R as an integer vector, so it is important to be careful with them when treating them as strings or transforming them to other data types.

Creating a factor

To create a factor we simply use the function factor to turn a standard character vector into a factor.

x <- c("orange", "apple", "apple", "orange")
x
## [1] "orange" "apple"  "apple"  "orange"
fruit <- factor(x) # convert vector x into a factor
fruit
## [1] orange apple  apple  orange
## Levels: apple orange
class(x)
## [1] "character"
class(fruit)
## [1] "factor"

Factors store the categories as “levels”

Once created, factors can only contain a pre-defined set values, known as levels. To examine what levels a particular factor has we can use the command levels:

levels(fruit)
## [1] "apple"  "orange"

Here we can see that fruit has two levels: “apple” and “orange”. These are now the only valid possible values. If we try to change one of the values to something else it will result in a missing value.

fruit[1] <- "apple"
fruit[2] <- "banana"
## Warning in `[<-.factor`(`*tmp*`, 2, value = "banana"): invalid factor level, NA
## generated
fruit
## [1] apple  <NA>   apple  orange
## Levels: apple orange

The values of the factors are actually stored as integers

R is actually storing these factors as integers (1, 2, 3, 4 …). When the factor is printed to the console for our benefit, the integers are replaced with the corresponding level label. In this way factors also represent a data storage solution, they can store large amounts of complex categorical data but only use a small amount of memory. We can see this if we convert the factor to a numeric vector using as.numeric.

x <- c("low", "high", "medium", "high", "low", "medium", "high")
response <- factor(x)
as.numeric(x)
## Warning: NAs introduced by coercion
## [1] NA NA NA NA NA NA NA
response
## [1] low    high   medium high   low    medium high  
## Levels: high low medium
as.numeric(response)
## [1] 2 1 3 1 2 3 1

We can change the order of the levels

For plotting or statistical analysis the order of the levels matters. In plotting the data pertaining to the different levels will be plotted in the order of the levels. In statistical analyses, such as linear modelling, the first level will often be automatically chosen as the reference.

Therefore, we might what to specify the order of the levels. In our current response vector the order of the levels is “high”, “low”, “medium”. We would probably want to switch this to “low”, “medium”, “high”. We can do this in the factor function.

response <- factor(x, levels = c("low", "medium", "high"))
response
## [1] low    high   medium high   low    medium high  
## Levels: low medium high

Converting factors to plain vector types

We can convert factors back to plain vector types - character, integer etc - using the “as” family of functions. as.character will return the original category values. as.numeric will return a vector of the level index for each value.

response
## [1] low    high   medium high   low    medium high  
## Levels: low medium high
as.character(response)
## [1] "low"    "high"   "medium" "high"   "low"    "medium" "high"
as.numeric(response)
## [1] 1 3 2 3 1 2 3

It is important to be careful with this when your categories look like numbers. For example if we have a categorical vector of years and we want to convert this to a numeric vector of years (so that perhaps we can do some arithmetic with it).

x <- c(1987, 2002, 2019, 2004, 1983, 1982, 1999, 2004)
dateOfBirth <- factor(x)
dateOfBirth
## [1] 1987 2002 2019 2004 1983 1982 1999 2004
## Levels: 1982 1983 1987 1999 2002 2004 2019
as.character(dateOfBirth)
## [1] "1987" "2002" "2019" "2004" "1983" "1982" "1999" "2004"
as.numeric(dateOfBirth)
## [1] 3 5 7 6 2 1 4 6

To get the result we want we need to do it in two stages:

dob <- as.character(dateOfBirth)
dob <- as.numeric(dob)
dob
## [1] 1987 2002 2019 2004 1983 1982 1999 2004
age <- 2023 - dob
age
## [1] 36 21  4 19 40 41 24 19

Beyond the atomic vector - matrices, data.frames and lists

Image source:http://venus.ifca.unican.es/Rintro/dataStruct.html

R has many data structures. These include:

  • atomic vector
  • matrix
  • data frame
  • list

Matrices

In R matrices are tables of values with the same data type. They are an extension of the atomic vector.

The matrix can be viewed as a collection of vectors:

  • Matrix rows and columns are vectors
  • All the vectors hold identical data type
  • The vectors must have the same length

Matrices are widely using in statistical analyses and modelling.

y <- matrix(data = 1:12, nrow = 4, ncol = 3)
y
##      [,1] [,2] [,3]
## [1,]    1    5    9
## [2,]    2    6   10
## [3,]    3    7   11
## [4,]    4    8   12
class(y)
## [1] "matrix" "array"
typeof(y)
## [1] "integer"

In contrast to vectors, matrices and data frames are two dimensional data structures.

  • First dimension: rows
  • Second dimension: columns

We can find out about the size of the matrix using the following functions:

  • dim(): Provides the dimensions of an object
  • nrow(): gives number of rows
  • ncol(): gives number of columns
dim(y)
## [1] 4 3
nrow(y)
## [1] 4
ncol(y)
## [1] 3

As with atomic vectors we can select a subset of the values of a matrix using the [] operator. However, as the matrix is a two dimensional data structure you must provide both the rows and the columns.

The general syntax is matrix[ rows , columns ]:

To extract the value in the first row and the second column:

y[1, 2]
## [1] 5

If we want an entire row or an entire column, we can omit the column or row index respectively.

y[1, ]
## [1] 1 5 9
y[, 3]
## [1]  9 10 11 12

As with atomic vectors, we can use a vector within the [] operator to select multiple values:

y[c(1, 3), 2:3]
##      [,1] [,2]
## [1,]    5    9
## [2,]    7   11

Like the atomic vector, matrix can hold only one type of data, if we try to mix two or more different data type, R implicitly convert into one type:

typeof(y)
## [1] "integer"
y
##      [,1] [,2] [,3]
## [1,]    1    5    9
## [2,]    2    6   10
## [3,]    3    7   11
## [4,]    4    8   12
y[2, 3]
## [1] 10
y[2, 3] <- "A"
y
##      [,1] [,2] [,3]
## [1,] "1"  "5"  "9" 
## [2,] "2"  "6"  "A" 
## [3,] "3"  "7"  "11"
## [4,] "4"  "8"  "12"
typeof(y)
## [1] "character"

Lists

R’s simplest structure that combines data of different types is a list. Lists are very flexible data structures that can hold anything. A list is simply a collection of multiple objects. The objects in the list can be of different types and different lengths. A list can even be a collection of lists.

w <- c(TRUE, FALSE)
x <- 1:10
y <- c("a", "b", "c")
z <- matrix(1:12, nrow = 3, ncol = 4)

myList <- list(w, x, y, z)
myList
## [[1]]
## [1]  TRUE FALSE
## 
## [[2]]
##  [1]  1  2  3  4  5  6  7  8  9 10
## 
## [[3]]
## [1] "a" "b" "c"
## 
## [[4]]
##      [,1] [,2] [,3] [,4]
## [1,]    1    4    7   10
## [2,]    2    5    8   11
## [3,]    3    6    9   12

myList has four elements and when printed out like this looks quite strange at first sight. Note how each of the elements of a list is referred to by an index within 2 sets of square brackets. This gives a clue to how you can access individual elements in the list:

  1. []: Standard subscript operator, works like in a vector. The output is still a list.
  2. [[]]: This operator can only take one index number at a time and the output will be the data structure for that element.
length(myList)
## [1] 4
myList[1]
## [[1]]
## [1]  TRUE FALSE
myList[4]
## [[1]]
##      [,1] [,2] [,3] [,4]
## [1,]    1    4    7   10
## [2,]    2    5    8   11
## [3,]    3    6    9   12
myList[1:3]
## [[1]]
## [1]  TRUE FALSE
## 
## [[2]]
##  [1]  1  2  3  4  5  6  7  8  9 10
## 
## [[3]]
## [1] "a" "b" "c"
length(myList)
## [1] 4
myList[[1]]
## [1]  TRUE FALSE
myList[[4]]
##      [,1] [,2] [,3] [,4]
## [1,]    1    4    7   10
## [2,]    2    5    8   11
## [3,]    3    6    9   12
myList[[1:3]]
## Error in myList[[1:3]]: recursive indexing failed at level 2

We can also name the elements in the list. This provides another way to access them:

  1. “$”: to extract an element from a list by name.
namedList <- list(
    city = c("Kampala", "London", "Paris"),
    population = c(1.51, 8.8, 2.1)
)

namedList
## $city
## [1] "Kampala" "London"  "Paris"  
## 
## $population
## [1] 1.51 8.80 2.10
namedList$city
## [1] "Kampala" "London"  "Paris"
namedList$population
## [1] 1.51 8.80 2.10
names(namedList)
## [1] "city"       "population"

You can modify lists either by adding additional elements or modifying existing ones.

namedList$city[2] <- "New York"
namedList$country <- c("Uganda", "USA", "France")
namedList
## $city
## [1] "Kampala"  "New York" "Paris"   
## 
## $population
## [1] 1.51 8.80 2.10
## 
## $country
## [1] "Uganda" "USA"    "France"

Lists can be thought of as a ragbag collection of things without a very clear structure. You probably won’t find yourself creating list objects of the kind we’ve seen above when analyzing your own data. However, the list provides the basic underlying structure to the data frame that we’ll be using throughout the rest of this course.

The other area where you’ll come across lists is as the return value for many of the statistical tests and procedures such as linear regression that you can carry out in R.

To demonstrate, we’ll run a t-test comparing two sets of samples drawn from subtly different normal distributions. We’ll use the rnorm() function for creating random numbers based on a normal distribution.

sample_1 <- rnorm(n = 100, mean = 5, sd = 1)
sample_2 <- rnorm(n = 100, mean = 7, sd = 1)

result <- t.test(x = sample_1, y = sample_2)

is.list(result)
## [1] TRUE
result
## 
##  Welch Two Sample t-test
## 
## data:  sample_1 and sample_2
## t = -12.799, df = 197.57, p-value < 2.2e-16
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -2.143159 -1.570914
## sample estimates:
## mean of x mean of y 
##  5.168136  7.025173
names(result)
##  [1] "statistic"   "parameter"   "p.value"     "conf.int"    "estimate"   
##  [6] "null.value"  "stderr"      "alternative" "method"      "data.name"
result$statistic
##         t 
## -12.79926
result$p.value
## [1] 1.037258e-27

Data frames

The most commmonly used data structure is the data.frame. This can be though of as a table of values. Unlike a matrix each column can have a different data class, but each value in that column must be of the same data class. Although it is easiest to think of the data.frame as a table, and R will present and treat it as a 2 dimensional table, it is actually a special type of list in which all the elements are vectors of the same length.

dat <- data.frame(
    city = c("Kampala", "London", "Paris"),
    population = c(1.51, 8.8, 2.1)
)
dat
##      city population
## 1 Kampala       1.51
## 2  London       8.80
## 3   Paris       2.10
class(dat)
## [1] "data.frame"
dim(dat)
## [1] 3 2

For the purposes of demonstration and testing, R provides many built-in data sets, most of which are represented as data frames. You can list all the avilable built-in data sets with the data() command.

We will look at the iris data set:

  • data frame
  • 150 observations (rows)
  • 5 variables (columns)
    • Sepal.Length
    • Sepal.Width
    • Petal.Length
    • Petal.Width
    • Species

To bring one of these internal data sets to the fore, you can just start using it by name.

head(iris)
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa

You can also get help for a data set such as iris in the usual way.

?iris # get help for iris data

This reveals that iris is a rather famous old data set of measurements taken by the esteemed British statistician and geneticist, Ronald Fisher (he of Fisher’s exact test fame).

Viewing data frames is made easier with these functions:

  • head(): Shows first 6 lines of a data frame
  • tail(): Shows first 6 lines of a data frame
  • View(): Shows data frame in excel like format
head(iris)
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa
tail(iris)
##     Sepal.Length Sepal.Width Petal.Length Petal.Width   Species
## 145          6.7         3.3          5.7         2.5 virginica
## 146          6.7         3.0          5.2         2.3 virginica
## 147          6.3         2.5          5.0         1.9 virginica
## 148          6.5         3.0          5.2         2.0 virginica
## 149          6.2         3.4          5.4         2.3 virginica
## 150          5.9         3.0          5.1         1.8 virginica
View(iris)

A data frame is a special type of list so you can access its columns in the same way as we saw previously for lists.

names(iris)
## [1] "Sepal.Length" "Sepal.Width"  "Petal.Length" "Petal.Width"  "Species"
iris$Petal.Width
##   [1] 0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 0.2 0.2 0.1 0.1 0.2 0.4 0.4 0.3
##  [19] 0.3 0.3 0.2 0.4 0.2 0.5 0.2 0.2 0.4 0.2 0.2 0.2 0.2 0.4 0.1 0.2 0.2 0.2
##  [37] 0.2 0.1 0.2 0.2 0.3 0.3 0.2 0.6 0.4 0.3 0.2 0.2 0.2 0.2 1.4 1.5 1.5 1.3
##  [55] 1.5 1.3 1.6 1.0 1.3 1.4 1.0 1.5 1.0 1.4 1.3 1.4 1.5 1.0 1.5 1.1 1.8 1.3
##  [73] 1.5 1.2 1.3 1.4 1.4 1.7 1.5 1.0 1.1 1.0 1.2 1.6 1.5 1.6 1.5 1.3 1.3 1.3
##  [91] 1.2 1.4 1.2 1.0 1.3 1.2 1.3 1.3 1.1 1.3 2.5 1.9 2.1 1.8 2.2 2.1 1.7 1.8
## [109] 1.8 2.5 2.0 1.9 2.1 2.0 2.4 2.3 1.8 2.2 2.3 1.5 2.3 2.0 2.0 1.8 2.1 1.8
## [127] 1.8 1.8 2.1 1.6 1.9 2.0 2.2 1.5 1.4 2.3 2.4 1.8 1.8 2.1 2.4 2.3 1.9 2.3
## [145] 2.5 2.3 1.9 2.0 2.3 1.8

We can further subset the values in that column to, say, return the first 10 values only.

iris$Petal.Length[1:10]
##  [1] 1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5

As with the matrix, the [, ] syntax can be used to access both rows and columns.

iris[1:4, 1:3]
##   Sepal.Length Sepal.Width Petal.Length
## 1          5.1         3.5          1.4
## 2          4.9         3.0          1.4
## 3          4.7         3.2          1.3
## 4          4.6         3.1          1.5
iris[c(1, 4, 7),  c(2, 4)]
##   Sepal.Width Petal.Width
## 1         3.5         0.2
## 4         3.1         0.2
## 7         3.4         0.3

To extract values from a data frame, row/column names may also be used

iris[c(1, 4, 7),  c("Sepal.Width", "Species" )]
##   Sepal.Width Species
## 1         3.5  setosa
## 4         3.1  setosa
## 7         3.4  setosa

We can also use conditional sub-setting to extract the rows that meet certain conditions, e.g. all the rows with Sepal.Length of 5.

iris[iris$Sepal.Length == 5, ]
##    Sepal.Length Sepal.Width Petal.Length Petal.Width    Species
## 5             5         3.6          1.4         0.2     setosa
## 8             5         3.4          1.5         0.2     setosa
## 26            5         3.0          1.6         0.2     setosa
## 27            5         3.4          1.6         0.4     setosa
## 36            5         3.2          1.2         0.2     setosa
## 41            5         3.5          1.3         0.3     setosa
## 44            5         3.5          1.6         0.6     setosa
## 50            5         3.3          1.4         0.2     setosa
## 61            5         2.0          3.5         1.0 versicolor
## 94            5         2.3          3.3         1.0 versicolor

One can use & (and), | (or) and ! (not) logical operators for complex sub-setting.

iris[iris$Sepal.Length == 5 & iris$Species == "setosa", ]
##    Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 5             5         3.6          1.4         0.2  setosa
## 8             5         3.4          1.5         0.2  setosa
## 26            5         3.0          1.6         0.2  setosa
## 27            5         3.4          1.6         0.4  setosa
## 36            5         3.2          1.2         0.2  setosa
## 41            5         3.5          1.3         0.3  setosa
## 44            5         3.5          1.6         0.6  setosa
## 50            5         3.3          1.4         0.2  setosa

A data frame is the most common data structure you will encounter as a beginner. As you can see from the examples above, the syntax is tricky and not intuitive at all. In order to overcome this problem, R has a package called tidyverse. The tidyverse package combines eight other packages into one. Two important packages in tidyverse are dplyr and ggplot2

  • dplyr: Makes working with data frames fun and easy
  • ggplot2: Creates beautiful plots with ease.

We will start working with tidyverse in the next session.

  • summary() is a very useful function that summarises each column in a data frame
summary(iris)
##   Sepal.Length    Sepal.Width     Petal.Length    Petal.Width   
##  Min.   :4.300   Min.   :2.000   Min.   :1.000   Min.   :0.100  
##  1st Qu.:5.100   1st Qu.:2.800   1st Qu.:1.600   1st Qu.:0.300  
##  Median :5.800   Median :3.000   Median :4.350   Median :1.300  
##  Mean   :5.843   Mean   :3.057   Mean   :3.758   Mean   :1.199  
##  3rd Qu.:6.400   3rd Qu.:3.300   3rd Qu.:5.100   3rd Qu.:1.800  
##  Max.   :7.900   Max.   :4.400   Max.   :6.900   Max.   :2.500  
##        Species  
##  setosa    :50  
##  versicolor:50  
##  virginica :50  
##                 
##                 
##

Exercises

  1. Using mtcars dataset answer the following.

    1. How many rows and columns in the mtcars data set?
    2. How many cars in the mtcars data set have 8 cylinders?
    3. How many cars in the mtcars data set have 6 cylinders and more than 3 gears?
Answer

1a. How many rows and columns in the mtcars data set?

nrow(mtcars)
## [1] 32
ncol(mtcars)
## [1] 11

1b. How many cars in the mtcars data set have 8 cylinders?

sum(mtcars$cyl == 8)
## [1] 14

1c. How many cars in the mtcars data set have 6 cylinders and more than 3 gears?

sum(mtcars$cyl == 6 & mtcars$gear > 3)
## [1] 5
  1. Extract 3rd and 5th rows with 1st and 3rd columns from mtcars data set.
Answer 
mtcars[c(3, 5), c(1, 3)]
##                    mpg disp
## Datsun 710        22.8  108
## Hornet Sportabout 18.7  360
  1. From airquality data set
    1. Identify the data structure of the first row by extracting it?
    2. Identify the data structure of the first column by extracting it?
Answer

3a. Identify the data structure of the first row by extracting it?

class(airquality[1, ])
## [1] "data.frame"

3b. Identify the data structure of the first column by extracting it?

class(airquality[, 1])
## [1] "integer"
is.vector(airquality[, 1])
## [1] TRUE
is.data.frame(airquality[, 1])
## [1] FALSE
  1. Using airquality data set
    1. Show first 10 rows
    2. Show last 2 rows
    3. list all the column names available
Hint Used help on head and tail to find out about their arguments.
Answer

4a. Show first 10 rows

head(airquality, n = 10)
##    Ozone Solar.R Wind Temp Month Day
## 1     41     190  7.4   67     5   1
## 2     36     118  8.0   72     5   2
## 3     12     149 12.6   74     5   3
## 4     18     313 11.5   62     5   4
## 5     NA      NA 14.3   56     5   5
## 6     28      NA 14.9   66     5   6
## 7     23     299  8.6   65     5   7
## 8     19      99 13.8   59     5   8
## 9      8      19 20.1   61     5   9
## 10    NA     194  8.6   69     5  10

4b. Show last 2 rows

tail(airquality, n = 2)
##     Ozone Solar.R Wind Temp Month Day
## 152    18     131  8.0   76     9  29
## 153    20     223 11.5   68     9  30

4c. list all the column names available

names(airquality)
## [1] "Ozone"   "Solar.R" "Wind"    "Temp"    "Month"   "Day"
colnames(airquality)
## [1] "Ozone"   "Solar.R" "Wind"    "Temp"    "Month"   "Day"
  1. Using airquality data set, can you test Ozone production and solar radiation are correlated and answer the following. Hint: Usecor.test function.
    1. Get correlation coefficient confidence interval
    2. what is the p-value?
    3. What is the estimated correlation coefficient?
Answer

5a. Get correlation coefficient confidence interval

results <- cor.test(airquality$Ozone, airquality$Solar.R)
results$conf.int
## [1] 0.173194 0.502132
## attr(,"conf.level")
## [1] 0.95

5b. what is the p-value?

result$p.value
## [1] 1.037258e-27

5c. What is the estimated correlation coefficient?

results$estimate
##       cor 
## 0.3483417