---
title: "Subsetting Data in R"
output:
  ioslides_presentation:
    css: ../styles.css
    widescreen: yes
  pdf_document:
    latex_engine: xelatex
---

```{r, echo = FALSE}
library(knitr)
opts_chunk$set(comment = "")
suppressPackageStartupMessages(library(dplyr))
library(dplyr)
library(jhur)
```

## Overview

In this module, we will show you how to:

1. Look at your data in different ways
2. Create a data frame and a tibble
3. Create new variables/make rownames a column
4. Rename columns of a data frame
5. Subset rows of a data frame
6. Subset columns of a data frame
7. Add/remove new columns to a data frame
8. Order the columns of a data frame
9. Order the rows of a data frame

## Setup

We will largely focus on the `dplyr` package which is part of the `tidyverse`.

```{r, fig.alt="dplyr", out.width = "25%", echo = FALSE, fig.align='center'}
knitr::include_graphics("https://tidyverse.tidyverse.org/logo.png")
```

Some resources on how to use `dplyr`: 

* https://dplyr.tidyverse.org/
* https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html
* https://www.opencasestudies.org/


## Why dplyr?

```{r, fig.alt="dplyr", out.width = "100%", echo = FALSE, fig.align='center'}
knitr::include_graphics("images/dplyr.png")
```
The `dplyr` package is one of the most helpful packages for altering your data to get it into a form that is useful for creating visualizations, summarizing, or more deeply analyzing. 

So you can imagine using pliers on your data. 

## Loading in dplyr and tidyverse

See this website for a list of the packages included in the `tidyverse`: https://www.tidyverse.org/packages/
```{r}
library(tidyverse) # loads dplyr and other packages!
```

## Getting data to work with

Here we use one of the datasets that comes with base R called `mtcars`. We will now create a toy data frame named `df` using this data. This way we can alter `df` without worrying about changing `mtcars`.

```{r}
df <- mtcars # df is a copy of mtcars
head(df) # changing df does **not** change mtcars!
```

## Checking the data `dim()`

The `dim()` , `nrow()`, and `ncol()` functions are good options to check the dimensions of your data before moving forward.

```{r}
dim(df) # rows, columns
nrow(df) # number of rows
ncol(df) # number of columns
```

## Checking the data: `glimpse()`

In addition to `head()` and `tail()`, the `glimpse()`function of the `dplyr` package  is another great function to look at your data. 

```{r}
glimpse(df)
```


## Checking your data: `slice_sample()`

What if you want to see the middle of your data? You can use the `slice_sample()` function of the `dplyr` package to see a random set of rows. You can specify the number of rows with the `n` argument or use a proportion with the `prop` argument.

```{r}
slice_sample(df, n = 3)
slice_sample(df, prop = .2)

```


# Making data frames(base R) and tibbles (tidyverse)
## Creating data frames using base R data frame function
```{r}
data.frame(df)
```


## Keep in mind...

Need to assign the output of the function to keep the result!
```{R}
df_updated <-data.frame(df)
# this would overwrite the existing df object
df<-data.frame(df) 
```

## Or create a data frame when reading in the file

Or directly when reading in a csv with the `read.csv()` function (also base R)

```{r, eval = FALSE}
# function comes from base R - no package loading required
df_example_readr <- read.csv("documents/data_analysis/data_file.csv")
```


## tibble

We can create a **fancier** version of the previous data frame which can be really helpful. 



## Creating a `tibble`

If we would like to create a `tibble` ("fancy" data frame), we can using the  `tibble()` function.  

```{r}
tbl <- dplyr::tibble(df) 
tbl
```

Note don't necessarily need to use `head()`- tibbles conveniently print a portion of the data.

## tibbles form read_csv()

Alternatively we can read data files using the tidyverse with the `read_csv()` function of the `readr` package from the `tidyverse` to make a tibble.


```{r, eval = FALSE}

df_example_readr <- read_csv("documents/data_analysis/data_file.csv")
```

You may start to notice how the tidyverse package work well together!

## Summary of tibbles and data frames

**Base R:**  
Using `read.csv()` and `data.frame()` you can make data frames  

**Tidyverse (fancier version):**  
Using `read_csv()` and `tibble()` you can make tibbles 

We generally recommend using tibbles, but you can do a lot with tibbles too.

## Data frames vs tibbles
 
In the "tidy" data format, rownames are removed.  For example, `df` has each car name as a row name. Here we use the `head()` function to see the first 2 rows of each. In this case we would want to make the rownames a new column first before making into a tibble.

```{r}
head(df, 2)
head(tibble(df), 2)
```




## rownames_to_column function

If you run into losing a variable contained in your row names, you can also use `rownames_to_column` to add it before turning it into a `tibble` to keep them:

```{r, size = "tiny"}
head(rownames_to_column(df, var = "car"),  2)
head(tibble(rownames_to_column(df, var = "car")),  2)
```


# Renaming Columns

## Renaming Columns of a data frame or tibble

To rename columns in `dplyr`, you can use the `rename` function.

For example, let's rename mpg to MPG. Notice the new name is listed **first**!

```{r, eval = FALSE}
# general format! not code!
{data you are creating or changing} <- rename({data you are using}, 
                                        {New Name} = {Old name})
```

```{r}
df <- dplyr::rename(df, MPG = mpg)
head(df)
```

## Renaming All Columns of a data frame: dplyr

To rename all columns you use the `rename_all()`. In this case we will use `toupper()` to make all letters upper case. Could also use `tolower()` function.

```{r}
df_upper <- dplyr::rename_all(df, toupper)
head(df_upper, 3)
```

```{r}
df <- dplyr::rename_all(df, tolower)
head(df, 3)
```

## Lab Part 1

[Website](http://https://jhudatascience.org/intro_to_R_class/index.html)

# Subsetting Columns

## Subset columns of a data frame: 

We can grab the `carb` column using the `$` operator. This is the base R way to do this:

```{r}
df$carb
```

## Subset columns of a data frame - `tidyverse` way: 

To grab the `carb` column the `tidyverse` way we can use the `pull` function:
```{r}
pull(df, carb)
```


## Subset columns of a data frame: dplyr

The `select` command from `dplyr` allows you to subset (gives a `tibble`!)
```{r}
select(df, mpg)
```

## Subset columns of a data frame: dplyr

Note that if you want a single vector (not a `tibble`), use `pull` or `$`:
```{r}
pull(df, mpg)

# pull with select works too!

pull(select(df,mpg))
```

## Select columns of a data frame: dplyr

The `select` command from `dplyr` allows you to subset columns matching strings:
```{r}
select(df, mpg, cyl)
select(df, starts_with("c"))
```

## See the Select "helpers"

Here are a few:

```{r, eval = FALSE}
one_of() # if they exist
last_col()
ends_with()
contains() # like searching
```

Type `tidyselect::` in the console and see what RStudio suggests:

```{r, eval = FALSE}
tidyslect:: 
```



## Lab Part 2

[Website](http://https://jhudatascience.org/intro_to_R_class/index.html)

# Subsetting Rows

## Subset rows of a data frame: dplyr

The command in `dplyr` for subsetting rows is `filter`.

```{r}
filter(df, mpg > 20)
```

Note, no `$` or subsetting is necessary.  R "knows" `mpg` refers to a column of `df`.


## Subset rows of a data frame: dplyr

You can have multiple logical conditions using the following:

* `==` : equals to
* `!=`: not equal to (`!` : not/negation)
* `>` / `<`: greater than / less than
* `>=` or `<=`: greater than or equal to / less than or equal to
* `&` : AND
* `|` : OR

## Subset rows of a data frame: dplyr

The `%in%` operator can be used find values from a pre-made list (using `c()`):

```{r}
filter(df, mpg %in% c(20,21,22))
```

## Subset rows of a data frame: dplyr

You can filter by two conditions using `&` or commas:

```{r}
filter(df, mpg > 20 & cyl == 4)
filter(df, mpg > 20, cyl == 4)
```

## Subset rows of a data frame: dplyr

If you want OR statements (meaning the data can meet either condition does not need to meet both), you need to use the pipe `|` between conditions:

```{r}
filter(df, mpg > 20 | cyl == 4)

```



## Lab Part 3

[Website](http://https://jhudatascience.org/intro_to_R_class/index.html)

## Combining `filter` and `select`

You can combine `filter` and `select` to subset the rows and columns, respectively, of a data frame:

```{r}
select(filter(df, mpg > 20 & cyl == 4), cyl, hp)
```

In `R`, the common way to perform multiple operations is to wrap functions around each other in a nested way such as above. 

## Assigning Temporary Objects

One can also create temporary objects and reassign them:

```{r}
df2 <- filter(df, mpg > 20 & cyl == 4)
df2 <- select(df2, cyl, hp)

head(df2,4)
```

## Using the `pipe` (comes with `dplyr`):

Recently, the pipe `%>%` makes things such as this much more readable.  It reads left side "pipes" into right side.  RStudio `CMD/Ctrl + Shift + M` shortcut. Pipe `df` into `filter`, then pipe that into `select`:


```{r}
df %>% filter(mpg > 20 & cyl == 4) %>% select(cyl, hp)
```


# Adding/Removing Columns

## Adding new columns to a data frame: base R

You can add a new column, called `newcol` to `df`, using the `$` operator:
```{r}
df$newcol <- df$wt/2.2
head(df,3)
```


## Adding columns to a data frame: dplyr (`tidyverse` way)

The `$` method is very common.

The `mutate` function in `dplyr` allows you to add or modify columns of a data frame.

```{r, eval = FALSE}
# General format - Not the code!
{data object to update} <- mutate({data to use}, 
                                {new variable name} = {new variable source}) 
```

```{r}
df <- mutate(df, newcol = wt/2.2)
```



## Removing columns of a data frame: base R

You can remove a column by assigning to `NULL`:
```{r, eval = FALSE}
df$newcol <- NULL
```

## Removing columns of a data frame: dplyr

The `NULL` method is still very common.

The `select` function can remove a column with minus (`-`):
```{r, eval = FALSE}
select(df, - newcol)
```

```{r, echo = FALSE}
head(select(df, -newcol))
```

**Or, you can simply select the columns you want to keep, ignoring the ones you want to remove.**

## Removing columns to a data frame: dplyr

You can use `c()` to list the columns to remove.

Remove `newcol` and `drat`:
```{r}
select(df, -c("newcol", "drat"))
```


# Ordering columns

## Ordering the columns of a data frame: dplyr

The `select` function can reorder columns.
```{r, eval = FALSE}
head(df)
select(df, cyl, mpg, wt, car) %>%
head()
```


## Ordering the columns of a data frame: dplyr

We can also use the `relocate()` function of dplyr to rearrange the columns.

For example, let say we just wanted `wt` to be first.

```{r}

head(df)

df_carb <- relocate(.data = df, wt,
                       .before = mpg)

df_carb
```


<!-- ## Ordering the columns of a data frame: dplyr -->

<!-- The `select` function can reorder columns.  Put `newcol` first, then select the rest of columns: -->
<!-- ```{r, eval = FALSE} -->
<!-- select(df, newcol, everything()) -->
<!-- ``` -->

<!-- ```{r, echo = FALSE} -->
<!-- head(select(df, newcol, everything())) -->
<!-- ``` -->

<!-- ## Ordering the columns of a data frame: dplyr -->

<!-- Put `newcol` at the end ("remove, everything, then add back in"): -->

<!-- ```{r, eval = FALSE} -->
<!-- select(df, -newcol, everything(), newcol) -->
<!-- ``` -->

<!-- ```{r, echo = FALSE} -->
<!-- head(select(df, -newcol, everything(), newcol)) -->
<!-- ``` -->

<!-- ## Odering the colunames of a data frame: alphabetically -->
<!-- Using the base R `order()` function. -->

<!-- ```{r} -->
<!-- order(colnames(df)) -->
<!-- df %>% select(order(colnames(df))) -->
<!-- ``` -->

# Ordering rows

## Ordering the rows of a data frame: dplyr

The `arrange` function can reorder rows  By default, `arrange` orders in ascending order:
```{r}
arrange(df, mpg)
```

## Ordering the rows of a data frame: dplyr
Use the `desc` to arrange the rows in descending order:
```{r}
arrange(df, desc(mpg))
```

## Ordering the rows of a data frame: dplyr
Increasing and decreasing orderings:
```{r}
arrange(df, mpg, desc(hp))
```

<!-- ## Transmutation -->

<!-- The `transmute` function in `dplyr` combines both the `mutate` and `select` functions. One can create new columns and keep the only the columns wanted: -->

<!-- ```{r} -->
<!-- transmute(df, newcol2 = wt/2.2, mpg, hp) -->
<!-- ``` -->




## Lab Part 4

[Website](http://https://jhudatascience.org/intro_to_R_class/index.html)


# Extra Slides

## Creating conditional variables

One frequently-used tool is creating variables with conditions.

A general function for creating new variables based on existing variables is the `ifelse()` function, which "returns a value depending on whether the element of test is `TRUE` or `FALSE`."

```
ifelse(test, yes, no)

# test: an object which can be coerced
    to logical mode.
# yes: return values for true elements of test.
# no: return values for false elements of test.
```

## ifelse example

```{r}
df$disp
```

Now with `ifelse`()
```{r}
#ifelse(test, yes, no)
ifelse(df$disp <= 200, "low", "high")
```

## Adding columns to a data frame: dplyr

Combined with `ifelse(condition, TRUE, FALSE)`, it can give you:

```{r}
df = mutate(df, 
            disp_cat = ifelse(disp <= 200, "Low","High")
            )

head(df,2)
```

## Adding columns to a data frame: dplyr

Alternatively, `case_when` provides a clean syntax as well:

```{r}
df = mutate(df, 
            disp_cat2 = case_when(
              disp <= 200 ~ "Low",
              disp > 200 ~ "High",
            ))
head(df$disp_cat2)
```



## Renaming Columns of a data frame: base R

We can use the `colnames` function to extract and/or directly reassign column names of `df`:

```{r}
colnames(df) # just prints
colnames(df)[1:3] = c("MPG", "CYL", "DISP") # reassigns
head(df)
colnames(df)[1:3] = c("mpg", "cyl", "disp") #reset - just to keep consistent
```

## Renaming Columns of a data frame: base R

We can assign the column names, change the ones we want, and then re-assign
the column names:
```{r}
cn = colnames(df)
cn[ cn == "drat"] = "DRAT"
colnames(df) = cn
head(df)
colnames(df)[ colnames(df) == "DRAT"] = "drat" #reset
```

## Subset rows of a data frame with indices: 

Let's select **rows** 1 and 3 from `df` using brackets:

```{r}
df[ c(1, 3), ]
```

## Subset columns of a data frame: 

We can also subset a data frame using the bracket `[, ]` subsetting. 

For data frames and matrices (2-dimensional objects), the brackets are `[rows, columns]` subsetting.  We can grab the `x` column using the index of the column or the column name ("`carb`") 

```{r}
df[, 11]
df[, "carb"]
```

## Biggest difference between `tbl` and data frame: 

Mostly, `tbl` (tibbles) are the same as data frames, except they don't print all lines.  When subsetting only one column using brackets, a data frame will return a vector, but a `tbl` will return a `tbl`

```{r}
df[, 1]
tbl[, 1]
tbl[, "mpg"]
df[, 1, drop = FALSE]
```

## Subset columns of a data frame: 

We can select multiple columns using multiple column names:

```{r}
df[, c("mpg", "cyl")]
```


