```{r, echo = FALSE, message=FALSE, error = FALSE}
library(knitr)
opts_chunk$set(comment = "", message = FALSE)
suppressWarnings({library(dplyr)})
library(readr)
library(tidyverse)
library(jhur)
```

<style type="text/css">
pre { /* Code block - slightly smaller in this lecture */
    font-size: 20px;
}
</style>

## Recap

- `select()`: subset and/or reorder columns
- `filter()`: remove rows
- `arrange()`: reorder rows
- `mutate()`: create new columns or modify them
- `select()` and `filter()` can be combined together
- remove a column: `select()` with `!` mark (`!col_name`)
- you can do sequential steps: especially using pipes `%>%`

📃[Cheatsheet](https://jhudatascience.org/intro_to_r/modules/cheatsheets/Day-3.pdf)


## Another Cheatsheet

https://raw.githubusercontent.com/rstudio/cheatsheets/main/data-transformation.pdf

```{r, fig.alt="A preview of the Data transformation cheatsheet produced by RStudio.", out.width = "80%", echo = FALSE, align = "center"}
knitr::include_graphics("images/Manip_cheatsheet.png")
```


## Data Summarization

* Basic statistical summarization
    * `mean(x)`: takes the mean of x
    * `sd(x)`: takes the standard deviation of x
    * `median(x)`: takes the median of x
    * `quantile(x)`: displays sample quantiles of x. Default is min, IQR, max
    * `range(x)`: displays the range. Same as `c(min(x), max(x))`
    * `sum(x)`: sum of x
    * `max(x)`: maximum value in x
    * `min(x)`: minimum value in x
    
* **all have the ** `na.rm =` **argument for missing data**


## Statistical summarization

The vector getting summarized goes inside the parentheses:

```{r}
x <- c(1, 5, 7, 4, 2, 8)
mean(x)
range(x)
sum(x)
```


## Statistical summarization

Note that many of these functions have additional inputs regarding missing data, typically requiring the `na.rm` argument ("remove NAs"). 

```{r error = TRUE}
x <- c(1, 5, 7, 4, 2, 8, NA)
mean(x)
mean(x, na.rm = TRUE)
quantile(x)
quantile(x, na.rm = TRUE)
```


## Statistical summarization{.codesmall}

We will talk more about data types later, but you can only do summarization on numeric or logical types. Not characters.

```{r error = TRUE}
x <- c(1, 5, 7, 4, 2, 8)
sum(x)
y <- c(TRUE, FALSE, FALSE, TRUE) # FALSE == 0 and TRUE == 1
sum(y)
z <- c("TRUE", "FALSE", "FALSE", "TRUE")
sum(z)
```


## Some examples

We can use the `jhu_cars` to explore different ways of summarizing data.  The `head` command displays the first rows of an object:

```{r}
library(jhur)
head(jhu_cars)
```


## The `dplyr` pipe `%>%` operator

A nice and readable way to chain together multiple R functions.

Changes `f(x, y)` to `x %>% f(y)`.

```{r eval=FALSE}
# Going to work
get_dressed(me,
            pack_lunch(
              check_pockets(
                wallet = TRUE, phone = TRUE, keys = TRUE),
              items = c("sandwich", "chips", "apple"), lunchbox = TRUE),
            pants = TRUE, shirt = TRUE, footwear = "sandals")

# Going to work, the tidy way
me %>%
  get_dressed(pants = TRUE, shirt = TRUE, footwear = "sandals") %>%
  pack_lunch(items = c("sandwich", "chips", "apple"), lunchbox = TRUE) %>%
  check_pockets(wallet = TRUE, phone = TRUE, keys = TRUE)
```


## Statistical summarization the "tidy" way

```{r}
jhu_cars %>% pull(hp) %>% mean() # alt: pull(jhu_cars, hp) %>% mean()
jhu_cars %>% pull(wt) %>% median()
jhu_cars %>% pull(hp) %>% quantile()
jhu_cars %>% pull(wt) %>% quantile(probs = 0.6)
```


## Behavior of `pull()` function

`pull()` converts a single data column into a vector. This allows you to run summary functions on these data. Once you have "pulled" the data column out, you don't have to name it again in any piped summary functions.

```{r}
cars_wt <- jhu_cars %>% pull(wt)
class(cars_wt)
cars_wt
```

```{r, eval=FALSE}
jhu_cars %>% pull(wt) %>% range(wt) # Incorrect
```

```{r}
jhu_cars %>% pull(wt) %>% range() # Correct
```


# Summarization on tibbles (data frames)

## TB Incidence 

Let's read in a `tibble` of values from TB incidence.

If you have the `jhur` package installed successfully:

```{r}
library(jhur)
tb <- read_tb()
```

<br>
If not, download the `xlsx` file from http://jhudatascience.org/intro_to_r/data/tb_incidence.xlsx and read it in:

```{r eval = FALSE}
library(readxl)
tb <- read_excel("tb_incidence.xlsx")
```


## TB Incidence 

Check out the data:

```{r}
head(tb)
```


## TB Incidence 

Check out the data:

```{r}
str(tb)
```


## Indicator of TB

Before we go further, let's rename the first column using the `rename()` function in `dplyr`.

In this case, we have to use the backticks (\`) because there are spaces and funky characters in the name. 

```{r}
library(dplyr)
tb <- tb %>%
  rename(country = `TB incidence, all forms (per 100 000 population per year)`)
```


## Indicator of TB

`colnames()` will show us the column names and show that country is renamed:

```{r}
colnames(tb)
```


## Summarize the data: `dplyr` `summarize()` function

`summarize` creates a summary table of a column you're interested in.

Can run multiple summary statistics at once (unlike `pull()` which can only do a single calculation on one column).

You can also do more elaborate summaries across different groups of data using `group_by()`. More on this later!

<div class = "codeexample">
```{r, eval = FALSE}
# General format - Not the code!
{data to use} %>% 
   summarize({summary column name} = {operator(source column)},
             {summary column name} = {operator(source column)}) 
```
</div>


## Summarize the data: `dplyr` `summarize()` function

`summarize` creates a summary table of a column you're interested in.

<div class = "codeexample">
```{r, eval = FALSE}
# General format - Not the code!
{data to use} %>% 
   summarize({summary column name} = {operator(source column)}) 
```
</div>

```{r}
tb %>% 
  summarize(mean_1991 = mean(`1991`)) # Note the backticks, this is a column name!
tb %>% 
  summarize(mean_1991 = mean(`1991`, na.rm = TRUE))
```


## Summarize the data: `dplyr` `summarize()` function

`summarize()` can do multiple operations at once. Just separate by a comma.

```{r}
tb %>% 
  summarize(mean_1991 = mean(`1991`, na.rm = TRUE),
            median_1991 = median(`1991`, na.rm = TRUE),
            median(`2000`, na.rm = TRUE))
```

<br>
Notice how when we forget to provide a new name, output is still provided, but the column name is messy.


## Summarize the data: `dplyr` `summarize()` function

This looks better.

```{r}
tb %>% 
  summarize(mean_1991 = mean(`1991`, na.rm = TRUE),
            median_1991 = median(`1991`, na.rm = TRUE),
            median_2000 = median(`2000`, na.rm = TRUE))
```


## Summarize the data: `dplyr` `summarize()` function

Note that `summarize()` creates a separate tibble from the original data, so you don't want to overwrite your original data if you decide to save the summary.

If you want to save a summary statistic in the original data, use `mutate()` instead to create a new column for the summary statistic.


## `summary()` Function

Using `summary()` can give you rough snapshots of each numeric column (character columns are skipped):

```{r}
summary(tb)
```


## Summary & Lab Part 1

- summary stats (`mean()`) work with `pull()`
- don't forget the `na.rm = TRUE` argument!
- `summary(x)`: quantile information
- `summarize`: creates a summary table of columns of interest

🏠 [Class Website](https://jhudatascience.org/intro_to_r/)

💻 [Lab](https://jhudatascience.org/intro_to_r/modules/Data_Summarization/lab/Data_Summarization_Lab.Rmd)


## Youth Tobacco Survey

Here we will be using the Youth Tobacco Survey data:
http://jhudatascience.org/intro_to_r/data/Youth_Tobacco_Survey_YTS_Data.csv

```{r}
yts <- read_yts()
head(yts)
```


## `distinct()` values

`distinct(x)` will return the unique elements of column `x`.

```{r, message = FALSE}
yts %>%
  distinct(LocationDesc)
```


## How many `distinct()` values?

`n_distinct()` tells you the number of unique elements. _Must pull the column first!_

```{r}
yts %>%
  pull(LocationDesc) %>% 
  n_distinct()
```


```{r echo=FALSE}
options(max.print = 1000)
```


## `dplyr`: `count` 

Use `count` to return a frequency table of unique elements of a data.frame.

```{r, message = FALSE}
yts %>% count(LocationDesc)
```


## `dplyr`: `count` 

Multiple columns listed further subdivides the count.

```{r, message = FALSE}
yts %>% count(LocationDesc, TopicDesc)
```


## `dplyr`: `count` 

Multiple columns listed further subdivides the count.

```{r, message = FALSE}
yts %>% count(LocationDesc, TopicDesc)
```

<br>

**Note:** `count()` includes NAs


# Grouping

## Perform Operations By Groups: dplyr

`group_by` allows you group the data set by variables/columns you specify:

```{r}
# Regular data
yts
```


## Perform Operations By Groups: dplyr

`group_by` allows you group the data set by variables/columns you specify:

```{r}
yts_grouped <- yts %>% group_by(Response)
yts_grouped
```


## Summarize the grouped data

It's grouped! Grouping doesn't change the data in any way, but how **functions operate on it**. Now we can summarize `Data_Value` (percent of respondents) by group:

```{r}
yts_grouped %>% summarize(avg_percent = mean(Data_Value, na.rm = TRUE))
```


## Use the `pipe` to string these together!

Pipe `yts` into `group_by`, then pipe that into `summarize`:

```{r}
yts %>%
  group_by(Response) %>%
  summarize(avg_percent = mean(Data_Value, na.rm = TRUE),
            max_percent = max(Data_Value, na.rm = TRUE))
```


## Group by as many variables as you want

`group_by` Response and Education:

```{r}
yts %>%
  group_by(Response, Education) %>%
  summarize(avg_percent = mean(Data_Value, na.rm = TRUE),
            max_percent = max(Data_Value, na.rm = TRUE))
```


## Only the last `group_by` is recognized...

You can overwrite the first `group_by` with a new one.

```{r}
yts %>% 
  group_by(Response, Education)  %>% 
  group_by(Education) 
```


## Ungroup the data

The `ungroup` function will allow you to clear the groups from the data.

```{r}
yts <- ungroup(yts)
yts
```


## `group_by` with `mutate` - just add data

We can also use `mutate` to calculate the mean value for each year and add it as a column:

```{r}
yts %>%
  group_by(YEAR) %>%
  mutate(year_avg = mean(Data_Value, na.rm = TRUE)) %>%
  select(LocationDesc, Data_Value, year_avg)
```


## Counting

There are other functions, such as `n()` count the number of observations (NAs included).

```{r}
yts %>%
  group_by(YEAR) %>%
  summarize(n = n(),
            mean = mean(Data_Value, na.rm = TRUE))
```


## Counting{.codesmall}

`count()` and `n()` can give very similar information.

```{r}
mtcars %>% count(cyl)
mtcars %>% group_by(cyl) %>% summarize(n()) # n() typically used with summarize
```


# A few miscellaneous topics .. 


## Base R functions you might see: `length` and `unique`

These functions require a column as a vector using `pull()`.

```{r, message = FALSE}
yts <- read_yts()
yts_loc <- yts %>% pull(LocationDesc) # pull() to make a vector
yts_loc %>% unique() # similar to distinct()
```

## Base R functions you might see: `length` and `unique`

These functions require a column as a vector using `pull()`.

```{r, message = FALSE}
yts_loc %>% unique() %>% length() # similar to n_distinct()
```

## * New! * Many dplyr functions now have a `.by=` argument

Pipe `yts` into `group_by`, then pipe that into `summarize`:

```{r eval = FALSE}
yts %>%
  group_by(Response) %>%
  summarize(avg_percent = mean(Data_Value, na.rm = TRUE),
            max_percent = max(Data_Value, na.rm = TRUE))
```

is the same as..

```{r eval = FALSE}
yts %>%
  summarize(avg_percent = mean(Data_Value, na.rm = TRUE),
            max_percent = max(Data_Value, na.rm = TRUE),
            .by = Response)
```


## `summary()` vs. `summarize()`

* `summary()` (base R) gives statistics table on a dataset. 
* `summarize()` (dplyr) creates a more customized summary tibble/dataframe.

## Summary & Lab Part 2

- `count(x)`: what unique values do you have? 
  - `distinct()`: what are the distinct values?
  - `n_distinct()` with `pull()`: how many distinct values?
- `group_by()`: changes all subsequent functions
  - combine with `summarize()` to get statistics per group
  - combine with `mutate()` to add column
- `summarize()` with `n()` gives the count (NAs included) 

🏠 [Class Website](https://jhudatascience.org/intro_to_r/)

💻 [Lab](https://jhudatascience.org/intro_to_r/modules/Data_Summarization/lab/Data_Summarization_Lab.Rmd)

```{r, fig.alt="The End", out.width = "50%", echo = FALSE, fig.align='center'}
knitr::include_graphics(here::here("images/the-end-g23b994289_1280.jpg"))
```

Image by <a href="https://pixabay.com/users/geralt-9301/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=812226">Gerd Altmann</a> from <a href="https://pixabay.com//?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=812226">Pixabay</a>


# Extra Slides: More advanced summarization

## Data Summarization on data frames
 
* Statistical summarization across the data frame
    * `rowMeans(x)`: takes the means of each row of x
    * `colMeans(x)`: takes the means of each column of x
    * `rowSums(x)`: takes the sum of each row of x
    * `colSums(x)`: takes the sum of each column of x


## `rowMeans()` example

Get means for each row.

Let's see what the mean TB incidence is across years each row (country):

```{r}
tb %>%
  select(starts_with("year")) %>%
  rowMeans(na.rm = TRUE) %>%
  head(n = 5)

tb %>%
  group_by(country) %>%
  summarize(mean = rowMeans(across(starts_with("year")), na.rm = TRUE)) %>%
  head(n = 5)
```


## `colMeans()` example

Get means for each column.

Let's see what the mean is across each column (year):

```{r}
tb %>%
  select(starts_with("year")) %>%
  colMeans(na.rm = TRUE) %>%
  head(n = 5)

tb %>%
  summarize(across(starts_with("year"), ~mean(.x, na.rm = TRUE)))
```
