modern_R.tex

% Options for packages loaded elsewhere
\PassOptionsToPackage{unicode}{hyperref}
\PassOptionsToPackage{hyphens}{url}
%
\documentclass[
]{article}
\usepackage{amsmath,amssymb}
\usepackage{lmodern}
\usepackage{iftex}
\ifPDFTeX
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
  \usepackage{textcomp} % provide euro and other symbols
\else % if luatex or xetex
  \usepackage{unicode-math}
  \defaultfontfeatures{Scale=MatchLowercase}
  \defaultfontfeatures[\rmfamily]{Ligatures=TeX,Scale=1}
\fi
% Use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
\IfFileExists{microtype.sty}{% use microtype if available
  \usepackage[]{microtype}
  \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\makeatletter
\@ifundefined{KOMAClassName}{% if non-KOMA class
  \IfFileExists{parskip.sty}{%
    \usepackage{parskip}
  }{% else
    \setlength{\parindent}{0pt}
    \setlength{\parskip}{6pt plus 2pt minus 1pt}}
}{% if KOMA class
  \KOMAoptions{parskip=half}}
\makeatother
\usepackage{xcolor}
\usepackage[margin=1in]{geometry}
\usepackage{color}
\usepackage{fancyvrb}
\newcommand{\VerbBar}{|}
\newcommand{\VERB}{\Verb[commandchars=\\\{\}]}
\DefineVerbatimEnvironment{Highlighting}{Verbatim}{commandchars=\\\{\}}
% Add ',fontsize=\small' for more characters per line
\usepackage{framed}
\definecolor{shadecolor}{RGB}{248,248,248}
\newenvironment{Shaded}{\begin{snugshade}}{\end{snugshade}}
\newcommand{\AlertTok}[1]{\textcolor[rgb]{0.94,0.16,0.16}{#1}}
\newcommand{\AnnotationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\AttributeTok}[1]{\textcolor[rgb]{0.77,0.63,0.00}{#1}}
\newcommand{\BaseNTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}}
\newcommand{\BuiltInTok}[1]{#1}
\newcommand{\CharTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\CommentTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textit{#1}}}
\newcommand{\CommentVarTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\ConstantTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\ControlFlowTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{\textbf{#1}}}
\newcommand{\DataTypeTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{#1}}
\newcommand{\DecValTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}}
\newcommand{\DocumentationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\ErrorTok}[1]{\textcolor[rgb]{0.64,0.00,0.00}{\textbf{#1}}}
\newcommand{\ExtensionTok}[1]{#1}
\newcommand{\FloatTok}[1]{\textcolor[rgb]{0.00,0.00,0.81}{#1}}
\newcommand{\FunctionTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\ImportTok}[1]{#1}
\newcommand{\InformationTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\newcommand{\KeywordTok}[1]{\textcolor[rgb]{0.13,0.29,0.53}{\textbf{#1}}}
\newcommand{\NormalTok}[1]{#1}
\newcommand{\OperatorTok}[1]{\textcolor[rgb]{0.81,0.36,0.00}{\textbf{#1}}}
\newcommand{\OtherTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{#1}}
\newcommand{\PreprocessorTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textit{#1}}}
\newcommand{\RegionMarkerTok}[1]{#1}
\newcommand{\SpecialCharTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\SpecialStringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\StringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\VariableTok}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\VerbatimStringTok}[1]{\textcolor[rgb]{0.31,0.60,0.02}{#1}}
\newcommand{\WarningTok}[1]{\textcolor[rgb]{0.56,0.35,0.01}{\textbf{\textit{#1}}}}
\usepackage{longtable,booktabs,array}
\usepackage{calc} % for calculating minipage widths
% Correct order of tables after \paragraph or \subparagraph
\usepackage{etoolbox}
\makeatletter
\patchcmd\longtable{\par}{\if@noskipsec\mbox{}\fi\par}{}{}
\makeatother
% Allow footnotes in longtable head/foot
\IfFileExists{footnotehyper.sty}{\usepackage{footnotehyper}}{\usepackage{footnote}}
\makesavenoteenv{longtable}
\usepackage{graphicx}
\makeatletter
\def\maxwidth{\ifdim\Gin@nat@width>\linewidth\linewidth\else\Gin@nat@width\fi}
\def\maxheight{\ifdim\Gin@nat@height>\textheight\textheight\else\Gin@nat@height\fi}
\makeatother
% Scale images if necessary, so that they will not overflow the page
% margins by default, and it is still possible to overwrite the defaults
% using explicit options in \includegraphics[width, height, ...]{}
\setkeys{Gin}{width=\maxwidth,height=\maxheight,keepaspectratio}
% Set default figure placement to htbp
\makeatletter
\def\fps@figure{htbp}
\makeatother
\setlength{\emergencystretch}{3em} % prevent overfull lines
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{5}
\usepackage{booktabs}
\ifLuaTeX
  \usepackage{selnolig}  % disable illegal ligatures
\fi
\usepackage[]{natbib}
\bibliographystyle{apalike}
\IfFileExists{bookmark.sty}{\usepackage{bookmark}}{\usepackage{hyperref}}
\IfFileExists{xurl.sty}{\usepackage{xurl}}{} % add URL line breaks if available
\urlstyle{same} % disable monospaced font for URLs
\hypersetup{
  pdftitle={Modern R with the tidyverse},
  pdfauthor={Bruno Rodrigues},
  hidelinks,
  pdfcreator={LaTeX via pandoc}}

\title{Modern R with the tidyverse}
\author{Bruno Rodrigues}
\date{2022-10-13}

\begin{document}
\maketitle

{
\setcounter{tocdepth}{2}
\tableofcontents
}
\hypertarget{preface}{%
\section*{Preface}\label{preface}}
\addcontentsline{toc}{section}{Preface}

\hypertarget{note-to-the-reader}{%
\subsection*{Note to the reader}\label{note-to-the-reader}}
\addcontentsline{toc}{subsection}{Note to the reader}

I have been working on this on and off for the past 4 years or so. In 2022, I have updated the
contents of the book to reflect updates introduced with R 4.1 and in several packages (especially
those from the \texttt{\{tidyverse\}}). I have also cut some content that I think is not that useful,
especially in later chapters.

This book is still being written. Chapters 1 to 8 are almost ready, but more content is being added
(especially to chapter 8). 9 and 10 are empty for now. Some exercises might be at the wrong place
too and more are coming.

You can purchase an ebook version of this book on \href{https://leanpub.com/modern_tidyverse}{leanpub}.
The version on leanpub is quite out of date, so if you buy it, it's really just to send some money
my money, so many thanks for that! You can also support me by \href{https://www.buymeacoffee.com/brodriguesco}{buying me a
coffee} or
\href{https://www.paypal.me/brodriguesco}{paypal.me}.

\hypertarget{what-is-r}{%
\subsection*{What is R?}\label{what-is-r}}
\addcontentsline{toc}{subsection}{What is R?}

Read R's official answer to this question
\href{https://cran.r-project.org/doc/FAQ/R-FAQ.html\#What-is-R_003f}{here}. To make it short: R is a
multi-paradigm (procedural, imperative, object-oriented and functional)\footnote{In this book we are going
  to focus on R's functional programming capabilities} programming language that
focuses on applications in \emph{statistics}. By \emph{statistics} I mean any field that uses statistics such
as official statistics, economics, finance, data science, machine learning, etc. For the sake of
simplicity, I will use the word ``statistics'' as a general term that encompasses all these fields and
disciplines for the remainder of this book.

\hypertarget{who-is-this-book-for}{%
\subsection*{Who is this book for?}\label{who-is-this-book-for}}
\addcontentsline{toc}{subsection}{Who is this book for?}

This book can be useful to different audiences. If you have never used R in your life, and want
to start, start with Chapter 1 of this book. Chapter 1 to 3 are the very basics, and should be
easy to follow up to Chapter 7.
Starting with Chapter 7, it gets more technical, and will be harder to follow. But I suggest
you keep on going, and do not hesitate to contact me for help if you struggle! Chapter 7
is also where you can start if you are already familiar with R \textbf{and} the \texttt{\{tidyverse\}}, but not
functional programming. If you are familiar with R but not the \texttt{\{tidyverse\}} (or have no clue
what the \texttt{\{tidyverse\}} is), then you can start with Chapter 4. If you are familiar with R, the
\texttt{\{tidyverse\}} and functional programming, you might still be interested in this book, especially
Chapter 9 and 10, which deal with package development and further advanced topics respectively.

\hypertarget{why-this-book}{%
\subsection*{Why this book?}\label{why-this-book}}
\addcontentsline{toc}{subsection}{Why this book?}

This book is first and foremost for myself. This book is the result of years of using and teaching
R at university and then at my jobs. During my university time, I wrote some notes to help me
teach R and which I shared with my students. These are still the basis of Chapter 2. Then, once
I had left university, and continued using R at my first ``real'' job, I wrote another book that
dealt mostly with package development and functional programming. This book is now merged to this
one and is the basis of Chapters 9 and 10. During these years at my first
job, I was also tasked with teaching R. By that time, I was already quite familiar with the
\texttt{\{tidyverse\}} so I wrote a lot of notes that were internal and adapted for the audience of my
first job. These are now the basis of Chapters 3 to 8.
Then, during all these years, I kept blogging about R, and reading blogs and further books. All
this knowledge is condensed here, so if you are familiar with my blog, you'll definitely recognize
a lot of my blog posts in here. So this book is first and foremost for me, because I need to write
all of this down in a central place. So because my target audience is myself, this book is free. If
you find it useful, and are in the mood of buying me a coffee, you can, but if this book is not
useful to you, no harm done (unless you paid for it before reading it, in which case, I am sorry
to have wasted your time). But I am quite sure you'll find some of the things written here useful,
regardless of your current experience level with R.

\hypertarget{why-modern-r}{%
\subsection*{\texorpdfstring{Why \emph{modern} R?}{Why modern R?}}\label{why-modern-r}}
\addcontentsline{toc}{subsection}{Why \emph{modern} R?}

\emph{Modern} R instead of ``just'' R because we are going to learn how to use modern packages (mostly
those from the \href{https://www.tidyverse.org/}{tidyverse}) and concepts, such as functional
programming (which is quite an old concept actually, but one that came into fashion recently). R is
derived from S, which is a programming language that has roots in FORTRAN and other languages too.
If you learned R at university, you've probably learned to use it as you would have used FORTRAN;
very long scripts where data are represented as matrices and where row-wise (or column-wise)
operations are implemented with \texttt{for} loops. There's nothing wrong with that, mind you, but R
was also influenced by Scheme and Common Lisp, which are functional programming languages.
In my opinion, functional programming is a programming paradigm that works really well when dealing
with statistical problems. This is because programming in a functional style is just like
writing math. For instance, suppose you want to sum all the elements of a vector. In mathematical
notation, you would write something like:

\[
\sum_{i = 1}^{100} x_{i}
\]

where \(x\) is a vector of length 100. Solving this using a loop would look something like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{res }\OtherTok{\textless{}{-}} \DecValTok{0}
\ControlFlowTok{for}\NormalTok{(i }\ControlFlowTok{in} \DecValTok{1}\SpecialCharTok{:}\FunctionTok{length}\NormalTok{(x))\{}
\NormalTok{  res }\OtherTok{\textless{}{-}}\NormalTok{ x[i] }\SpecialCharTok{+}\NormalTok{ res}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

This does not look like the math notation at all! You have to define a variable that will hold
the result outside of the loop, and then you have to define \texttt{res} as something plus \texttt{res} inside
the body of the loop. This is really unnatural. The functional programming approach is much
easier:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{Reduce}\NormalTok{(}\StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{, x)}
\end{Highlighting}
\end{Shaded}

We will learn about \texttt{Reduce()} later (to be more precise, we will learn about \texttt{purrr::reduce()},
the ``tidy'' version of \texttt{Reduce()}), but already you see that the notation looks a lot more
like the mathematical notation.

At its core, functional programming uses functions, and functions are so-called \emph{first
class} objects in R, which means that there is nothing special about them\ldots{} you can pass them to
other functions, create functions that return functions and do any kind of operation on them just as
with any other object. This means that functions in R are extremely powerful and flexible tools.
In the first part of the book, we are going to use functions that are already available in R, and
then use those available in packages, mostly those from the \texttt{tidyverse}. The \texttt{tidyverse} is a
collection of packages developed by \href{http://hadley.nz/}{Hadley Wickham}, and several of his colleagues
at RStudio, Inc.~By using the packages from the \texttt{tidyverse} and R's built-in functional programming
capabilities, we can write code that is faster and easier to explain to colleagues, and also easier
to maintain. This also means that you might have to change your expectations and what you know
already from R, if you learned it at University but haven't touched it in a long time. For example
for and while loops, are relegated to chapter 8. This does not mean that you will have to wait for
8 chapter to know how to repeat instructions \emph{N} times, but that \emph{for} and \emph{while} loops are tools that
are very useful for very specific situations that will be discussed at that point.

In the second part of the book, we are going to move from using R to solve statistical problems to
developing with R. We are going to learn about creating your own package. If you do not know what
packages are, don't worry, this will be discussed just below.

\hypertarget{what-is-rstudio}{%
\subsection*{What is RStudio?}\label{what-is-rstudio}}
\addcontentsline{toc}{subsection}{What is RStudio?}

RStudio is a modern IDE that makes writing R code easier. The first thing we are going to learn is
how to use it.
R and RStudio are both open source: this means that the source code is freely available on
the internet and contributions by anyone are welcome and integrated; provided they are meaningful
and useful.

\hypertarget{what-to-expect-from-this-book}{%
\subsection*{What to expect from this book?}\label{what-to-expect-from-this-book}}
\addcontentsline{toc}{subsection}{What to expect from this book?}

The idea of Chapters 1 to 7 is to make you efficient with R as quickly as possible, especially if
you already have prior programming knowledge. Starting with Chapter 8 you will learn more advanced
topics, especially programming with R. R is a programming language, and you can't write
``programming language'' without ``language''. And just as you wouldn't expect to learn
French, Portuguese or Icelandic by reading a single book, you shouldn't expect to become fluent in R
by reading a single book, not even by reading 10 books. Programming is an art which requires a lot of
practice. \href{http://www.norvig.com/21-days.html}{Teach yourself programming in 10 years} is a blog
post written by Peter Norvig which explains that just as with any craft, mastering programming
takes time. And even if you don't need or want to become an expert in R, if you wish to use R
effectively and in a way that ultimately saves you time, you need to have some fluency in it, and
this only comes by continuing to learn about the language, and most importantly practicing. If you
keep using R every day, you'll definitely become very fluent. To stay informed about developments of
the language, and the latest news, I advise you read blogs, especially
\href{https://www.r-bloggers.com/}{R-bloggers} which aggregates blog posts by more than 750 blogs
discussing R.

So what you can expect from this book is that this book is not the only one you should read.

\hypertarget{prerequisites}{%
\subsection*{Prerequisites}\label{prerequisites}}
\addcontentsline{toc}{subsection}{Prerequisites}

R and RStudio are the two main pieces of software that we are going to use. R is the programming
language and RStudio is a modern IDE for it. You can use R without RStudio; but you cannot use
RStudio without R.

If you wish to install R and RStudio at home to follow the examples in this book you can do it as
both pieces of software are available free of charge (paid options for RStudio exist, for companies
that need technical support). Installation is simple, but operating system dependent. To download
and install R for Windows, follow \href{https://cloud.r-project.org/bin/windows/base/}{this link}.
For macOS, follow \href{https://cloud.r-project.org/bin/macosx/}{this one}. If you run a GNU+Linux
distribution, you can install R using the system's package manager. On Ubuntu, install \texttt{r-base}.

For RStudio, look for your operating system \href{https://www.rstudio.com/products/rstudio/download/\#download}{here}.

\hypertarget{what-are-packages}{%
\subsection*{What are packages?}\label{what-are-packages}}
\addcontentsline{toc}{subsection}{What are packages?}

There is one more step; we are going to install some packages. Packages are additional pieces of
code that can be installed from within R with the following function: \texttt{install.packages()}. These
packages extend R's capabilities significantly, and are probably one of the main reasons R is so
popular. As of November 2018, R has over 13000 packages.

To install the packages we need, first open RStudio and then copy and paste this line in the console:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{install.packages}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\StringTok{"tidyverse"}\NormalTok{, }\StringTok{"rsample"}\NormalTok{, }\StringTok{"recipes"}\NormalTok{, }\StringTok{"blogdown"}\NormalTok{ ,}\StringTok{"yardstick"}\NormalTok{, }\StringTok{"parsnip"}\NormalTok{, }\StringTok{"plm"}\NormalTok{, }\StringTok{"pwt9"}\NormalTok{, }
                   \StringTok{"checkpoint"}\NormalTok{, }\StringTok{"Ecdat"}\NormalTok{, }\StringTok{"ggthemes"}\NormalTok{, }\StringTok{"ggfortify"}\NormalTok{, }\StringTok{"margins"}\NormalTok{, }\StringTok{"janitor"}\NormalTok{, }\StringTok{"rio"}\NormalTok{, }\StringTok{"stopwords"}\NormalTok{,}
                   \StringTok{"colourpicker"}\NormalTok{, }\StringTok{"glmnet"}\NormalTok{, }\StringTok{"lhs"}\NormalTok{, }\StringTok{"mlrMBO"}\NormalTok{, }\StringTok{"mlbench"}\NormalTok{, }\StringTok{"ranger"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\includegraphics[width=14.9in]{pics/install_packages}

or go to the \textbf{Packages} pane and then click on \emph{Install}:

\includegraphics{pics/rstudio_install_packages.gif}

\hypertarget{the-author}{%
\subsection*{The author}\label{the-author}}
\addcontentsline{toc}{subsection}{The author}

My name is Bruno Rodrigues and I program almost exclusively in R and have been teaching some R
courses for a few years now. I first started teaching for students at the University of Strasbourg
while working on my PhD. I hold a PhD in economics, with a focus on quantitative methods.
I'm currently head of the statistics department of the Ministry of Higher education and Research
in Luxembourg, and before that worked as a manager in the data science team of PWC Luxembourg.

This book is an adaptation of notes I've used in the past during my time as a teacher, but also
a lot of things I've learned about R since I left academia.
In my free time I like cooking, working out and \href{https://www.brodrigues.co}{blogging}, while listening to
\href{http://www.fipradio.fr/player}{Fip} or
\href{https://chillsky.com/listen/}{Chillsky Radio}.
I also like to get my butt handed to me by playing roguelikes
such as \href{http://nethack.wikia.com/wiki/NetHack}{NetHack}, for which I wrote a
\href{https://github.com/b-rodrigues/nethack}{package} that contains functions to analyze the data that
is saved on your computer after you win or lose (it will be lose 99\% of the time) the game.

You can follow me on \href{https://www.twitter.com/brodriguesco}{twitter}, I tweet mostly about R or
what's happening in Luxembourg.

\hypertarget{getting-to-know-rstudio}{%
\section{Getting to know RStudio}\label{getting-to-know-rstudio}}

RStudio is a company that develops and maintains several products. Their best-known product is
an IDE (Integrated development environment) for the R programming language, also called RStudio.
You can install RStudio by visiting this \href{https://www.rstudio.com/products/rstudio/download/}{link}.
There is also a server version that can be used to have a centralized version of R within, say, a
company. RStudio, the company, also develops \href{https://shiny.rstudio.com/}{Shiny}, a package to
create full-fledged web-apps. I am not going to cover Shiny in this book, since there's already
\href{http://shiny.rstudio.com/tutorial/}{a lot} of material that you can learn from.

Once you have installed RStudio, launch it and let's go through the interface together.

\hypertarget{panes}{%
\subsection{Panes}\label{panes}}

RStudio is divided into different panes. Each pane has a specific function. The gif below shows
some of these panes:

\includegraphics{pics/rstudio_panes.gif}

Take some time to look around what each pane shows you. Some panes are empty; for example the \emph{Plots}
pane or the \emph{Viewer} pane. \emph{Plots} shows you the plots you make. You can browse the plots and save
them. We will see this in more detail in a later chapter. \emph{Viewer} shows you previews of documents
that you generate with R. More on this later.

\hypertarget{console}{%
\subsection{Console}\label{console}}

The \emph{Console} pane is where you can execute R code. Write the following in the console:

\begin{Shaded}
\begin{Highlighting}[]
\DecValTok{2} \SpecialCharTok{+} \DecValTok{3}
\end{Highlighting}
\end{Shaded}

and you'll get the answer, \texttt{5}. However, do not write a lot of lines in the console. It is better
write your code inside a script. Output is also shown inside the console.

\hypertarget{scripts}{%
\subsection{Scripts}\label{scripts}}

Look at the gif below:

\includegraphics{pics/rstudio_new_script.gif}

In this gif, we see the user creating a new R script. R scripts are simple text files that hold R
code. Think of \texttt{.do} files in STATA or \texttt{.c} files for C. R scripts have the extension \texttt{.r} or \texttt{.R}.

It is possible to create a lot of other files. We'll take a look at \texttt{R\ Markdown} files in Chapter 11.

\hypertarget{the-help-pane}{%
\subsubsection{The help pane}\label{the-help-pane}}

The \emph{Help} pane allows you to consult documentation for functions or packages. The gif below shows
how it works:

\includegraphics{pics/rstudio_help.gif}

you can also access help using the following syntax: \texttt{?lm}. This will bring up the documentation for
the function \texttt{lm()}. You can also type \texttt{??lm} which will look for the string \texttt{lm} in every package.

\hypertarget{the-environment-pane}{%
\subsubsection{The Environment pane}\label{the-environment-pane}}

The \emph{Environment} pane shows every object created in the current section. It is especially useful
if you have defined lists or have loaded data into R as it makes it easy to explore these more
complex objects.

\hypertarget{options}{%
\subsection{Options}\label{options}}

It is also possible to customize RStudio's look and feel:

\includegraphics{pics/rstudio_options.gif}

Take some time to go through the options.

\hypertarget{keyboard-shortcuts}{%
\subsection{Keyboard shortcuts}\label{keyboard-shortcuts}}

It is a good idea to familiarize yourself with at least some keyboard shortcuts. This is more
convenient than having to move the mouse around:

\includegraphics{pics/rstudio_shortcuts.gif}

If there is only one keyboard shortcut you need to know, it's \texttt{Ctrl-Enter} that executes a line of code
from your script. However, these other shortcuts are also worth knowing:

\begin{itemize}
\tightlist
\item
  \texttt{CTRL-ALT-R}: run entire script
\item
  \texttt{CTRL-ALT-UP\ or\ DOWN}: make cursor taller or shorter, allowing you to edit multiple lines at the same time
\item
  \texttt{CTRL-F}: Search and replace
\item
  \texttt{ALT-UP\ or\ DOWN}: Move line up or down
\item
  \texttt{CTRL-SHIFT-C}: Comment/uncomment line
\item
  \texttt{ALT-SHIFT-K}: Bring up the list of keyboard shortcuts
\item
  \texttt{CTRL-SHIFT-M}: Insert the pipe operator (\texttt{\%\textgreater{}\%}, more on this later)
\item
  \texttt{CTRL-S}: Save script
\end{itemize}

This is just a few keyboard shortcuts that I personally find useful. However, I strongly advise you
to learn and use whatever shortcuts are useful and feel natural to you!

\hypertarget{projects}{%
\subsection{Projects}\label{projects}}

One of the best features of RStudio are projects. Creating a project is simple; the gif below
shows how you can create a project and how you can switch between projects.

\includegraphics{pics/rstudio_projects.gif}

Projects make a lot of things easier, such as managing paths. More on this in the chapter about
reading data. Another useful feature of projects is that the scripts you open in project A will
stay open even if you switch to another project B, and then switch back to the project A again.

You can also use version control (with git) inside a project. Version control is very useful, but
I won't discuss it here. You can find a lot of resources online to get you started with git.

\hypertarget{history}{%
\subsection{History}\label{history}}

The \emph{History} pane saves all the previous lines you executed. You can then select these lines and
send them back to the console or the script.

\includegraphics{pics/rstudio_history.gif}

\hypertarget{plots}{%
\subsection{Plots}\label{plots}}

All the plots you make during a session are visible in the \emph{Plots} pane. From there, you can
export them in different formats.

\includegraphics{pics/rstudio_plots.gif}

The plots shown in the gif are made using basic R functions. Later, we will learn how to make nicer
looking plots using the package \texttt{ggplot2}.

\hypertarget{addins}{%
\subsection{Addins}\label{addins}}

Some packages install addins, which are accessible through the addins button:

\includegraphics[width=17.08in]{pics/rstudio_addins}

These addins make it easier to use some functions and you can read more about them \href{https://rstudio.github.io/rstudioaddins/\#overview}{here}.

My favorite addins are the ones you get when installing the \texttt{\{datapasta\}} package. Read more about
it \href{https://github.com/MilesMcBain/datapasta}{here}.

There are other panes that I will not discuss here, but you will naturally discover their use as you
go. For example, we will discuss the \emph{Build} pane in Chapter 11.

\hypertarget{packages}{%
\subsection{Packages}\label{packages}}

You can think of packages as addons that extend R's core functionality. You can browse all available
packages on \href{https://cloud.r-project.org/}{CRAN}. To make it easier to find what you might be
interested in, you can also browse the \href{https://cloud.r-project.org/web/views/}{CRAN Task Views}.
Each package has a landing page that summarises its dependencies, version number etc. For example,
for the \texttt{dplyr} package: \url{https://cran.r-project.org/web/packages/dplyr/index.html}.
Take a look at the \emph{Downloads} section, and especially at the Reference Manual and Vignettes:

\includegraphics[width=13.49in]{pics/packages_vignette}

Vignettes are valuable documents; inside vignettes, the purpose of the package is explained in
plain English, usually with accompanying examples. The reference manuals list the available functions
inside the packages. You can also find vignettes from within Rstudio:

\includegraphics{pics/rstudio_vignette.gif}

Go to the \emph{Packages} pane and click on the package you're interested in. Then you can consult the
help for the functions that come with the package as well as the package's vignettes.

Once you installed a package, you have to load it before you can use it. To load packages you use the
\texttt{library()} function:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(dplyr)}
\FunctionTok{library}\NormalTok{(janitor)}
\CommentTok{\# and so on...}
\end{Highlighting}
\end{Shaded}

If you only need to use one single function once, you don't need to load an entire package. You can
write the following:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{dplyr}\SpecialCharTok{::}\FunctionTok{full\_join}\NormalTok{(A, B)}
\end{Highlighting}
\end{Shaded}

using the \texttt{::} operator, you can access functions from packages without having to load the whole
package beforehand.

It is possible and easy to create your own packages. This is useful if you have to write a lot of
functions that you use daily. We will lean about that, in Chapter 10.

\hypertarget{exercises}{%
\subsection{Exercises}\label{exercises}}

\hypertarget{exercise-1}{%
\subsubsection*{Exercise 1}\label{exercise-1}}
\addcontentsline{toc}{subsubsection}{Exercise 1}

Change the look and feel of RStudio to suit your tastes! I personally like to move the console
to the right and use a dark theme. Take some 5 minutes to customize it and browse through all the options.

\hypertarget{objects-their-classes-and-types-and-useful-r-functions-to-get-you-started}{%
\section{Objects, their classes and types, and useful R functions to get you started}\label{objects-their-classes-and-types-and-useful-r-functions-to-get-you-started}}

All objects in R have a given \emph{type}. You already know most of them, as these types are also used
in mathematics. Integers, floating point numbers (floats), matrices, etc, are all objects you
are already familiar with. But R has other, maybe lesser known data types (that you can find in a
lot of other programming languages) that you need to become familiar with. But first, we need to
learn how to assign a value to a variable. This can be done in two ways:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \DecValTok{3}
\end{Highlighting}
\end{Shaded}

or

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{=} \DecValTok{3}
\end{Highlighting}
\end{Shaded}

in very practical terms, there is no difference between the two. I prefer using \texttt{\textless{}-} for assigning
values to variables and reserve \texttt{=} for passing arguments to functions, for example:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{spam }\OtherTok{\textless{}{-}} \FunctionTok{mean}\NormalTok{(}\AttributeTok{x =} \FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{3}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

I think this is less confusing than:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{spam }\OtherTok{=} \FunctionTok{mean}\NormalTok{(}\AttributeTok{x =} \FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{3}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

but as I explained above you can use whatever you feel most comfortable with.

\hypertarget{the-numeric-class}{%
\subsection{\texorpdfstring{The \texttt{numeric} class}{The numeric class}}\label{the-numeric-class}}

To define single numbers, you can do the following:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \DecValTok{3}
\end{Highlighting}
\end{Shaded}

The \texttt{class()} function allows you to check the class of an object:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "numeric"
\end{verbatim}

Decimals are defined with the character \texttt{.}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \FloatTok{3.14}
\end{Highlighting}
\end{Shaded}

R also supports integers. If you find yourself in a situation where you explicitly need an integer
and not a floating point number, you can use the following:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a  }\OtherTok{\textless{}{-}} \FunctionTok{as.integer}\NormalTok{(}\DecValTok{3}\NormalTok{)}
\FunctionTok{class}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "integer"
\end{verbatim}

The \texttt{as.integer()} function is very useful, because it converts its argument into an integer. There
is a whole family of \texttt{as.*()} functions. To convert \texttt{a} into a floating point number again:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(}\FunctionTok{as.numeric}\NormalTok{(a))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "numeric"
\end{verbatim}

There is also \texttt{is.numeric()} which tests whether a number is of the \texttt{numeric} class:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{is.numeric}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] TRUE
\end{verbatim}

It is also possible to create an integer using \texttt{L}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}}\NormalTok{ 5L}

\FunctionTok{class}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "integer"
\end{verbatim}

Another way to convert this integer back to a floating point number is to use \texttt{as.double()} instead of
as numeric:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(}\FunctionTok{as.double}\NormalTok{(a))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "numeric"
\end{verbatim}

The functions prefixed with \texttt{is.*} and \texttt{as.*} are quite useful, there is one for any of the supported types in R, such
as \texttt{as/is.character()}, \texttt{as/is.factor()}, etc\ldots{}

\hypertarget{the-character-class}{%
\subsection{\texorpdfstring{The \texttt{character} class}{The character class}}\label{the-character-class}}

Use \texttt{"\ "} to define characters (called strings in other programming languages):

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \StringTok{"this is a string"}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "character"
\end{verbatim}

To convert something to a character you can use the \texttt{as.character()} function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \FloatTok{4.392}

\FunctionTok{class}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "numeric"
\end{verbatim}

Now let's convert it:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(}\FunctionTok{as.character}\NormalTok{(a))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "character"
\end{verbatim}

It is also possible to convert a character to a numeric:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \StringTok{"4.392"}

\FunctionTok{class}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "character"
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(}\FunctionTok{as.numeric}\NormalTok{(a))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "numeric"
\end{verbatim}

But this only works if it makes sense:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \StringTok{"this won\textquotesingle{}t work, chief"}

\FunctionTok{class}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "character"
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{as.numeric}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: NAs introduced by coercion
\end{verbatim}

\begin{verbatim}
## [1] NA
\end{verbatim}

A very nice package to work with characters is \texttt{\{stringr\}}, which is also part of the \texttt{\{tidyverse\}}.

\hypertarget{the-factor-class}{%
\subsection{\texorpdfstring{The \texttt{factor} class}{The factor class}}\label{the-factor-class}}

Factors look like characters, but are very different. They are the representation of categorical
variables. A \texttt{\{tidyverse\}} package to work with factors is \texttt{\{forcats\}}. You would rarely use
factor variables outside of datasets, so for now, it is enough to know that this class exists.
We are going to learn more about factor variables in Chapter 4, by using the \texttt{\{forcats\}} package.

\hypertarget{the-date-class}{%
\subsection{\texorpdfstring{The \texttt{Date} class}{The Date class}}\label{the-date-class}}

Dates also look like characters, but are very different too:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{as.Date}\NormalTok{(}\StringTok{"2019/03/19"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "2019-03-19"
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(}\FunctionTok{as.Date}\NormalTok{(}\StringTok{"2019/03/19"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "Date"
\end{verbatim}

Manipulating dates and time can be tricky, but thankfully there's a \texttt{\{tidyverse\}} package for that,
called \texttt{\{lubridate\}}. We are going to go over this package in Chapter 4.

\hypertarget{the-logical-class}{%
\subsection{\texorpdfstring{The \texttt{logical} class}{The logical class}}\label{the-logical-class}}

This is the class of predicates, expressions that evaluate to \emph{true} or \emph{false}. For example, if you type:

\begin{Shaded}
\begin{Highlighting}[]
\DecValTok{4} \SpecialCharTok{\textgreater{}} \DecValTok{3}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] TRUE
\end{verbatim}

R returns \texttt{TRUE}, which is an object of class \texttt{logical}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{k }\OtherTok{\textless{}{-}} \DecValTok{4} \SpecialCharTok{\textgreater{}} \DecValTok{3}
\FunctionTok{class}\NormalTok{(k)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "logical"
\end{verbatim}

In other programming languages, \texttt{logical}s are often called \texttt{bool}s. A \texttt{logical} variable can only have
two values, either \texttt{TRUE} or \texttt{FALSE}. You can test the truthiness of a variable with \texttt{isTRUE()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{k }\OtherTok{\textless{}{-}} \DecValTok{4} \SpecialCharTok{\textgreater{}} \DecValTok{3}
\FunctionTok{isTRUE}\NormalTok{(k)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] TRUE
\end{verbatim}

How can you test if a variable is false? There is not a \texttt{isFALSE()} function (at least not without having
to load a package containing this function), but there is way to do it:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{k }\OtherTok{\textless{}{-}} \DecValTok{4} \SpecialCharTok{\textgreater{}} \DecValTok{3}
\SpecialCharTok{!}\FunctionTok{isTRUE}\NormalTok{(k)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] FALSE
\end{verbatim}

The \texttt{!} operator indicates negation, so the above expression could be translated as \emph{is k not TRUE?}.
There are other operators for boolean algebra, namely \texttt{\&,\ \&\&,\ \textbar{},\ \textbar{}\textbar{}}. \texttt{\&} means \emph{and} and \texttt{\textbar{}} stands for \emph{or}.
You might be wondering what the difference between \texttt{\&} and \texttt{\&\&} is? Or between \texttt{\textbar{}} and \texttt{\textbar{}\textbar{}}? \texttt{\&} and
\texttt{\textbar{}} work on vectors, doing pairwise comparisons:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{one }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{)}
\NormalTok{two }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\ConstantTok{FALSE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{)}
\NormalTok{one }\SpecialCharTok{\&}\NormalTok{ two}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] FALSE FALSE  TRUE FALSE
\end{verbatim}

Compare this to the \texttt{\&\&} operator:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{one }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{)}
\NormalTok{two }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\ConstantTok{FALSE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{)}
\NormalTok{one }\SpecialCharTok{\&\&}\NormalTok{ two}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning in one && two: 'length(x) = 4 > 1' in coercion to 'logical(1)'

## Warning in one && two: 'length(x) = 4 > 1' in coercion to 'logical(1)'
\end{verbatim}

\begin{verbatim}
## [1] FALSE
\end{verbatim}

The \texttt{\&\&} and \texttt{\textbar{}\textbar{}} operators only compare the first element of the vectors and stop as soon as a the return
value can be safely determined. This is called short-circuiting. Consider the following:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{one }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{)}
\NormalTok{two }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\ConstantTok{FALSE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{)}
\NormalTok{three }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{)}
\NormalTok{one }\SpecialCharTok{\&\&}\NormalTok{ two }\SpecialCharTok{\&\&}\NormalTok{ three}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning in one && two: 'length(x) = 4 > 1' in coercion to 'logical(1)'

## Warning in one && two: 'length(x) = 4 > 1' in coercion to 'logical(1)'
\end{verbatim}

\begin{verbatim}
## [1] FALSE
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{one }\SpecialCharTok{||}\NormalTok{ two }\SpecialCharTok{||}\NormalTok{ three}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning in one || two: 'length(x) = 4 > 1' in coercion to 'logical(1)'
\end{verbatim}

\begin{verbatim}
## [1] TRUE
\end{verbatim}

The \texttt{\textbar{}\textbar{}} operator stops as soon it evaluates to \texttt{TRUE} whereas the \texttt{\&\&} stops as soon as it evaluates to \texttt{FALSE}.
Personally, I rarely use \texttt{\textbar{}\textbar{}} or \texttt{\&\&} because I get confused. I find using \texttt{\textbar{}} or \texttt{\&} in combination with the
\texttt{all()} or \texttt{any()} functions much more useful:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{one }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{)}
\NormalTok{two }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\ConstantTok{FALSE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{)}
\FunctionTok{any}\NormalTok{(one }\SpecialCharTok{\&}\NormalTok{ two)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] TRUE
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{all}\NormalTok{(one }\SpecialCharTok{\&}\NormalTok{ two)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] FALSE
\end{verbatim}

\texttt{any()} checks whether any of the vector's elements are \texttt{TRUE} and \texttt{all()} checks if all elements of the vector are
\texttt{TRUE}.

As a final note, you should know that is possible to use \texttt{T} for \texttt{TRUE} and \texttt{F} for \texttt{FALSE} but I
would advise against doing this, because it is not very explicit.

\hypertarget{vectors-and-matrices}{%
\subsection{Vectors and matrices}\label{vectors-and-matrices}}

You can create a vector in different ways. But first of all, it is important to understand that a
vector in most programming languages is nothing more than a list of things. These things can be
numbers (either integers or floats), strings, or even other vectors. A vector in R can only contain elements of one
single type. This is not the case for a list, which is much more flexible. We will talk about lists shortly, but
let's first focus on vectors and matrices.

\hypertarget{the-c-function}{%
\subsubsection{\texorpdfstring{The \texttt{c()} function}{The c() function}}\label{the-c-function}}

A very important function that allows you to build a vector is \texttt{c()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{3}\NormalTok{,}\DecValTok{4}\NormalTok{,}\DecValTok{5}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

This creates a vector with elements 1, 2, 3, 4, 5. If you check its class:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "numeric"
\end{verbatim}

This can be confusing: you where probably expecting a to be of class \emph{vector} or
something similar. This is not the case if you use \texttt{c()} to create the vector, because \texttt{c()}
doesn't build a vector in the mathematical sense, but a so-called atomic vector.
Checking its dimension:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{dim}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## NULL
\end{verbatim}

returns \texttt{NULL} because an atomic vector doesn't have a dimension.
If you want to create a true vector, you need to use \texttt{cbind()} or \texttt{rbind()}.

But before continuing, be aware that atomic vectors can only contain elements of the same type:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{, }\StringTok{"3"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "1" "2" "3"
\end{verbatim}

because ``3'' is a character, all the other values get implicitly converted to characters. You have
to be very careful about this, and if you use atomic vectors in your programming, you have to make
absolutely sure that no characters or logicals or whatever else are going to convert your atomic
vector to something you were not expecting.

\hypertarget{cbind-and-rbind}{%
\subsubsection{\texorpdfstring{\texttt{cbind()} and \texttt{rbind()}}{cbind() and rbind()}}\label{cbind-and-rbind}}

You can create a \emph{true} vector with \texttt{cbind()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \FunctionTok{cbind}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{4}\NormalTok{, }\DecValTok{5}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Check its class now:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "matrix" "array"
\end{verbatim}

This is exactly what we expected. Let's check its dimension:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{dim}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1 5
\end{verbatim}

This returns the dimension of \texttt{a} using the LICO notation (number of LInes first, the number of COlumns).

It is also possible to bind vectors together to create a matrix.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{b }\OtherTok{\textless{}{-}} \FunctionTok{cbind}\NormalTok{(}\DecValTok{6}\NormalTok{,}\DecValTok{7}\NormalTok{,}\DecValTok{8}\NormalTok{,}\DecValTok{9}\NormalTok{,}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Now let's put vector \texttt{a} and \texttt{b} into a matrix called \texttt{matrix\_c} using \texttt{rbind()}.
\texttt{rbind()} functions the same way as \texttt{cbind()} but glues the vectors together by rows and not by columns.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{matrix\_c }\OtherTok{\textless{}{-}} \FunctionTok{rbind}\NormalTok{(a,b)}
\FunctionTok{print}\NormalTok{(matrix\_c)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      [,1] [,2] [,3] [,4] [,5]
## [1,]    1    2    3    4    5
## [2,]    6    7    8    9   10
\end{verbatim}

\hypertarget{the-matrix-class}{%
\subsubsection{\texorpdfstring{The \texttt{matrix} class}{The matrix class}}\label{the-matrix-class}}

R also has support for matrices. For example, you can create a matrix of dimension (5,5) filled
with 0's with the \texttt{matrix()} function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{matrix\_a }\OtherTok{\textless{}{-}} \FunctionTok{matrix}\NormalTok{(}\DecValTok{0}\NormalTok{, }\AttributeTok{nrow =} \DecValTok{5}\NormalTok{, }\AttributeTok{ncol =} \DecValTok{5}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

If you want to create the following matrix:

\[
B = \left(
\begin{array}{ccc}
 2 & 4 & 3 \\
 1 & 5 & 7
\end{array} \right)
\]

you would do it like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{B }\OtherTok{\textless{}{-}} \FunctionTok{matrix}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{2}\NormalTok{, }\DecValTok{4}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{1}\NormalTok{, }\DecValTok{5}\NormalTok{, }\DecValTok{7}\NormalTok{), }\AttributeTok{nrow =} \DecValTok{2}\NormalTok{, }\AttributeTok{byrow =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

The option \texttt{byrow\ =\ TRUE} means that the rows of the matrix will be filled first.

You can access individual elements of \texttt{matrix\_a} like so:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{matrix\_a[}\DecValTok{2}\NormalTok{, }\DecValTok{3}\NormalTok{]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0
\end{verbatim}

and R returns its value, 0. We can assign a new value to this element if we want. Try:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{matrix\_a[}\DecValTok{2}\NormalTok{, }\DecValTok{3}\NormalTok{] }\OtherTok{\textless{}{-}} \DecValTok{7}
\end{Highlighting}
\end{Shaded}

and now take a look at \texttt{matrix\_a} again.

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{print}\NormalTok{(matrix\_a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      [,1] [,2] [,3] [,4] [,5]
## [1,]    0    0    0    0    0
## [2,]    0    0    7    0    0
## [3,]    0    0    0    0    0
## [4,]    0    0    0    0    0
## [5,]    0    0    0    0    0
\end{verbatim}

Recall our vector \texttt{b}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{b }\OtherTok{\textless{}{-}} \FunctionTok{cbind}\NormalTok{(}\DecValTok{6}\NormalTok{,}\DecValTok{7}\NormalTok{,}\DecValTok{8}\NormalTok{,}\DecValTok{9}\NormalTok{,}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

To access its third element, you can simply write:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{b[}\DecValTok{3}\NormalTok{]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 8
\end{verbatim}

I have heard many people praising R for being a matrix based language. Matrices are indeed useful,
and statisticians are used to working with them. However, I very rarely use matrices in my
day to day work, and prefer an approach based on data frames (which will be discussed below). This
is because working with data frames makes it easier to use R's advanced functional programming
language capabilities, and this is where R really shines in my opinion. Working with matrices
almost automatically implies using loops and all the iterative programming techniques, \emph{à la Fortran},
which I personally believe are ill-suited for interactive statistical programming (as discussed in
the introduction).

\hypertarget{the-list-class}{%
\subsection{\texorpdfstring{The \texttt{list} class}{The list class}}\label{the-list-class}}

The \texttt{list} class is a very flexible class, and thus, very useful. You can put anything inside a list,
such as numbers:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{list1 }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\DecValTok{3}\NormalTok{, }\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

or other lists constructed with \texttt{c()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{list2 }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{), }\FunctionTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{, }\DecValTok{4}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

you can also put objects of different classes in the same list:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{list3 }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\DecValTok{3}\NormalTok{, }\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{), }\StringTok{"lists are amazing!"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

and of course create list of lists:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_lists }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(list1, list2, list3)}
\end{Highlighting}
\end{Shaded}

To check the contents of a list, you can use the structure function \texttt{str()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{str}\NormalTok{(my\_lists)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 3
##  $ :List of 2
##   ..$ : num 3
##   ..$ : num 2
##  $ :List of 2
##   ..$ : num [1:2] 1 2
##   ..$ : num [1:2] 3 4
##  $ :List of 3
##   ..$ : num 3
##   ..$ : num [1:2] 1 2
##   ..$ : chr "lists are amazing!"
\end{verbatim}

or you can use RStudio's \emph{Environment} pane:

\includegraphics{pics/rstudio_environment_list.gif}

You can also create named lists:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{list4 }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\StringTok{"name\_1"} \OtherTok{=} \DecValTok{2}\NormalTok{, }\StringTok{"name\_2"} \OtherTok{=} \DecValTok{8}\NormalTok{, }\StringTok{"name\_3"} \OtherTok{=} \StringTok{"this is a named list"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

and you can access the elements in two ways:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{list4[[}\DecValTok{1}\NormalTok{]]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 2
\end{verbatim}

or, for named lists:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{list4}\SpecialCharTok{$}\NormalTok{name\_3}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "this is a named list"
\end{verbatim}

Take note of the \texttt{\$} operator, because it is going to be quite useful for \texttt{data.frame}s as well,
which we are going to get to know in the next section.

Lists are used extensively because they are so flexible. You can build lists of datasets and apply
functions to all the datasets at once, build lists of models, lists of plots, etc\ldots{} In the later
chapters we are going to learn all about them. Lists are central objects in a functional programming
workflow for interactive statistical analysis.

\hypertarget{the-data.frame-and-tibble-classes}{%
\subsection{\texorpdfstring{The \texttt{data.frame} and \texttt{tibble} classes}{The data.frame and tibble classes}}\label{the-data.frame-and-tibble-classes}}

In the next chapter we are going to learn how to import datasets into R. Once you import data, the
resulting object is either a \texttt{data.frame} or a \texttt{tibble} depending on which package you used to
import the data. \texttt{tibble}s extend \texttt{data.frame}s so if you know about \texttt{data.frame} objects already,
working with \texttt{tibble}s will be very easy. \texttt{tibble}s have a better \texttt{print()} method, and some other
niceties.

However, I want to stress that these objects are central to R and are thus very important; they are
actually special cases of lists, discussed above. There are different ways to print a \texttt{data.frame} or
a \texttt{tibble} if you wish to inspect it. You can use \texttt{View(my\_data)} to show the \texttt{my\_data} \texttt{data.frame}
in the \emph{View} pane of RStudio:

\includegraphics{pics/rstudio_view_data.gif}

You can also use the \texttt{str()} function:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{str}\NormalTok{(my\_data)}
\end{Highlighting}
\end{Shaded}

And if you need to access an individual column, you can use the \texttt{\$} sign, same as for a list:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_data}\SpecialCharTok{$}\NormalTok{col1}
\end{Highlighting}
\end{Shaded}

\hypertarget{formulas}{%
\subsection{Formulas}\label{formulas}}

We will learn more about formulas later, but because it is an important object, it is useful if you
already know about them early on. A formula is defined in the following way:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_formula }\OtherTok{\textless{}{-}} \ErrorTok{\textasciitilde{}}\NormalTok{x}

\FunctionTok{class}\NormalTok{(my\_formula)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "formula"
\end{verbatim}

Formula objects are defined using the \texttt{\textasciitilde{}} symbol. Formulas are useful to define statistical models,
for example for a linear regression:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{lm}\NormalTok{(y }\SpecialCharTok{\textasciitilde{}}\NormalTok{ x)}
\end{Highlighting}
\end{Shaded}

or also to define anonymous functions, but more on this later.

\hypertarget{models}{%
\subsection{Models}\label{models}}

A statistical model is an object like any other in R:

Here, I have already a model that I ran on some test data:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(my\_model)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "lm"
\end{verbatim}

\texttt{my\_model} is an object of class \texttt{lm}, for \emph{linear model}. You can apply different functions to a model object:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{summary}\NormalTok{(my\_model)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Call:
## lm(formula = mpg ~ hp, data = mtcars)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -5.7121 -2.1122 -0.8854  1.5819  8.2360 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 30.09886    1.63392  18.421  < 2e-16 ***
## hp          -0.06823    0.01012  -6.742 1.79e-07 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 3.863 on 30 degrees of freedom
## Multiple R-squared:  0.6024, Adjusted R-squared:  0.5892 
## F-statistic: 45.46 on 1 and 30 DF,  p-value: 1.788e-07
\end{verbatim}

This class will be explored in later chapters.

\hypertarget{null-na-and-nan}{%
\subsection{NULL, NA and NaN}\label{null-na-and-nan}}

The \texttt{NULL}, \texttt{NA} and \texttt{NaN} classes are pretty special. \texttt{NULL} is returned when the result of function is undetermined.
For example, consider \texttt{list4}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{list4}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## $name_1
## [1] 2
## 
## $name_2
## [1] 8
## 
## $name_3
## [1] "this is a named list"
\end{verbatim}

if you try to access an element that does not exist, such as \texttt{d}, you will get \texttt{NULL} back:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{list4}\SpecialCharTok{$}\NormalTok{d}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## NULL
\end{verbatim}

\texttt{NaN} means ``Not a Number'' and is returned when a function return something that is not a number:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sqrt}\NormalTok{(}\SpecialCharTok{{-}}\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning in sqrt(-1): NaNs produced
\end{verbatim}

\begin{verbatim}
## [1] NaN
\end{verbatim}

or:

\begin{Shaded}
\begin{Highlighting}[]
\DecValTok{0}\SpecialCharTok{/}\DecValTok{0}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] NaN
\end{verbatim}

Basically, numbers that cannot be represented as floating point numbers are \texttt{NaN}.

Finally, there's \texttt{NA} which is closely related to \texttt{NaN} but is used for missing values. \texttt{NA} stands for \texttt{Not\ Available}. There are
several types of \texttt{NA}s:

\begin{itemize}
\tightlist
\item
  \texttt{NA\_integer\_}
\item
  \texttt{NA\_real\_}
\item
  \texttt{NA\_complex\_}
\item
  \texttt{NA\_character\_}
\end{itemize}

but these are in principle only used when you need to program your own functions and need
to explicitly test for the missingness of, say, a character value.

To test whether a value is \texttt{NA}, use the \texttt{is.na()} function.

\hypertarget{useful-functions-to-get-you-started}{%
\subsection{Useful functions to get you started}\label{useful-functions-to-get-you-started}}

This section will list several basic R functions that are very useful and should be part of your toolbox.

\hypertarget{sequences}{%
\subsubsection{Sequences}\label{sequences}}

There are several functions that create sequences, \texttt{seq()}, \texttt{seq\_along()} and \texttt{rep()}. \texttt{rep()} is easy enough:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{rep}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] 1 1 1 1 1 1 1 1 1 1
\end{verbatim}

This simply repeats \texttt{1} 10 times. You can repeat other objects too:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{rep}\NormalTok{(}\StringTok{"HAHA"}\NormalTok{, }\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] "HAHA" "HAHA" "HAHA" "HAHA" "HAHA" "HAHA" "HAHA" "HAHA" "HAHA" "HAHA"
\end{verbatim}

To create a sequence, things are not as straightforward. There is \texttt{seq()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{seq}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1]  1  2  3  4  5  6  7  8  9 10
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{seq}\NormalTok{(}\DecValTok{70}\NormalTok{, }\DecValTok{80}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] 70 71 72 73 74 75 76 77 78 79 80
\end{verbatim}

It is also possible to provide a \texttt{by} argument:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{seq}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{10}\NormalTok{, }\AttributeTok{by =} \DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1 3 5 7 9
\end{verbatim}

\texttt{seq\_along()} behaves similarly, but returns the length of the object passed to it. So if you pass \texttt{list4} to
\texttt{seq\_along()}, it will return a sequence from 1 to 3:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{seq\_along}\NormalTok{(list4)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1 2 3
\end{verbatim}

which is also true for \texttt{seq()} actually:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{seq}\NormalTok{(list4)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1 2 3
\end{verbatim}

but these two functions behave differently for arguments of length equal to 1:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{seq}\NormalTok{(}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1]  1  2  3  4  5  6  7  8  9 10
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{seq\_along}\NormalTok{(}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1
\end{verbatim}

So be quite careful about that. I would advise you do not use \texttt{seq()}, but only \texttt{seq\_along()} and \texttt{seq\_len()}. \texttt{seq\_len()}
only takes arguments of length 1:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{seq\_len}\NormalTok{(}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1]  1  2  3  4  5  6  7  8  9 10
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{seq\_along}\NormalTok{(}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1
\end{verbatim}

The problem with \texttt{seq()} is that it is unpredictable; depending on its input, the output will either be an integer or a sequence.
When programming, it is better to have function that are stricter and fail when confronted to special cases, instead of returning
some result. This is a bit of a recurrent issue with R, and the functions from the \texttt{\{tidyverse\}} mitigate this issue by being
stricter than their base R counterparts. For example, consider the \texttt{ifelse()} function from base R:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ifelse}\NormalTok{(}\DecValTok{3} \SpecialCharTok{\textgreater{}} \DecValTok{5}\NormalTok{, }\DecValTok{1}\NormalTok{, }\StringTok{"this is false"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "this is false"
\end{verbatim}

and compare it to \texttt{\{dplyr\}}'s implementation, \texttt{if\_else()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{if\_else}\NormalTok{(}\DecValTok{3} \SpecialCharTok{\textgreater{}} \DecValTok{5}\NormalTok{, }\DecValTok{1}\NormalTok{, }\StringTok{"this is false"}\NormalTok{)}
\NormalTok{Error}\SpecialCharTok{:} \StringTok{\textasciigrave{}}\AttributeTok{false}\StringTok{\textasciigrave{}}\NormalTok{ must be type double, not character}
\NormalTok{Call }\StringTok{\textasciigrave{}}\AttributeTok{rlang::last\_error()}\StringTok{\textasciigrave{}}\NormalTok{ to see a backtrace}
\end{Highlighting}
\end{Shaded}

\texttt{if\_else()} fails because the return value when \texttt{FALSE} is not a double (a real number) but a character. This might seem unnecessarily
strict, but at least it is predictable. This makes debugging easier when used inside functions. In Chapter 8 we are going to learn how
to write our own functions, and being strict makes programming easier.

\hypertarget{basic-string-manipulation}{%
\subsubsection{Basic string manipulation}\label{basic-string-manipulation}}

For now, we have not closely studied \texttt{character} objects, we only learned how to define them. Later, in Chapter 5 we will learn about the
\texttt{\{stringr\}} package which provides useful function to work with strings. However, there are several base R functions that are very
useful that you might want to know nonetheless, such as \texttt{paste()} and \texttt{paste0()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{paste}\NormalTok{(}\StringTok{"Hello"}\NormalTok{, }\StringTok{"amigo"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "Hello amigo"
\end{verbatim}

but you can also change the separator if needed:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{paste}\NormalTok{(}\StringTok{"Hello"}\NormalTok{, }\StringTok{"amigo"}\NormalTok{, }\AttributeTok{sep =} \StringTok{"{-}{-}"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "Hello--amigo"
\end{verbatim}

\texttt{paste0()} is the same as \texttt{paste()} but does not have any \texttt{sep} argument:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{paste0}\NormalTok{(}\StringTok{"Hello"}\NormalTok{, }\StringTok{"amigo"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "Helloamigo"
\end{verbatim}

If you provide a vector of characters, you can also use the \texttt{collapse} argument,
which places whatever you provide for \texttt{collapse} between the
characters of the vector:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{paste0}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\StringTok{"Joseph"}\NormalTok{, }\StringTok{"Mary"}\NormalTok{, }\StringTok{"Jesus"}\NormalTok{), }\AttributeTok{collapse =} \StringTok{", and "}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "Joseph, and Mary, and Jesus"
\end{verbatim}

To change the case of characters, you can use \texttt{toupper()} and \texttt{tolower()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{tolower}\NormalTok{(}\StringTok{"HAHAHAHAH"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "hahahahah"
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{toupper}\NormalTok{(}\StringTok{"hueuehuehuheuhe"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "HUEUEHUEHUHEUHE"
\end{verbatim}

Finally, there are the classical mathematical functions that you know and love:

\begin{itemize}
\tightlist
\item
  \texttt{sqrt()}
\item
  \texttt{exp()}
\item
  \texttt{log()}
\item
  \texttt{abs()}
\item
  \texttt{sin()}, \texttt{cos()}, \texttt{tan()}, and others
\item
  \texttt{sum()}, \texttt{cumsum()}, \texttt{prod()}, \texttt{cumprod()}
\item
  \texttt{max()}, \texttt{min()}
\end{itemize}

and many others\ldots{}

\hypertarget{exercises-1}{%
\subsection{Exercises}\label{exercises-1}}

\hypertarget{exercise-1-1}{%
\subsubsection*{Exercise 1}\label{exercise-1-1}}
\addcontentsline{toc}{subsubsection}{Exercise 1}

Try to create the following vector:

\[a = (6,3,8,9)\]

and add it this other vector:

\[b = (9,1,3,5)\]

and save the result to a new variable called \texttt{result}.

\hypertarget{exercise-2}{%
\subsubsection*{Exercise 2}\label{exercise-2}}
\addcontentsline{toc}{subsubsection}{Exercise 2}

Using \texttt{a} and \texttt{b} from before, try to get their dot product.

Try with \texttt{a\ *\ b} in the R console. What happened?
Try to find the right function to get the dot product. Don't hesitate to google the answer!

\hypertarget{exercise-3}{%
\subsubsection*{Exercise 3}\label{exercise-3}}
\addcontentsline{toc}{subsubsection}{Exercise 3}

How can you create a matrix of dimension (30,30) filled with 2's by only using the function \texttt{matrix()}?

\hypertarget{exercise-4}{%
\subsubsection*{Exercise 4}\label{exercise-4}}
\addcontentsline{toc}{subsubsection}{Exercise 4}

Save your first name in a variable \texttt{a} and your surname in a variable \texttt{b}. What does the function:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{paste}\NormalTok{(a, b)}
\end{Highlighting}
\end{Shaded}

do? Look at the help for \texttt{paste()} with \texttt{?paste} or using the \emph{Help} pane in RStudio. What does the
optional argument \texttt{sep} do?

\hypertarget{exercise-5}{%
\subsubsection*{Exercise 5}\label{exercise-5}}
\addcontentsline{toc}{subsubsection}{Exercise 5}

Define the following variables: \texttt{a\ \textless{}-\ 8}, \texttt{b\ \textless{}-\ 3}, \texttt{c\ \textless{}-\ 19}. What do the following lines check?
What do they return?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\SpecialCharTok{\textgreater{}}\NormalTok{ b}
\NormalTok{a }\SpecialCharTok{==}\NormalTok{ b}
\NormalTok{a }\SpecialCharTok{!=}\NormalTok{ b}
\NormalTok{a }\SpecialCharTok{\textless{}}\NormalTok{ b}
\NormalTok{(a }\SpecialCharTok{\textgreater{}}\NormalTok{ b) }\SpecialCharTok{\&\&}\NormalTok{ (a }\SpecialCharTok{\textless{}}\NormalTok{ c)}
\NormalTok{(a }\SpecialCharTok{\textgreater{}}\NormalTok{ b) }\SpecialCharTok{\&\&}\NormalTok{ (a }\SpecialCharTok{\textgreater{}}\NormalTok{ c)}
\NormalTok{(a }\SpecialCharTok{\textgreater{}}\NormalTok{ b) }\SpecialCharTok{||}\NormalTok{ (a }\SpecialCharTok{\textless{}}\NormalTok{ b)}
\end{Highlighting}
\end{Shaded}

\hypertarget{exercise-6}{%
\subsubsection*{Exercise 6}\label{exercise-6}}
\addcontentsline{toc}{subsubsection}{Exercise 6}

Define the following matrix:

\[
\text{matrix_a} = \left(
\begin{array}{ccc}
 9 & 4 & 12 \\
 5 & 0 & 7 \\
 2 & 6 & 8 \\
 9 & 2 & 9
\end{array} \right)
\]

\begin{itemize}
\tightlist
\item
  What does \texttt{matrix\_a\ \textgreater{}=\ 5} do?
\item
  What does \texttt{matrix\_a{[}\ ,\ 2{]}} do?
\item
  Can you find which function gives you the transpose of this matrix?
\end{itemize}

\hypertarget{exercise-7}{%
\subsubsection*{Exercise 7}\label{exercise-7}}
\addcontentsline{toc}{subsubsection}{Exercise 7}

Solve the following system of equations using the \texttt{solve()} function:

\[
\left(
\begin{array}{cccc}
 9 & 4 & 12 & 2 \\
 5 & 0 & 7 & 9\\
 2 & 6 & 8 & 0\\
 9 & 2 & 9 & 11
\end{array} \right) \times \left(
\begin{array}{ccc}
 x \\
 y \\
 z \\
 t \\
\end{array}\right) =
\left(
\begin{array}{ccc}
7\\
18\\
1\\
0
\end{array}
\right)
\]

\hypertarget{exercise-8}{%
\subsubsection*{Exercise 8}\label{exercise-8}}
\addcontentsline{toc}{subsubsection}{Exercise 8}

Load the \texttt{mtcars} data (\texttt{mtcars} is include in R, so you only need to use the \texttt{data()} function to
load the data):

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(mtcars)}
\end{Highlighting}
\end{Shaded}

if you run \texttt{class(mtcars)}, you get ``data.frame''. Try now with \texttt{typeof(mtcars)}. The answer is now
``list''! This is because the class of an object is an attribute of that object, which can even
be assigned by the user:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{class}\NormalTok{(mtcars) }\OtherTok{\textless{}{-}} \StringTok{"don\textquotesingle{}t do this"}

\FunctionTok{class}\NormalTok{(mtcars)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "don't do this"
\end{verbatim}

The type of an object is R's internal type of that object, which cannot be manipulated by the user.
It is always useful to know the type of an object (not just its class). For example, in the particular
case of data frames, because the type of a data frame is a list, you can use all that you learned
about lists to manipulate data frames! Recall that \texttt{\$} allowed you to select the element of a list
for instance:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_list }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\StringTok{"one"} \OtherTok{=} \DecValTok{1}\NormalTok{, }\StringTok{"two"} \OtherTok{=} \DecValTok{2}\NormalTok{, }\StringTok{"three"} \OtherTok{=} \DecValTok{3}\NormalTok{)}

\NormalTok{my\_list}\SpecialCharTok{$}\NormalTok{one}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1
\end{verbatim}

Because data frames are nothing but fancy lists, this is why you can access columns the same way:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars}\SpecialCharTok{$}\NormalTok{mpg}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] 21.0 21.0 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 17.8 16.4 17.3 15.2 10.4
## [16] 10.4 14.7 32.4 30.4 33.9 21.5 15.5 15.2 13.3 19.2 27.3 26.0 30.4 15.8 19.7
## [31] 15.0 21.4
\end{verbatim}

\hypertarget{reading-and-writing-data}{%
\section{Reading and writing data}\label{reading-and-writing-data}}

In this chapter, we are going to import example datasets that are available in R, \texttt{mtcars} and
\texttt{iris}. I have converted these datasets into several formats. Download those datasets
\href{https://github.com/b-rodrigues/modern_R/tree/master/datasets}{here} if you want to follow the
examples below. R can import some formats without the need of external packages, such as the \texttt{.csv}
format. However, for other formats, you will need to use different packages. Because there are a
lot of different formats available I suggest you use the \texttt{\{rio\}} package.
\texttt{\{rio\}} is a wrapper around different packages that import/export data in different formats.
This package is nice because you don't need to remember which package to use to import, say,
STATA datasets and then you need to remember which one for SAS datasets, and so on. Read \texttt{\{rio\}}'s
\href{https://cran.r-project.org/web/packages/rio/vignettes/rio.html}{vignette} for more details. Below
I show some of \texttt{\{rio\}}'s functions presented in the vignette. It is also possible to import data from
other, less ``traditional'' sources, such as your clipboard. Also note that it is possible to import
more than one dataset at once. There are two ways of doing that, either by importing all the
datasets, binding their rows together and add a new variable with the name of the data, or import
all the datasets into a list, where each element of that list is a data frame. We are going to
explore this second option later.

\hypertarget{the-swiss-army-knife-of-data-import-and-export-rio}{%
\subsection{\texorpdfstring{The swiss army knife of data import and export: \texttt{\{rio\}}}{The swiss army knife of data import and export: \{rio\}}}\label{the-swiss-army-knife-of-data-import-and-export-rio}}

To import data with \texttt{\{rio\}}, \texttt{import()} is all you need:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(rio)}

\NormalTok{mtcars }\OtherTok{\textless{}{-}} \FunctionTok{import}\NormalTok{(}\StringTok{"datasets/mtcars.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{head}\NormalTok{(mtcars)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## 1 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## 2 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## 3 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## 4 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## 5 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## 6 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1
\end{verbatim}

\texttt{import()} needs the path to the data, and you can specify additional options if needed. On a
Windows computer, you have to pay attention to the path; you cannot simply copy and paste it, because
paths in Windows use the \texttt{\textbackslash{}} symbol whereas R uses \texttt{/} (just like on Linux or macOS).
Importing a STATA or a SAS file is done just the same:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_stata }\OtherTok{\textless{}{-}} \FunctionTok{import}\NormalTok{(}\StringTok{"datasets/mtcars.dta"}\NormalTok{)}
\FunctionTok{head}\NormalTok{(mtcars\_stata)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## 1 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## 2 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## 3 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## 4 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## 5 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## 6 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_sas }\OtherTok{\textless{}{-}} \FunctionTok{import}\NormalTok{(}\StringTok{"datasets/mtcars.sas7bdat"}\NormalTok{)}
\FunctionTok{head}\NormalTok{(mtcars\_sas)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## 1 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## 2 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## 3 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## 4 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## 5 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## 6 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1
\end{verbatim}

It is also possible to import Excel files where each sheet is a single table, but you will need
\texttt{import\_list()} for that. The file \texttt{multi.xlsx} has two sheets, each with a table in it:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{multi }\OtherTok{\textless{}{-}} \FunctionTok{import\_list}\NormalTok{(}\StringTok{"datasets/multi.xlsx"}\NormalTok{)}
\FunctionTok{str}\NormalTok{(multi)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 2
##  $ mtcars:'data.frame':  32 obs. of  11 variables:
##   ..$ mpg : num [1:32] 21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
##   ..$ cyl : num [1:32] 6 6 4 6 8 6 8 4 4 6 ...
##   ..$ disp: num [1:32] 160 160 108 258 360 ...
##   ..$ hp  : num [1:32] 110 110 93 110 175 105 245 62 95 123 ...
##   ..$ drat: num [1:32] 3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
##   ..$ wt  : num [1:32] 2.62 2.88 2.32 3.21 3.44 ...
##   ..$ qsec: num [1:32] 16.5 17 18.6 19.4 17 ...
##   ..$ vs  : num [1:32] 0 0 1 1 0 1 0 1 1 1 ...
##   ..$ am  : num [1:32] 1 1 1 0 0 0 0 0 0 0 ...
##   ..$ gear: num [1:32] 4 4 4 3 3 3 3 4 4 4 ...
##   ..$ carb: num [1:32] 4 4 1 1 2 1 4 2 2 4 ...
##  $ iris  :'data.frame':  150 obs. of  5 variables:
##   ..$ Sepal.Length: num [1:150] 5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
##   ..$ Sepal.Width : num [1:150] 3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
##   ..$ Petal.Length: num [1:150] 1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
##   ..$ Petal.Width : num [1:150] 0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
##   ..$ Species     : chr [1:150] "setosa" "setosa" "setosa" "setosa" ...
\end{verbatim}

As you can see \texttt{multi} is a list of datasets. Told you lists were very flexible! It is also possible
to import all the datasets in a single directory at once. For this, you first need a vector of paths:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{paths }\OtherTok{\textless{}{-}} \FunctionTok{Sys.glob}\NormalTok{(}\StringTok{"datasets/unemployment/*.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\texttt{Sys.glob()} allows you to find files using a regular expression. ``datasets/unemployment/*.csv''
matches all the \texttt{.csv} files inside the ``datasets/unemployment/'' folder.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{all\_data }\OtherTok{\textless{}{-}} \FunctionTok{import\_list}\NormalTok{(paths)}

\FunctionTok{str}\NormalTok{(all\_data)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 4
##  $ unemp_2013:'data.frame':  118 obs. of  8 variables:
##   ..$ Commune                   : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ Total employed population : int [1:118] 223407 17802 1703 844 1431 4094 2146 971 1218 3002 ...
##   ..$ of which: Wage-earners    : int [1:118] 203535 15993 1535 750 1315 3800 1874 858 1029 2664 ...
##   ..$ of which: Non-wage-earners: int [1:118] 19872 1809 168 94 116 294 272 113 189 338 ...
##   ..$ Unemployed                : int [1:118] 19287 1071 114 25 74 261 98 45 66 207 ...
##   ..$ Active population         : int [1:118] 242694 18873 1817 869 1505 4355 2244 1016 1284 3209 ...
##   ..$ Unemployment rate (in %)  : num [1:118] 7.95 5.67 6.27 2.88 4.92 ...
##   ..$ Year                      : int [1:118] 2013 2013 2013 2013 2013 2013 2013 2013 2013 2013 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2013.csv"
##  $ unemp_2014:'data.frame':  118 obs. of  8 variables:
##   ..$ Commune                   : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ Total employed population : int [1:118] 228423 18166 1767 845 1505 4129 2172 1007 1268 3124 ...
##   ..$ of which: Wage-earners    : int [1:118] 208238 16366 1606 757 1390 3840 1897 887 1082 2782 ...
##   ..$ of which: Non-wage-earners: int [1:118] 20185 1800 161 88 115 289 275 120 186 342 ...
##   ..$ Unemployed                : int [1:118] 19362 1066 122 19 66 287 91 38 61 202 ...
##   ..$ Active population         : int [1:118] 247785 19232 1889 864 1571 4416 2263 1045 1329 3326 ...
##   ..$ Unemployment rate (in %)  : num [1:118] 7.81 5.54 6.46 2.2 4.2 ...
##   ..$ Year                      : int [1:118] 2014 2014 2014 2014 2014 2014 2014 2014 2014 2014 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2014.csv"
##  $ unemp_2015:'data.frame':  118 obs. of  8 variables:
##   ..$ Commune                   : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ Total employed population : int [1:118] 233130 18310 1780 870 1470 4130 2170 1050 1300 3140 ...
##   ..$ of which: Wage-earners    : int [1:118] 212530 16430 1620 780 1350 3820 1910 920 1100 2770 ...
##   ..$ of which: Non-wage-earners: int [1:118] 20600 1880 160 90 120 310 260 130 200 370 ...
##   ..$ Unemployed                : int [1:118] 18806 988 106 29 73 260 80 41 72 169 ...
##   ..$ Active population         : int [1:118] 251936 19298 1886 899 1543 4390 2250 1091 1372 3309 ...
##   ..$ Unemployment rate (in %)  : num [1:118] 7.46 5.12 5.62 3.23 4.73 ...
##   ..$ Year                      : int [1:118] 2015 2015 2015 2015 2015 2015 2015 2015 2015 2015 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2015.csv"
##  $ unemp_2016:'data.frame':  118 obs. of  8 variables:
##   ..$ Commune                   : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ Total employed population : int [1:118] 236100 18380 1790 870 1470 4160 2160 1030 1330 3150 ...
##   ..$ of which: Wage-earners    : int [1:118] 215430 16500 1640 780 1350 3840 1900 900 1130 2780 ...
##   ..$ of which: Non-wage-earners: int [1:118] 20670 1880 150 90 120 320 260 130 200 370 ...
##   ..$ Unemployed                : int [1:118] 18185 975 91 27 66 246 76 35 70 206 ...
##   ..$ Active population         : int [1:118] 254285 19355 1881 897 1536 4406 2236 1065 1400 3356 ...
##   ..$ Unemployment rate (in %)  : num [1:118] 7.15 5.04 4.84 3.01 4.3 ...
##   ..$ Year                      : int [1:118] 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2016.csv"
\end{verbatim}

in a subsequent chapter we will learn how to actually use these lists of datasets.

If you know that each dataset in each file has the same columns, you can also import them directly
into a single dataset by binding each dataset together using \texttt{rbind\ =\ TRUE}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{bind\_data }\OtherTok{\textless{}{-}} \FunctionTok{import\_list}\NormalTok{(paths, }\AttributeTok{rbind =} \ConstantTok{TRUE}\NormalTok{)}
\FunctionTok{str}\NormalTok{(bind\_data)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 'data.frame':    472 obs. of  9 variables:
##  $ Commune                   : chr  "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##  $ Total employed population : int  223407 17802 1703 844 1431 4094 2146 971 1218 3002 ...
##  $ of which: Wage-earners    : int  203535 15993 1535 750 1315 3800 1874 858 1029 2664 ...
##  $ of which: Non-wage-earners: int  19872 1809 168 94 116 294 272 113 189 338 ...
##  $ Unemployed                : int  19287 1071 114 25 74 261 98 45 66 207 ...
##  $ Active population         : int  242694 18873 1817 869 1505 4355 2244 1016 1284 3209 ...
##  $ Unemployment rate (in %)  : num  7.95 5.67 6.27 2.88 4.92 ...
##  $ Year                      : int  2013 2013 2013 2013 2013 2013 2013 2013 2013 2013 ...
##  $ _file                     : chr  "datasets/unemployment/unemp_2013.csv" "datasets/unemployment/unemp_2013.csv" "datasets/unemployment/unemp_2013.csv" "datasets/unemployment/unemp_2013.csv" ...
##  - attr(*, ".internal.selfref")=<externalptr>
\end{verbatim}

This also adds a further column called \texttt{\_file} indicating the name of the file that contained the
original data.

If something goes wrong, you might need to take a look at the underlying function \texttt{\{rio\}} is
actually using to import the file. Let's look at the following example:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{testdata }\OtherTok{\textless{}{-}} \FunctionTok{import}\NormalTok{(}\StringTok{"datasets/problems/mtcars.csv"}\NormalTok{)}

\FunctionTok{head}\NormalTok{(testdata)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   mpg&cyl&disp&hp&drat&wt&qsec&vs&am&gear&carb
## 1          21&6&160&110&3.9&2.62&16.46&0&1&4&4
## 2         21&6&160&110&3.9&2.875&17.02&0&1&4&4
## 3        22.8&4&108&93&3.85&2.32&18.61&1&1&4&1
## 4      21.4&6&258&110&3.08&3.215&19.44&1&0&3&1
## 5       18.7&8&360&175&3.15&3.44&17.02&0&0&3&2
## 6       18.1&6&225&105&2.76&3.46&20.22&1&0&3&1
\end{verbatim}

as you can see, the import didn't work quite well! This is because the separator is the \texttt{\&} for
some reason. Because we are trying to read a \texttt{.csv} file, \texttt{rio::import()} is using
\texttt{data.table::fread()} under the hood (you can read this in \texttt{import()}'s help). If you then read
\texttt{data.table::fread()}'s help, you see that the \texttt{fread()} function has an optional \texttt{sep\ =} argument
that you can use to specify the separator. You can use this argument in \texttt{import()} too, and it will
be passed down to \texttt{data.table::fread()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{testdata }\OtherTok{\textless{}{-}} \FunctionTok{import}\NormalTok{(}\StringTok{"datasets/problems/mtcars.csv"}\NormalTok{, }\AttributeTok{sep =} \StringTok{"\&"}\NormalTok{)}

\FunctionTok{head}\NormalTok{(testdata)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## 1   21   6  160 110  3.9  2.62 16.46  0  1    4    4
## 2   21   6  160 110  3.9 2.875 17.02  0  1    4    4
## 3 22.8   4  108  93 3.85  2.32 18.61  1  1    4    1
## 4 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## 5 18.7   8  360 175 3.15  3.44 17.02  0  0    3    2
## 6 18.1   6  225 105 2.76  3.46 20.22  1  0    3    1
\end{verbatim}

\texttt{export()} allows you to write data to disk, by simply providing the path and name of the file you
wish to save.

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{export}\NormalTok{(testdata, }\StringTok{"path/where/to/save/testdata.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

If you end the name with \texttt{.csv} the file is exported to the csv format, if instead you write \texttt{.dta}
the data will be exported to the STATA format, and so on.

If you wish to export to Excel, this is possible, but it may require that you change a file on your
computer (you only have to do this once). Try running:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{export}\NormalTok{(testdata, }\StringTok{"path/where/to/save/testdata.xlsx"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

if this results in an error, try the following:

\begin{itemize}
\tightlist
\item
  Run the following lines in Rstudio:
\end{itemize}

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{if}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{file.exists}\NormalTok{(}\StringTok{"\textasciitilde{}/.Rprofile"}\NormalTok{)) }\CommentTok{\# only create if not already there}
    \FunctionTok{file.create}\NormalTok{(}\StringTok{"\textasciitilde{}/.Rprofile"}\NormalTok{)    }\CommentTok{\# (don\textquotesingle{}t overwrite it)}
\FunctionTok{file.edit}\NormalTok{(}\StringTok{"\textasciitilde{}/.Rprofile"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

These lines, taken shamelessly from \href{https://csgillespie.github.io/efficientR/3-3-r-startup.html\#rprofile}{Efficient R
programming} (go read it,
it's a very great resource) look for and open the \texttt{.Rprofile} file which is a file that is run
every time you open Rstudio. This means that you can put any line of code there that will always be
executed whenever you launch Rstudio.

\begin{itemize}
\tightlist
\item
  Add this line to the file:
\end{itemize}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{Sys.setenv}\NormalTok{(}\StringTok{"R\_ZIPCMD"} \OtherTok{=} \StringTok{"C:/Program Files (x86)/Rtools/zip.exe"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

This tells Rstudio to use \texttt{zip.exe} as the default zip tool, which is needed to export files to the
Excel format. Try it out by restarting Rstudio, and then running the following lines:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(rio)}

\FunctionTok{data}\NormalTok{(mtcars)}

\FunctionTok{export}\NormalTok{(mtcars, }\StringTok{"mtcars.xlsx"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

You should find the \texttt{mtcars.xlsx} inside your working directory. You can check what is your working
directory with \texttt{getwd()}.

\texttt{\{rio\}} should cover all your needs, but if not, there is very likely a package out there that will
import the data you need.

\hypertarget{writing-any-object-to-disk}{%
\subsection{Writing any object to disk}\label{writing-any-object-to-disk}}

\texttt{\{rio\}} is an amazing package, but is only able to write tabular representations of data. What if you
would like to save, say, a list containing any arbitrary object? This is possible with the
\texttt{saveRDS()} function. Literally anything can be saved with \texttt{saveRDS()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_list }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\StringTok{"this is a list"}\NormalTok{,}
                \FunctionTok{list}\NormalTok{(}\StringTok{"which contains a list"}\NormalTok{, }\DecValTok{12}\NormalTok{),}
                \FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{4}\NormalTok{),}
                \FunctionTok{matrix}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{2}\NormalTok{, }\DecValTok{4}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{1}\NormalTok{, }\DecValTok{5}\NormalTok{, }\DecValTok{7}\NormalTok{),}
                       \AttributeTok{nrow =} \DecValTok{2}\NormalTok{))}

\FunctionTok{str}\NormalTok{(my\_list)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 4
##  $ : chr "this is a list"
##  $ :List of 2
##   ..$ : chr "which contains a list"
##   ..$ : num 12
##  $ : num [1:4] 1 2 3 4
##  $ : num [1:2, 1:3] 2 4 3 1 5 7
\end{verbatim}

\texttt{my\_list} is a list containing a string, a list which contains a string and a number, a vector and
a matrix\ldots{} Now suppose that computing this list takes a very long time. For example, imagine that
each element of the list is the result of estimating a very complex model on a simulated
dataset, which takes hours to run. Because this takes so long to compute, you'd want to save
it to disk. This is possible with \texttt{saveRDS()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{saveRDS}\NormalTok{(my\_list, }\StringTok{"my\_list.RDS"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

The next day, after having freshly started your computer and launched RStudio, it is possible to
retrieve the object exactly like it was using \texttt{readRDS()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_list }\OtherTok{\textless{}{-}} \FunctionTok{readRDS}\NormalTok{(}\StringTok{"my\_list.RDS"}\NormalTok{)}

\FunctionTok{str}\NormalTok{(my\_list)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 4
##  $ : chr "this is a list"
##  $ :List of 2
##   ..$ : chr "which contains a list"
##   ..$ : num 12
##  $ : num [1:4] 1 2 3 4
##  $ : num [1:2, 1:3] 2 4 3 1 5 7
\end{verbatim}

Even if you want to save a regular dataset, using \texttt{saveRDS()} might be a good idea because the data
gets compressed if you add the option \texttt{compress\ =\ TRUE} to \texttt{saveRDS()}. However keep in mind that
this will only be readable by R, so if you need to share this data with colleagues that use another
tool, save it in another format.

\hypertarget{using-rstudio-projects-to-manage-paths}{%
\subsection{Using RStudio projects to manage paths}\label{using-rstudio-projects-to-manage-paths}}

Managing paths can be painful, especially if you're collaborating with a colleague and both of you
saved the data in paths that are different. Whenever one of you wants to work on the script, the
path will need to be adapted first. The best way to avoid that is to use projects with RStudio.

\includegraphics{pics/rstudio_projects.gif}

Imagine that you are working on a project entitled ``housing''. You will create a folder called
``housing'' somewhere on your computer and inside this folder have another folder called ``data'', then
a bunch of other folders containing different files or the outputs of your analysis. What matters
here is that you have a folder called ``data'' which contains the datasets you will ananlyze. When
you are inside an RStudio project, granted that you chose your ``housing'' folder as the folder to
host the project, you can read the data by simply specifying the path like so:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_data }\OtherTok{\textless{}{-}} \FunctionTok{import}\NormalTok{(}\StringTok{"/data/data.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Constrast this to what you would need to write if you were not using a project:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_data }\OtherTok{\textless{}{-}} \FunctionTok{import}\NormalTok{(}\StringTok{"C:/My Documents/Castor/Work/Projects/Housing/data/data.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Not only is that longer, but if Castor is working on this project with Pollux, Pollux would need
to change the above line to this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_data }\OtherTok{\textless{}{-}} \FunctionTok{import}\NormalTok{(}\StringTok{"C:/My Documents/Pollux/Work/Projects/Housing/data/data.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

whenever Pollux needs to work on it. Another, similar issue, is that if you need to write something
to disk, such as a dataset or a plot, you would also need to specify the whole path:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{export}\NormalTok{(my\_data, }\StringTok{"C:/My Documents/Pollux/Work/Projects/Housing/data/data.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

If you forget to write the whole path, then the dataset will be saved in the standard working
directory, which is your ``My Documents'' folder on Windows, and ``Home'' on GNU+Linux or macOS. You
can check what is the working directory with the \texttt{getwd()} function:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{getwd}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

On a fresh session on my computer this returns:

\begin{verbatim}
"/home/bruno"
\end{verbatim}

or, on Windows:

\begin{verbatim}
"C:/Users/Bruno/Documents"
\end{verbatim}

but if you call this function inside a project, it will return the path to your project. It is also
possible to set the working directory with \texttt{setwd()}, so you don't need to always write the full
path, meaning that you can this:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{setwd}\NormalTok{(}\StringTok{"the/path/I/want/"}\NormalTok{)}

\FunctionTok{import}\NormalTok{(}\StringTok{"data/my\_data.csv"}\NormalTok{)}

\FunctionTok{export}\NormalTok{(processed\_data, }\StringTok{"processed\_data.xlsx"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

instead of:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{import}\NormalTok{(}\StringTok{"the/path/I/want/data/my\_data.csv"}\NormalTok{)}

\FunctionTok{export}\NormalTok{(processed\_data, }\StringTok{"the/path/I/want/processed\_data.xlsx"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

However, I really, really, really urge you never to use \texttt{setwd()}. Use projects instead!
Using projects saves a lot of pain in the long run.

\hypertarget{descriptive-statistics-and-data-manipulation}{%
\section{Descriptive statistics and data manipulation}\label{descriptive-statistics-and-data-manipulation}}

Now that we are familiar with some R objects and know how to import data, it is time to write some
code. In this chapter, we are going to compute descriptive statistics for a single dataset, but
also for a list of datasets later in the chapter. However, I will not give a list of functions to
compute descriptive statistics; if you need a specific function you can find easily in the \emph{Help}
pane in Rstudio or using any modern internet search engine. What I will do is show you a workflow
that allows you to compute the descripitive statisics you need fast. R has a lot of built-in
functions for descriptive statistics; however, if you want to compute statistics for different
sub-groups, some more complex manipulations are needed. At least this was true in the past.
Nowadays, thanks to the packages from the \texttt{\{tidyverse\}}, it is very easy and fast to compute
descriptive statistics by any stratifying variable(s). The package we are going to use for this is
called \texttt{\{dplyr\}}. \texttt{\{dplyr\}} contains a lot of functions that make manipulating data and computing
descriptive statistics very easy. To make things easier for now, we are going to use example data
included with \texttt{\{dplyr\}}. So no need to import an external dataset; this does not change anything to
the example that we are going to study here; the source of the data does not matter for this. Using
\texttt{\{dplyr\}} is possible only if the data you are working with is already in a useful shape. When data
is more messy, you will need to first manipulate it to bring it a \emph{tidy} format. For this, we will
use \texttt{\{tidyr\}}, which is very useful package to reshape data and to do advanced cleaning of your
data. All these tidyverse functions are also called \emph{verbs}. However, before getting to know these
verbs, let's do an analysis using standard, or \emph{base} R functions. This will be the benchmark
against which we are going to measure a \texttt{\{tidyverse\}} workflow.

\hypertarget{a-data-exploration-exercice-using-base-r}{%
\subsection{\texorpdfstring{A data exploration exercice using \emph{base} R}{A data exploration exercice using base R}}\label{a-data-exploration-exercice-using-base-r}}

Let's first load the \texttt{starwars} data set, included in the \texttt{\{dplyr\}} package:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(dplyr)}
\FunctionTok{data}\NormalTok{(starwars)}
\end{Highlighting}
\end{Shaded}

Let's first take a look at the data:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{head}\NormalTok{(starwars)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 14
##   name         height  mass hair_~1 skin_~2 eye_c~3 birth~4 sex   gender homew~5
##   <chr>         <int> <dbl> <chr>   <chr>   <chr>     <dbl> <chr> <chr>  <chr>  
## 1 Luke Skywal~    172    77 blond   fair    blue       19   male  mascu~ Tatooi~
## 2 C-3PO           167    75 <NA>    gold    yellow    112   none  mascu~ Tatooi~
## 3 R2-D2            96    32 <NA>    white,~ red        33   none  mascu~ Naboo  
## 4 Darth Vader     202   136 none    white   yellow     41.9 male  mascu~ Tatooi~
## 5 Leia Organa     150    49 brown   light   brown      19   fema~ femin~ Aldera~
## 6 Owen Lars       178   120 brown,~ light   blue       52   male  mascu~ Tatooi~
## # ... with 4 more variables: species <chr>, films <list>, vehicles <list>,
## #   starships <list>, and abbreviated variable names 1: hair_color,
## #   2: skin_color, 3: eye_color, 4: birth_year, 5: homeworld
\end{verbatim}

This data contains information on Star Wars characters. The first question you have to answer is
to find the average height of the characters:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{mean}\NormalTok{(starwars}\SpecialCharTok{$}\NormalTok{height)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] NA
\end{verbatim}

As discussed in Chapter 2, \texttt{\$} allows you to access columns of a \texttt{data.frame} objects.

Because there are \texttt{NA} values in the data, the result is also \texttt{NA}. To get the result, you need to
add an option to \texttt{mean()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{mean}\NormalTok{(starwars}\SpecialCharTok{$}\NormalTok{height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 174.358
\end{verbatim}

Let's also take a look at the standard deviation:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sd}\NormalTok{(starwars}\SpecialCharTok{$}\NormalTok{height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 34.77043
\end{verbatim}

It might be more informative to compute these two statistics by sex, so for this, we are going
to use \texttt{aggregate()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{aggregate}\NormalTok{(starwars}\SpecialCharTok{$}\NormalTok{height,}
          \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{sex =}\NormalTok{ starwars}\SpecialCharTok{$}\NormalTok{sex),}
\NormalTok{          mean)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##              sex   x
## 1         female  NA
## 2 hermaphroditic 175
## 3           male  NA
## 4           none  NA
\end{verbatim}

Oh, shoot! Most groups have missing values in them, so we get \texttt{NA} back. We need to use \texttt{na.rm\ =\ TRUE}
just like before. Thankfully, it is possible to pass this option to \texttt{mean()} inside \texttt{aggregate()} as well:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{aggregate}\NormalTok{(starwars}\SpecialCharTok{$}\NormalTok{height,}
          \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{sex =}\NormalTok{ starwars}\SpecialCharTok{$}\NormalTok{sex),}
\NormalTok{          mean, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##              sex        x
## 1         female 169.2667
## 2 hermaphroditic 175.0000
## 3           male 179.1053
## 4           none 131.2000
\end{verbatim}

Later in the book, we are also going to see how to define our own functions (with the default options that
are useful to us), and this will also help in this sort of situation.
Even though we can use \texttt{na.rm\ =\ TRUE}, let's also use \texttt{subset()} to filter out the \texttt{NA} values beforehand:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars\_no\_nas }\OtherTok{\textless{}{-}} \FunctionTok{subset}\NormalTok{(starwars,}
                          \SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(height))}

\FunctionTok{aggregate}\NormalTok{(starwars\_no\_nas}\SpecialCharTok{$}\NormalTok{height,}
          \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{sex =}\NormalTok{ starwars\_no\_nas}\SpecialCharTok{$}\NormalTok{sex),}
\NormalTok{          mean)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##              sex        x
## 1         female 169.2667
## 2 hermaphroditic 175.0000
## 3           male 179.1053
## 4           none 131.2000
\end{verbatim}

(\texttt{aggregate()} also has a \texttt{subset\ =} option, but I prefer to explicitely subset the data set with \texttt{subset()}).

Even if you are not familiar with \texttt{aggregate()}, I believe the above lines are quite
self-explanatory. You need to provide \texttt{aggregate()} with 3 things; the variable you want to
summarize (or only the data frame, if you want to summarize all variables), a list of grouping
variables and then the function that will be applied to each subgroup. And by the way, to test for
\texttt{NA}, one uses the function \texttt{is.na()} not something like \texttt{species\ ==\ "NA"} or anything like that.
\texttt{!is.na()} does the opposite (\texttt{!} reverses booleans, so \texttt{!TRUE} becomes \texttt{FALSE} and vice-versa).

You can easily add another grouping variable:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{aggregate}\NormalTok{(starwars\_no\_nas}\SpecialCharTok{$}\NormalTok{height,}
          \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{Sex =}\NormalTok{ starwars\_no\_nas}\SpecialCharTok{$}\NormalTok{sex,}
                    \StringTok{\textasciigrave{}}\AttributeTok{Hair color}\StringTok{\textasciigrave{}} \OtherTok{=}\NormalTok{ starwars\_no\_nas}\SpecialCharTok{$}\NormalTok{hair\_color),}
\NormalTok{          mean)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##       Sex    Hair color        x
## 1  female        auburn 150.0000
## 2    male  auburn, grey 180.0000
## 3    male auburn, white 182.0000
## 4  female         black 166.3333
## 5    male         black 176.2500
## 6    male         blond 176.6667
## 7  female        blonde 168.0000
## 8  female         brown 160.4000
## 9    male         brown 182.6667
## 10   male   brown, grey 178.0000
## 11   male          grey 170.0000
## 12 female          none 188.2500
## 13   male          none 182.2414
## 14   none          none 148.0000
## 15 female         white 167.0000
## 16   male         white 152.3333
\end{verbatim}

or use another function:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{aggregate}\NormalTok{(starwars\_no\_nas}\SpecialCharTok{$}\NormalTok{height,}
          \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{Sex =}\NormalTok{ starwars\_no\_nas}\SpecialCharTok{$}\NormalTok{sex),}
\NormalTok{          sd)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##              Sex        x
## 1         female 15.32256
## 2 hermaphroditic       NA
## 3           male 36.01075
## 4           none 49.14977
\end{verbatim}

(let's ignore the \texttt{NA}s). It is important to note that \texttt{aggregate()} returns a \texttt{data.frame} object.

You can only give one function to \texttt{aggregate()}, so if you need the mean and the standard deviation of \texttt{height},
you must do it in two steps.

Since R 4.1, a new infix operator \texttt{\textbar{}\textgreater{}} has been introduced, which is really handy for writing the kind of
code we've been looking at in this chapter. \texttt{\textbar{}\textgreater{}} is also called a pipe, or the \emph{base} pipe to distinguish
it from \emph{another} pipe that we'll discuss in the next section. For now, let's learn about \texttt{\textbar{}\textgreater{}}.
Consider the following:

\begin{Shaded}
\begin{Highlighting}[]
\DecValTok{10} \SpecialCharTok{|\textgreater{}} \FunctionTok{sqrt}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 3.162278
\end{verbatim}

This computes \texttt{sqrt(10)}; so what \texttt{\textbar{}\textgreater{}} does, is pass the left hand side (\texttt{10}, in the example above) to the
right hand side (\texttt{sqrt()}). Using \texttt{\textbar{}\textgreater{}} might seem more complicated and verbose than not using it, but you
will see in a bit why it can be useful. The next function I would like to introduce at this point is \texttt{with()}.
\texttt{with()} makes it possible to apply functions on \texttt{data.frame} columns without having to write \texttt{\$} all the time.
For example, consider this:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{mean}\NormalTok{(starwars}\SpecialCharTok{$}\NormalTok{height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 174.358
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{with}\NormalTok{(starwars,}
     \FunctionTok{mean}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 174.358
\end{verbatim}

The advantage of using \texttt{with()} is that we can directly reference \texttt{height} without using \texttt{\$}. Here again, this
is more verbose than simply using \texttt{\$}\ldots{} so why bother with it? It turns out that by combining \texttt{\textbar{}\textgreater{}} and \texttt{with()},
we can write very clean and concise code. Let's go back to a previous example to illustrate this idea:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars\_no\_nas }\OtherTok{\textless{}{-}} \FunctionTok{subset}\NormalTok{(starwars,}
                          \SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(height))}

\FunctionTok{aggregate}\NormalTok{(starwars\_no\_nas}\SpecialCharTok{$}\NormalTok{height,}
          \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{sex =}\NormalTok{ starwars\_no\_nas}\SpecialCharTok{$}\NormalTok{sex),}
\NormalTok{          mean)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##              sex        x
## 1         female 169.2667
## 2 hermaphroditic 175.0000
## 3           male 179.1053
## 4           none 131.2000
\end{verbatim}

First, we created a new dataset where we filtered out rows where \texttt{height} is \texttt{NA}. This dataset is useless otherwise,
but we need it for the next part, where we actually do what we want (computing the average \texttt{height} by \texttt{sex}).
Using \texttt{\textbar{}\textgreater{}} and \texttt{with()}, we can write this in one go:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{|\textgreater{}}
  \FunctionTok{subset}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(sex)) }\SpecialCharTok{|\textgreater{}}
  \FunctionTok{with}\NormalTok{(}\FunctionTok{aggregate}\NormalTok{(height,}
                 \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{Species =}\NormalTok{ species,}
                           \AttributeTok{Sex =}\NormalTok{ sex),}
\NormalTok{                 mean))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##           Species            Sex        x
## 1        Clawdite         female 168.0000
## 2           Human         female       NA
## 3        Kaminoan         female 213.0000
## 4        Mirialan         female 168.0000
## 5      Tholothian         female 184.0000
## 6         Togruta         female 178.0000
## 7         Twi'lek         female 178.0000
## 8            Hutt hermaphroditic 175.0000
## 9          Aleena           male  79.0000
## 10       Besalisk           male 198.0000
## 11         Cerean           male 198.0000
## 12       Chagrian           male 196.0000
## 13            Dug           male 112.0000
## 14           Ewok           male  88.0000
## 15      Geonosian           male 183.0000
## 16         Gungan           male 208.6667
## 17          Human           male       NA
## 18       Iktotchi           male 188.0000
## 19        Kaleesh           male 216.0000
## 20       Kaminoan           male 229.0000
## 21        Kel Dor           male 188.0000
## 22   Mon Calamari           male 180.0000
## 23           Muun           male 191.0000
## 24       Nautolan           male 196.0000
## 25      Neimodian           male 191.0000
## 26         Pau'an           male 206.0000
## 27       Quermian           male 264.0000
## 28         Rodian           male 173.0000
## 29        Skakoan           male 193.0000
## 30      Sullustan           male 160.0000
## 31          Toong           male 163.0000
## 32      Toydarian           male 137.0000
## 33     Trandoshan           male 190.0000
## 34        Twi'lek           male 180.0000
## 35     Vulptereen           male  94.0000
## 36        Wookiee           male 231.0000
## 37          Xexto           male 122.0000
## 38 Yoda's species           male  66.0000
## 39         Zabrak           male 173.0000
## 40          Droid           none       NA
\end{verbatim}

So let's unpack this. In the first two rows, using \texttt{\textbar{}\textgreater{}}, we pass the \texttt{starwars} \texttt{data.frame} to \texttt{subset()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{|\textgreater{}}
  \FunctionTok{subset}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(sex))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 83 x 14
##    name        height  mass hair_~1 skin_~2 eye_c~3 birth~4 sex   gender homew~5
##    <chr>        <int> <dbl> <chr>   <chr>   <chr>     <dbl> <chr> <chr>  <chr>  
##  1 Luke Skywa~    172    77 blond   fair    blue       19   male  mascu~ Tatooi~
##  2 C-3PO          167    75 <NA>    gold    yellow    112   none  mascu~ Tatooi~
##  3 R2-D2           96    32 <NA>    white,~ red        33   none  mascu~ Naboo  
##  4 Darth Vader    202   136 none    white   yellow     41.9 male  mascu~ Tatooi~
##  5 Leia Organa    150    49 brown   light   brown      19   fema~ femin~ Aldera~
##  6 Owen Lars      178   120 brown,~ light   blue       52   male  mascu~ Tatooi~
##  7 Beru White~    165    75 brown   light   blue       47   fema~ femin~ Tatooi~
##  8 R5-D4           97    32 <NA>    white,~ red        NA   none  mascu~ Tatooi~
##  9 Biggs Dark~    183    84 black   light   brown      24   male  mascu~ Tatooi~
## 10 Obi-Wan Ke~    182    77 auburn~ fair    blue-g~    57   male  mascu~ Stewjon
## # ... with 73 more rows, 4 more variables: species <chr>, films <list>,
## #   vehicles <list>, starships <list>, and abbreviated variable names
## #   1: hair_color, 2: skin_color, 3: eye_color, 4: birth_year, 5: homeworld
\end{verbatim}

as I explained before, this is exactly the same as \texttt{subset(starwars,\ !is.na(sex))}. Then, we pass the result of
\texttt{subset()} to the next function, \texttt{with()}. The first argument of \texttt{with()} must be a \texttt{data.frame}, and this is exactly
what \texttt{subset()} returns! So now the output of \texttt{subset()} is passed down to \texttt{with()}, which makes it now possible
to reference the columns of the \texttt{data.frame} in \texttt{aggregate()} directly. If you have a hard time understanding what
is going on, you can use \texttt{quote()} to see what's going on. \texttt{quote()} returns an expression with evaluating it:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{quote}\NormalTok{(}\FunctionTok{log}\NormalTok{(}\DecValTok{10}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## log(10)
\end{verbatim}

Why am I bring this up? Well, since \texttt{a\ \textbar{}\textgreater{}\ f()} is exactly equal to \texttt{f(a)}, quoting the code above will return
an expression with \texttt{\textbar{}\textgreater{}}. For instance:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{quote}\NormalTok{(}\DecValTok{10} \SpecialCharTok{|\textgreater{}} \FunctionTok{log}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## log(10)
\end{verbatim}

So let's quote the big block of code from above:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{quote}\NormalTok{(}
\NormalTok{  starwars }\SpecialCharTok{|\textgreater{}}
  \FunctionTok{subset}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(sex)) }\SpecialCharTok{|\textgreater{}}
  \FunctionTok{with}\NormalTok{(}\FunctionTok{aggregate}\NormalTok{(height,}
                 \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{Species =}\NormalTok{ species,}
                           \AttributeTok{Sex =}\NormalTok{ sex),}
\NormalTok{                 mean))}
\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## with(subset(starwars, !is.na(sex)), aggregate(height, by = list(Species = species, 
##     Sex = sex), mean))
\end{verbatim}

I think now you see why using \texttt{\textbar{}\textgreater{}} makes code much clearer; the nested expression you would need to write otherwise
is much less readable, unless you define intermediate objects. And without \texttt{with()}, this is what you
would need to write:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{b }\OtherTok{\textless{}{-}} \FunctionTok{subset}\NormalTok{(starwars, }\SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(height))}

\FunctionTok{aggregate}\NormalTok{(b}\SpecialCharTok{$}\NormalTok{height, }\AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{Species =}\NormalTok{ b}\SpecialCharTok{$}\NormalTok{species, }\AttributeTok{Sex =}\NormalTok{ b}\SpecialCharTok{$}\NormalTok{sex), mean)}
\end{Highlighting}
\end{Shaded}

To finish this section, let's say that you wanted to have the average \texttt{height} and \texttt{mass} by sex. In this case
you need to specify the columns in \texttt{aggregate()} with \texttt{cbind()} (let's use \texttt{na.rm\ =\ TRUE} again instead of
\texttt{subset()}ing the data beforehand):

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{|\textgreater{}}
  \FunctionTok{with}\NormalTok{(}\FunctionTok{aggregate}\NormalTok{(}\FunctionTok{cbind}\NormalTok{(height, mass),}
       \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{Sex =}\NormalTok{ sex),}
       \AttributeTok{FUN =}\NormalTok{ mean, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##              Sex   height       mass
## 1         female 169.2667   54.68889
## 2 hermaphroditic 175.0000 1358.00000
## 3           male 179.1053   81.00455
## 4           none 131.2000   69.75000
\end{verbatim}

Let's now continue with some more advanced operations using this fake dataset:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_base }\OtherTok{\textless{}{-}} \FunctionTok{as.data.frame}\NormalTok{(}
\NormalTok{    tibble}\SpecialCharTok{::}\FunctionTok{tribble}\NormalTok{(}
        \SpecialCharTok{\textasciitilde{}}\NormalTok{id, }\SpecialCharTok{\textasciitilde{}}\NormalTok{var1, }\SpecialCharTok{\textasciitilde{}}\NormalTok{var2, }\SpecialCharTok{\textasciitilde{}}\NormalTok{var3,}
        \DecValTok{1}\NormalTok{, }\DecValTok{1}\NormalTok{, }\FloatTok{0.2}\NormalTok{, }\FloatTok{0.3}\NormalTok{,}
        \DecValTok{2}\NormalTok{, }\FloatTok{1.4}\NormalTok{, }\FloatTok{1.9}\NormalTok{, }\FloatTok{4.1}\NormalTok{,}
        \DecValTok{3}\NormalTok{, }\FloatTok{0.1}\NormalTok{, }\FloatTok{2.8}\NormalTok{, }\FloatTok{8.9}\NormalTok{,}
        \DecValTok{4}\NormalTok{, }\FloatTok{1.7}\NormalTok{, }\FloatTok{1.9}\NormalTok{, }\FloatTok{7.6}
\NormalTok{        )}
\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_base}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   id var1 var2 var3
## 1  1  1.0  0.2  0.3
## 2  2  1.4  1.9  4.1
## 3  3  0.1  2.8  8.9
## 4  4  1.7  1.9  7.6
\end{verbatim}

Depending on what you want to do with this data, it is not in the right shape. For example, it
would not be possible to simply compute the average of \texttt{var1}, \texttt{var2} and \texttt{var3} for each \texttt{id}.
This is because this would require running \texttt{mean()} by row, but this is not very easy. This is
because R is not suited to row-based workflows. Well I'm lying a little bit here, it turns here
that R comes with a \texttt{rowMeans()} function. So this would work:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_base }\SpecialCharTok{|\textgreater{}}
  \FunctionTok{transform}\NormalTok{(}\AttributeTok{mean\_id =} \FunctionTok{rowMeans}\NormalTok{(}\FunctionTok{cbind}\NormalTok{(var1, var2, var3))) }\CommentTok{\#transform adds a column to a data.frame}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   id var1 var2 var3  mean_id
## 1  1  1.0  0.2  0.3 0.500000
## 2  2  1.4  1.9  4.1 2.466667
## 3  3  0.1  2.8  8.9 3.933333
## 4  4  1.7  1.9  7.6 3.733333
\end{verbatim}

But there is no \texttt{rowSD()} or \texttt{rowMax()}, etc\ldots{} so it is much better to reshape the data and put it in a
format that gives us maximum flexibility. To reshape the data, we'll be using the aptly-called \texttt{reshape()} command:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_long }\OtherTok{\textless{}{-}} \FunctionTok{reshape}\NormalTok{(survey\_data\_base,}
        \AttributeTok{varying =} \FunctionTok{list}\NormalTok{(}\DecValTok{2}\SpecialCharTok{:}\DecValTok{4}\NormalTok{), }\AttributeTok{v.names =} \StringTok{"variable"}\NormalTok{, }\AttributeTok{direction =} \StringTok{"long"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

We can now easily compute the average of \texttt{variable} for each \texttt{id}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{aggregate}\NormalTok{(survey\_data\_long}\SpecialCharTok{$}\NormalTok{variable,}
          \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{Id =}\NormalTok{ survey\_data\_long}\SpecialCharTok{$}\NormalTok{id),}
\NormalTok{          mean)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   Id        x
## 1  1 0.500000
## 2  2 2.466667
## 3  3 3.933333
## 4  4 3.733333
\end{verbatim}

or any other variable:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{aggregate}\NormalTok{(survey\_data\_long}\SpecialCharTok{$}\NormalTok{variable,}
          \AttributeTok{by =} \FunctionTok{list}\NormalTok{(}\AttributeTok{Id =}\NormalTok{ survey\_data\_long}\SpecialCharTok{$}\NormalTok{id),}
\NormalTok{          max)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   Id   x
## 1  1 1.0
## 2  2 4.1
## 3  3 8.9
## 4  4 7.6
\end{verbatim}

As you can see, R comes with very powerful functions right out of the box, ready to use. When I was
studying, unfortunately, my professors had been brought up on FORTRAN loops, so we had to do to all
this using loops (not reshaping, thankfully), which was not so easy.
Now that we have seen how \emph{base} R works, let's redo the analysis using \texttt{\{tidyverse\}} verbs.
The \texttt{\{tidyverse\}} provides many more functions, each of them doing only one single thing. You will
shortly see why this is quite important; by focusing on just one task, and by focusing on the data frame
as the central object, it becomes possible to build really complex workflows, piece by piece,
very easily.

But before deep diving into the \texttt{\{tidyverse\}}, let's take a moment to discuss about another infix
operator, \texttt{\%\textgreater{}\%}.

\hypertarget{smoking-is-bad-for-you-but-pipes-are-your-friend}{%
\subsection{Smoking is bad for you, but pipes are your friend}\label{smoking-is-bad-for-you-but-pipes-are-your-friend}}

The title of this section might sound weird at first, but by the end of it, you'll get this
(terrible) pun.

You probably know the following painting by René Magritte, \emph{La trahison des images}:

\includegraphics[width=9.72in]{assets/pas_une_pipe}

It turns out there's an R package from the \texttt{tidyverse} that is called \texttt{magrittr}. What does this
package do? This package introduced \emph{pipes} to R, way before \texttt{\textbar{}\textgreater{}} in R 4.1. Pipes are a concept
from the Unix operating system; if you're using a GNU+Linux distribution or macOS, you're basically
using a \emph{modern} unix (that's an oversimplification, but I'm an economist by training, and
outrageously oversimplifying things is what we do, deal with it). The \emph{magrittr} pipe is written as
\texttt{\%\textgreater{}\%}. Just like \texttt{\textbar{}\textgreater{}}, \texttt{\%\textgreater{}\%} takes the left hand side to feed it as the first argument of the
function in the right hand side. Try the following:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(magrittr)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\DecValTok{16} \SpecialCharTok{\%\textgreater{}\%}\NormalTok{ sqrt}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 4
\end{verbatim}

You can chain multiple functions, as you can with \texttt{\textbar{}\textgreater{}}:

\begin{Shaded}
\begin{Highlighting}[]
\DecValTok{16} \SpecialCharTok{\%\textgreater{}\%}
\NormalTok{  sqrt }\SpecialCharTok{\%\textgreater{}\%}
\NormalTok{  log}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1.386294
\end{verbatim}

But unlike with \texttt{\textbar{}\textgreater{}}, you can omit \texttt{()}. \texttt{\%\textgreater{}\%} also has other features. For example, you can
pipe things to other infix operators. For example, \texttt{+}. You can use \texttt{+} as usual:

\begin{Shaded}
\begin{Highlighting}[]
\DecValTok{2} \SpecialCharTok{+} \DecValTok{12}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 14
\end{verbatim}

Or as a prefix operator:

\begin{Shaded}
\begin{Highlighting}[]
\StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{(}\DecValTok{2}\NormalTok{, }\DecValTok{12}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 14
\end{verbatim}

You can use this notation with \texttt{\%\textgreater{}\%}:

\begin{Shaded}
\begin{Highlighting}[]
\DecValTok{16} \SpecialCharTok{\%\textgreater{}\%}\NormalTok{ sqrt }\SpecialCharTok{\%\textgreater{}\%} \StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{(}\DecValTok{18}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 22
\end{verbatim}

This also works using \texttt{\textbar{}\textgreater{}} since R version 4.2, but only if you use the \texttt{\_} pipe placeholder:

\begin{Shaded}
\begin{Highlighting}[]
\DecValTok{16} \SpecialCharTok{|\textgreater{}} \FunctionTok{sqrt}\NormalTok{() }\SpecialCharTok{|\textgreater{}} \StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{(}\AttributeTok{x =}\NormalTok{ \_, }\DecValTok{18}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 22
\end{verbatim}

The output of \texttt{16} (\texttt{16}) got fed to \texttt{sqrt()}, and the output of \texttt{sqrt(16)} (4) got fed to \texttt{+(18)}
(so we got \texttt{+(4,\ 18)} = 22). Without \texttt{\%\textgreater{}\%} you'd write the line just above like this:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sqrt}\NormalTok{(}\DecValTok{16}\NormalTok{) }\SpecialCharTok{+} \DecValTok{18}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 22
\end{verbatim}

Just like before, with \texttt{\textbar{}\textgreater{}}, this might seem overly complicated, but using these pipes will
make our code much more readable. I'm sure you'll be convinced by the end of this chapter.

\texttt{\%\textgreater{}\%} is not the only pipe operator in \texttt{magrittr}. There's \texttt{\%T\%}, \texttt{\%\textless{}\textgreater{}\%} and \texttt{\%\$\%}. All have their
uses, but are basically shortcuts to some common tasks with \texttt{\%\textgreater{}\%} plus another function. Which
means that you can live without them, and because of this, I will not discuss them.

\hypertarget{the-tidyverses-enfant-prodige-dplyr}{%
\subsection{\texorpdfstring{The \texttt{\{tidyverse\}}'s \emph{enfant prodige}: \texttt{\{dplyr\}}}{The \{tidyverse\}'s enfant prodige: \{dplyr\}}}\label{the-tidyverses-enfant-prodige-dplyr}}

The best way to get started with the tidyverse packages is to get to know \texttt{\{dplyr\}}. \texttt{\{dplyr\}}
provides a lot of very useful functions that makes it very easy to get discriptive statistics or
add new columns to your data.

\hypertarget{a-first-taste-of-data-manipulation-with-dplyr}{%
\subsubsection{\texorpdfstring{A first taste of data manipulation with \texttt{\{dplyr\}}}{A first taste of data manipulation with \{dplyr\}}}\label{a-first-taste-of-data-manipulation-with-dplyr}}

This section will walk you through a typical analysis using \texttt{\{dplyr\}} funcitons. Just go with it; I
will give more details in the next sections.

First, let's load \texttt{\{dplyr\}} and the included \texttt{starwars} dataset. Let's also take a look at the
first 5 lines of the dataset:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(dplyr)}

\FunctionTok{data}\NormalTok{(starwars)}

\FunctionTok{head}\NormalTok{(starwars)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 14
##   name         height  mass hair_~1 skin_~2 eye_c~3 birth~4 sex   gender homew~5
##   <chr>         <int> <dbl> <chr>   <chr>   <chr>     <dbl> <chr> <chr>  <chr>  
## 1 Luke Skywal~    172    77 blond   fair    blue       19   male  mascu~ Tatooi~
## 2 C-3PO           167    75 <NA>    gold    yellow    112   none  mascu~ Tatooi~
## 3 R2-D2            96    32 <NA>    white,~ red        33   none  mascu~ Naboo  
## 4 Darth Vader     202   136 none    white   yellow     41.9 male  mascu~ Tatooi~
## 5 Leia Organa     150    49 brown   light   brown      19   fema~ femin~ Aldera~
## 6 Owen Lars       178   120 brown,~ light   blue       52   male  mascu~ Tatooi~
## # ... with 4 more variables: species <chr>, films <list>, vehicles <list>,
## #   starships <list>, and abbreviated variable names 1: hair_color,
## #   2: skin_color, 3: eye_color, 4: birth_year, 5: homeworld
\end{verbatim}

\texttt{data(starwars)} loads the example dataset called \texttt{starwars} that is included in the package
\texttt{\{dplyr\}}. As I said earlier, this is just an example; you could have loaded an external dataset,
from a \texttt{.csv} file for instance. This does not matter for what comes next.

Like we saw earlier, R includes a lot of functions for descriptive statistics, such as \texttt{mean()},
\texttt{sd()}, \texttt{cov()}, and many more. What \texttt{\{dplyr\}} brings to the table is a grammar of data
manipulation that makes it very easy to apply descriptive statistics functions, or any other,
very easily.

Just like before, we are going to compute the average height by \texttt{sex}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(sex) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_height =} \FunctionTok{mean}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 5 x 2
##   sex            mean_height
##   <chr>                <dbl>
## 1 female                169.
## 2 hermaphroditic        175 
## 3 male                  179.
## 4 none                  131.
## 5 <NA>                  181.
\end{verbatim}

The very nice thing about using \texttt{\%\textgreater{}\%} and \texttt{\{dplyr\}} verbs/functions, is that this is really
readable. The above three lines can be translated like so in English:

\emph{Take the starwars dataset, then group by sex, then compute the mean height (for each subgroup) by
omitting missing values.}

\texttt{\%\textgreater{}\%} can be translated by ``then''. Without \texttt{\%\textgreater{}\%} you would need to change the code to:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{summarise}\NormalTok{(}\FunctionTok{group\_by}\NormalTok{(starwars, sex), }\FunctionTok{mean}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 5 x 2
##   sex            `mean(height, na.rm = TRUE)`
##   <chr>                                 <dbl>
## 1 female                                 169.
## 2 hermaphroditic                         175 
## 3 male                                   179.
## 4 none                                   131.
## 5 <NA>                                   181.
\end{verbatim}

Unlike with the \emph{base} approach, each function does only one thing. With the base function
\texttt{aggregate()} was used to also define the subgroups. This is not the case with \texttt{\{dplyr\}}; one
function to create the groups (\texttt{group\_by()}) and then one function to compute the summaries
(\texttt{summarise()}). Also, \texttt{group\_by()} creates a specific subgroup for individuals where \texttt{sex} is
missing. This is the last line in the data frame, where \texttt{sex} is \texttt{NA}. Another nice thing is that
you can specify the column containing the average height. I chose to name it \texttt{mean\_height}.

Now, let's suppose that we want to filter some data first:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(gender }\SpecialCharTok{==} \StringTok{"masculine"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(sex) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_height =} \FunctionTok{mean}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 3 x 2
##   sex            mean_height
##   <chr>                <dbl>
## 1 hermaphroditic        175 
## 2 male                  179.
## 3 none                  140
\end{verbatim}

Again, the \texttt{\%\textgreater{}\%} makes the above lines of code very easy to read. Without it, one would need to
write:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{summarise}\NormalTok{(}\FunctionTok{group\_by}\NormalTok{(}\FunctionTok{filter}\NormalTok{(starwars, gender }\SpecialCharTok{==} \StringTok{"masculine"}\NormalTok{), sex), }\FunctionTok{mean}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 3 x 2
##   sex            `mean(height, na.rm = TRUE)`
##   <chr>                                 <dbl>
## 1 hermaphroditic                         175 
## 2 male                                   179.
## 3 none                                   140
\end{verbatim}

I think you agree with me that this is not very readable. One way to make it more readable would
be to save intermediary variables:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{filtered\_data }\OtherTok{\textless{}{-}} \FunctionTok{filter}\NormalTok{(starwars, gender }\SpecialCharTok{==} \StringTok{"masculine"}\NormalTok{)}

\NormalTok{grouped\_data }\OtherTok{\textless{}{-}} \FunctionTok{group\_by}\NormalTok{(}\FunctionTok{filter}\NormalTok{(starwars, gender }\SpecialCharTok{==} \StringTok{"masculine"}\NormalTok{), sex)}

\FunctionTok{summarise}\NormalTok{(grouped\_data, }\FunctionTok{mean}\NormalTok{(height))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 3 x 2
##   sex            `mean(height)`
##   <chr>                   <dbl>
## 1 hermaphroditic            175
## 2 male                       NA
## 3 none                       NA
\end{verbatim}

But this can get very tedious. Once you're used to \texttt{\%\textgreater{}\%}, you won't go back to not use it.

Before continuing and to make things clearer; \texttt{filter()}, \texttt{group\_by()} and \texttt{summarise()} are
functions that are included in \texttt{\{dplyr\}}. \texttt{\%\textgreater{}\%} is actually a function from \texttt{\{magrittr\}}, but this
package gets loaded on the fly when you load \texttt{\{dplyr\}}, so you do not need to worry about it.

The result of all these operations that use \texttt{\{dplyr\}} functions are actually other datasets, or
\texttt{tibbles}. This means that you can save them in variable, or write them to disk, and then work with
these as any other datasets.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mean\_height }\OtherTok{\textless{}{-}}\NormalTok{ starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(sex) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\FunctionTok{mean}\NormalTok{(height))}

\FunctionTok{class}\NormalTok{(mean\_height)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "tbl_df"     "tbl"        "data.frame"
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{head}\NormalTok{(mean\_height)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 5 x 2
##   sex            `mean(height)`
##   <chr>                   <dbl>
## 1 female                     NA
## 2 hermaphroditic            175
## 3 male                       NA
## 4 none                       NA
## 5 <NA>                       NA
\end{verbatim}

You could then write this data to disk using \texttt{rio::export()} for instance. If you need more than
the mean of the height, you can keep adding as many functions as needed (another advantage over
\texttt{aggregate()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{summary\_table }\OtherTok{\textless{}{-}}\NormalTok{ starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(sex) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_height =} \FunctionTok{mean}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
            \AttributeTok{var\_height =} \FunctionTok{var}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
            \AttributeTok{n\_obs =} \FunctionTok{n}\NormalTok{())}

\NormalTok{summary\_table}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 5 x 4
##   sex            mean_height var_height n_obs
##   <chr>                <dbl>      <dbl> <int>
## 1 female                169.     235.      16
## 2 hermaphroditic        175       NA        1
## 3 male                  179.    1297.      60
## 4 none                  131.    2416.       6
## 5 <NA>                  181.       8.33     4
\end{verbatim}

I've added more functions, namely \texttt{var()}, to get the variance of height, and \texttt{n()}, which
is a function from \texttt{\{dplyr\}}, not base R, to get the number of observations. This is quite useful,
because we see that there is a group with only one individual. Let's focus on the
sexes for which we have more than 1 individual. Since we save all the previous operations (which
produce a \texttt{tibble}) in a variable, we can keep going from there:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{summary\_table2 }\OtherTok{\textless{}{-}}\NormalTok{ summary\_table }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(n\_obs }\SpecialCharTok{\textgreater{}} \DecValTok{1}\NormalTok{)}

\NormalTok{summary\_table2}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 4 x 4
##   sex    mean_height var_height n_obs
##   <chr>        <dbl>      <dbl> <int>
## 1 female        169.     235.      16
## 2 male          179.    1297.      60
## 3 none          131.    2416.       6
## 4 <NA>          181.       8.33     4
\end{verbatim}

As mentioned before, there's a lot of \texttt{NA}s; this is because by default, \texttt{mean()} and \texttt{var()}
return \texttt{NA} if even one single observation is \texttt{NA}. This is good, because it forces you to look at
the data to see what is going on. If you would get a number, even if there were \texttt{NA}s you could
very easily miss these missing values. It is better for functions to fail early and often than the
opposite. This is way we keep using \texttt{na.rm\ =\ TRUE} for \texttt{mean()} and \texttt{var()}.

Now let's actually take a look at the rows where \texttt{sex} is \texttt{NA}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{is.na}\NormalTok{(sex))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 4 x 14
##   name         height  mass hair_~1 skin_~2 eye_c~3 birth~4 sex   gender homew~5
##   <chr>         <int> <dbl> <chr>   <chr>   <chr>     <dbl> <chr> <chr>  <chr>  
## 1 Ric Olié        183    NA brown   fair    blue         NA <NA>  <NA>   Naboo  
## 2 Quarsh Pana~    183    NA black   dark    brown        62 <NA>  <NA>   Naboo  
## 3 Sly Moore       178    48 none    pale    white        NA <NA>  <NA>   Umbara 
## 4 Captain Pha~     NA    NA unknown unknown unknown      NA <NA>  <NA>   <NA>   
## # ... with 4 more variables: species <chr>, films <list>, vehicles <list>,
## #   starships <list>, and abbreviated variable names 1: hair_color,
## #   2: skin_color, 3: eye_color, 4: birth_year, 5: homeworld
\end{verbatim}

There's only 4 rows where \texttt{sex} is \texttt{NA}. Let's ignore them:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(sex)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(sex) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{ave\_height =} \FunctionTok{mean}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
            \AttributeTok{var\_height =} \FunctionTok{var}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
            \AttributeTok{n\_obs =} \FunctionTok{n}\NormalTok{()) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(n\_obs }\SpecialCharTok{\textgreater{}} \DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 3 x 4
##   sex    ave_height var_height n_obs
##   <chr>       <dbl>      <dbl> <int>
## 1 female       169.       235.    16
## 2 male         179.      1297.    60
## 3 none         131.      2416.     6
\end{verbatim}

And why not compute the same table, but first add another stratifying variable?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(sex)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(sex, eye\_color) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{ave\_height =} \FunctionTok{mean}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
            \AttributeTok{var\_height =} \FunctionTok{var}\NormalTok{(height, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
            \AttributeTok{n\_obs =} \FunctionTok{n}\NormalTok{()) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(n\_obs }\SpecialCharTok{\textgreater{}} \DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `summarise()` has grouped output by 'sex'. You can override using the `.groups`
## argument.
\end{verbatim}

\begin{verbatim}
## # A tibble: 12 x 5
## # Groups:   sex [3]
##    sex    eye_color ave_height var_height n_obs
##    <chr>  <chr>          <dbl>      <dbl> <int>
##  1 female black           196.      612.      2
##  2 female blue            167       118.      6
##  3 female brown           160        42       5
##  4 female hazel           178        NA       2
##  5 male   black           182      1197       7
##  6 male   blue            190.      434.     12
##  7 male   brown           167.     1663.     15
##  8 male   orange          181.     1306.      7
##  9 male   red             190.        0.5     2
## 10 male   unknown         136      6498       2
## 11 male   yellow          180.     2196.      9
## 12 none   red             131      3571       3
\end{verbatim}

Ok, that's it for a first taste. We have already discovered some very useful \texttt{\{dplyr\}} functions,
\texttt{filter()}, \texttt{group\_by()} and summarise \texttt{summarise()}.

Now, we are going to learn more about these functions in more detail.

\hypertarget{filter-the-rows-of-a-dataset-with-filter}{%
\subsubsection{\texorpdfstring{Filter the rows of a dataset with \texttt{filter()}}{Filter the rows of a dataset with filter()}}\label{filter-the-rows-of-a-dataset-with-filter}}

We're going to use the \texttt{Gasoline} dataset from the \texttt{plm} package, so install that first:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{install.packages}\NormalTok{(}\StringTok{"plm"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Then load the required data:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(Gasoline, }\AttributeTok{package =} \StringTok{"plm"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

and load dplyr:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(dplyr)}
\end{Highlighting}
\end{Shaded}

This dataset gives the consumption of gasoline for 18 countries from 1960 to 1978. When you load
the data like this, it is a standard \texttt{data.frame}. \texttt{\{dplyr\}} functions can be used on standard
\texttt{data.frame} objects, but also on \texttt{tibble}s. \texttt{tibble}s are just like data frame, but with a better
print method (and other niceties). I'll discuss the \texttt{\{tibble\}} package later, but for now, let's
convert the data to a \texttt{tibble} and change its name, and also transform the \texttt{country} column to
lower case:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\OtherTok{\textless{}{-}} \FunctionTok{as\_tibble}\NormalTok{(Gasoline)}

\NormalTok{gasoline }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{country =} \FunctionTok{tolower}\NormalTok{(country))}
\end{Highlighting}
\end{Shaded}

\texttt{filter()} is pretty straightforward. What if you would like to subset the data to focus on the
year 1969? Simple:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{filter}\NormalTok{(gasoline, year }\SpecialCharTok{==} \DecValTok{1969}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 6
##    country   year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>    <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria   1969     4.05    -6.15 -0.559    -8.79
##  2 belgium   1969     3.85    -5.86 -0.355    -8.52
##  3 canada    1969     4.86    -5.56 -1.04     -8.10
##  4 denmark   1969     4.17    -5.72 -0.407    -8.47
##  5 france    1969     3.77    -5.84 -0.315    -8.37
##  6 germany   1969     3.90    -5.83 -0.589    -8.44
##  7 greece    1969     4.89    -6.59 -0.180   -10.7 
##  8 ireland   1969     4.21    -6.38 -0.272    -8.95
##  9 italy     1969     3.74    -6.28 -0.248    -8.67
## 10 japan     1969     4.52    -6.16 -0.417    -9.61
## 11 netherla  1969     3.99    -5.88 -0.417    -8.63
## 12 norway    1969     4.09    -5.74 -0.338    -8.69
## 13 spain     1969     3.99    -5.60  0.669    -9.72
## 14 sweden    1969     3.99    -7.77 -2.73     -8.20
## 15 switzerl  1969     4.21    -5.91 -0.918    -8.47
## 16 turkey    1969     5.72    -7.39 -0.298   -12.5 
## 17 u.k.      1969     3.95    -6.03 -0.383    -8.47
## 18 u.s.a.    1969     4.84    -5.41 -1.22     -7.79
\end{verbatim}

Let's use \texttt{\%\textgreater{}\%}, since we're familiar with it now:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{==} \DecValTok{1969}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 6
##    country   year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>    <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria   1969     4.05    -6.15 -0.559    -8.79
##  2 belgium   1969     3.85    -5.86 -0.355    -8.52
##  3 canada    1969     4.86    -5.56 -1.04     -8.10
##  4 denmark   1969     4.17    -5.72 -0.407    -8.47
##  5 france    1969     3.77    -5.84 -0.315    -8.37
##  6 germany   1969     3.90    -5.83 -0.589    -8.44
##  7 greece    1969     4.89    -6.59 -0.180   -10.7 
##  8 ireland   1969     4.21    -6.38 -0.272    -8.95
##  9 italy     1969     3.74    -6.28 -0.248    -8.67
## 10 japan     1969     4.52    -6.16 -0.417    -9.61
## 11 netherla  1969     3.99    -5.88 -0.417    -8.63
## 12 norway    1969     4.09    -5.74 -0.338    -8.69
## 13 spain     1969     3.99    -5.60  0.669    -9.72
## 14 sweden    1969     3.99    -7.77 -2.73     -8.20
## 15 switzerl  1969     4.21    -5.91 -0.918    -8.47
## 16 turkey    1969     5.72    -7.39 -0.298   -12.5 
## 17 u.k.      1969     3.95    -6.03 -0.383    -8.47
## 18 u.s.a.    1969     4.84    -5.41 -1.22     -7.79
\end{verbatim}

You can also filter more than just one year, by using the \texttt{\%in\%} operator:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{\%in\%} \FunctionTok{seq}\NormalTok{(}\DecValTok{1969}\NormalTok{, }\DecValTok{1973}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 90 x 6
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1969     4.05    -6.15 -0.559    -8.79
##  2 austria  1970     4.08    -6.08 -0.597    -8.73
##  3 austria  1971     4.11    -6.04 -0.654    -8.64
##  4 austria  1972     4.13    -5.98 -0.596    -8.54
##  5 austria  1973     4.20    -5.90 -0.594    -8.49
##  6 belgium  1969     3.85    -5.86 -0.355    -8.52
##  7 belgium  1970     3.87    -5.80 -0.378    -8.45
##  8 belgium  1971     3.87    -5.76 -0.399    -8.41
##  9 belgium  1972     3.91    -5.71 -0.311    -8.36
## 10 belgium  1973     3.90    -5.64 -0.373    -8.31
## # ... with 80 more rows
\end{verbatim}

It is also possible use \texttt{between()}, a helper function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{between}\NormalTok{(year, }\DecValTok{1969}\NormalTok{, }\DecValTok{1973}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 90 x 6
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1969     4.05    -6.15 -0.559    -8.79
##  2 austria  1970     4.08    -6.08 -0.597    -8.73
##  3 austria  1971     4.11    -6.04 -0.654    -8.64
##  4 austria  1972     4.13    -5.98 -0.596    -8.54
##  5 austria  1973     4.20    -5.90 -0.594    -8.49
##  6 belgium  1969     3.85    -5.86 -0.355    -8.52
##  7 belgium  1970     3.87    -5.80 -0.378    -8.45
##  8 belgium  1971     3.87    -5.76 -0.399    -8.41
##  9 belgium  1972     3.91    -5.71 -0.311    -8.36
## 10 belgium  1973     3.90    -5.64 -0.373    -8.31
## # ... with 80 more rows
\end{verbatim}

To select non-consecutive years:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\DecValTok{1969}\NormalTok{, }\DecValTok{1973}\NormalTok{, }\DecValTok{1977}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 54 x 6
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1969     4.05    -6.15 -0.559    -8.79
##  2 austria  1973     4.20    -5.90 -0.594    -8.49
##  3 austria  1977     3.93    -5.83 -0.422    -8.25
##  4 belgium  1969     3.85    -5.86 -0.355    -8.52
##  5 belgium  1973     3.90    -5.64 -0.373    -8.31
##  6 belgium  1977     3.85    -5.56 -0.432    -8.14
##  7 canada   1969     4.86    -5.56 -1.04     -8.10
##  8 canada   1973     4.90    -5.41 -1.13     -7.94
##  9 canada   1977     4.81    -5.34 -1.07     -7.77
## 10 denmark  1969     4.17    -5.72 -0.407    -8.47
## # ... with 44 more rows
\end{verbatim}

\texttt{\%in\%} tests if an object is part of a set.

\hypertarget{select-columns-with-select}{%
\subsubsection{\texorpdfstring{Select columns with \texttt{select()}}{Select columns with select()}}\label{select-columns-with-select}}

While \texttt{filter()} allows you to keep or discard rows of data, \texttt{select()} allows you to keep or
discard entire columns. To keep columns:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(country, year, lrpmg)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 3
##    country  year  lrpmg
##    <chr>   <int>  <dbl>
##  1 austria  1960 -0.335
##  2 austria  1961 -0.351
##  3 austria  1962 -0.380
##  4 austria  1963 -0.414
##  5 austria  1964 -0.445
##  6 austria  1965 -0.497
##  7 austria  1966 -0.467
##  8 austria  1967 -0.506
##  9 austria  1968 -0.522
## 10 austria  1969 -0.559
## # ... with 332 more rows
\end{verbatim}

To discard them:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{country, }\SpecialCharTok{{-}}\NormalTok{year, }\SpecialCharTok{{-}}\NormalTok{lrpmg)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 3
##    lgaspcar lincomep lcarpcap
##       <dbl>    <dbl>    <dbl>
##  1     4.17    -6.47    -9.77
##  2     4.10    -6.43    -9.61
##  3     4.07    -6.41    -9.46
##  4     4.06    -6.37    -9.34
##  5     4.04    -6.32    -9.24
##  6     4.03    -6.29    -9.12
##  7     4.05    -6.25    -9.02
##  8     4.05    -6.23    -8.93
##  9     4.05    -6.21    -8.85
## 10     4.05    -6.15    -8.79
## # ... with 332 more rows
\end{verbatim}

To rename them:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(country, }\AttributeTok{date =}\NormalTok{ year, lrpmg)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 3
##    country  date  lrpmg
##    <chr>   <int>  <dbl>
##  1 austria  1960 -0.335
##  2 austria  1961 -0.351
##  3 austria  1962 -0.380
##  4 austria  1963 -0.414
##  5 austria  1964 -0.445
##  6 austria  1965 -0.497
##  7 austria  1966 -0.467
##  8 austria  1967 -0.506
##  9 austria  1968 -0.522
## 10 austria  1969 -0.559
## # ... with 332 more rows
\end{verbatim}

There's also \texttt{rename()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{rename}\NormalTok{(}\AttributeTok{date =}\NormalTok{ year)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 6
##    country  date lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77
##  2 austria  1961     4.10    -6.43 -0.351    -9.61
##  3 austria  1962     4.07    -6.41 -0.380    -9.46
##  4 austria  1963     4.06    -6.37 -0.414    -9.34
##  5 austria  1964     4.04    -6.32 -0.445    -9.24
##  6 austria  1965     4.03    -6.29 -0.497    -9.12
##  7 austria  1966     4.05    -6.25 -0.467    -9.02
##  8 austria  1967     4.05    -6.23 -0.506    -8.93
##  9 austria  1968     4.05    -6.21 -0.522    -8.85
## 10 austria  1969     4.05    -6.15 -0.559    -8.79
## # ... with 332 more rows
\end{verbatim}

\texttt{rename()} does not do any kind of selection, but just renames.

You can also use \texttt{select()} to re-order columns:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(year, country, lrpmg, }\FunctionTok{everything}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 6
##     year country  lrpmg lgaspcar lincomep lcarpcap
##    <int> <chr>    <dbl>    <dbl>    <dbl>    <dbl>
##  1  1960 austria -0.335     4.17    -6.47    -9.77
##  2  1961 austria -0.351     4.10    -6.43    -9.61
##  3  1962 austria -0.380     4.07    -6.41    -9.46
##  4  1963 austria -0.414     4.06    -6.37    -9.34
##  5  1964 austria -0.445     4.04    -6.32    -9.24
##  6  1965 austria -0.497     4.03    -6.29    -9.12
##  7  1966 austria -0.467     4.05    -6.25    -9.02
##  8  1967 austria -0.506     4.05    -6.23    -8.93
##  9  1968 austria -0.522     4.05    -6.21    -8.85
## 10  1969 austria -0.559     4.05    -6.15    -8.79
## # ... with 332 more rows
\end{verbatim}

\texttt{everything()} is a helper function, and there's also \texttt{starts\_with()}, and \texttt{ends\_with()}. For
example, what if we are only interested in columns whose name start with ``l''?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(}\FunctionTok{starts\_with}\NormalTok{(}\StringTok{"l"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 4
##    lgaspcar lincomep  lrpmg lcarpcap
##       <dbl>    <dbl>  <dbl>    <dbl>
##  1     4.17    -6.47 -0.335    -9.77
##  2     4.10    -6.43 -0.351    -9.61
##  3     4.07    -6.41 -0.380    -9.46
##  4     4.06    -6.37 -0.414    -9.34
##  5     4.04    -6.32 -0.445    -9.24
##  6     4.03    -6.29 -0.497    -9.12
##  7     4.05    -6.25 -0.467    -9.02
##  8     4.05    -6.23 -0.506    -8.93
##  9     4.05    -6.21 -0.522    -8.85
## 10     4.05    -6.15 -0.559    -8.79
## # ... with 332 more rows
\end{verbatim}

\texttt{ends\_with()} works in a similar fashion. There is also \texttt{contains()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(country, year, }\FunctionTok{contains}\NormalTok{(}\StringTok{"car"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 4
##    country  year lgaspcar lcarpcap
##    <chr>   <int>    <dbl>    <dbl>
##  1 austria  1960     4.17    -9.77
##  2 austria  1961     4.10    -9.61
##  3 austria  1962     4.07    -9.46
##  4 austria  1963     4.06    -9.34
##  5 austria  1964     4.04    -9.24
##  6 austria  1965     4.03    -9.12
##  7 austria  1966     4.05    -9.02
##  8 austria  1967     4.05    -8.93
##  9 austria  1968     4.05    -8.85
## 10 austria  1969     4.05    -8.79
## # ... with 332 more rows
\end{verbatim}

You can read more about these helper functions \href{https://tidyselect.r-lib.org/reference/language.html}{here}, but we're going to look more into
them in a coming section.

Another verb, similar to \texttt{select()}, is \texttt{pull()}. Let's compare the two:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(lrpmg)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 1
##     lrpmg
##     <dbl>
##  1 -0.335
##  2 -0.351
##  3 -0.380
##  4 -0.414
##  5 -0.445
##  6 -0.497
##  7 -0.467
##  8 -0.506
##  9 -0.522
## 10 -0.559
## # ... with 332 more rows
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{pull}\NormalTok{(lrpmg) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{head}\NormalTok{() }\CommentTok{\# using head() because there\textquotesingle{}s 337 elements in total}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] -0.3345476 -0.3513276 -0.3795177 -0.4142514 -0.4453354 -0.4970607
\end{verbatim}

\texttt{pull()}, unlike \texttt{select()}, does not return a \texttt{tibble}, but only the column you want, as a
vector.

\hypertarget{group-the-observations-of-your-dataset-with-group_by}{%
\subsubsection{\texorpdfstring{Group the observations of your dataset with \texttt{group\_by()}}{Group the observations of your dataset with group\_by()}}\label{group-the-observations-of-your-dataset-with-group_by}}

\texttt{group\_by()} is a very useful verb; as the name implies, it allows you to create groups and then,
for example, compute descriptive statistics by groups. For example, let's group our data by
country:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 6
## # Groups:   country [18]
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77
##  2 austria  1961     4.10    -6.43 -0.351    -9.61
##  3 austria  1962     4.07    -6.41 -0.380    -9.46
##  4 austria  1963     4.06    -6.37 -0.414    -9.34
##  5 austria  1964     4.04    -6.32 -0.445    -9.24
##  6 austria  1965     4.03    -6.29 -0.497    -9.12
##  7 austria  1966     4.05    -6.25 -0.467    -9.02
##  8 austria  1967     4.05    -6.23 -0.506    -8.93
##  9 austria  1968     4.05    -6.21 -0.522    -8.85
## 10 austria  1969     4.05    -6.15 -0.559    -8.79
## # ... with 332 more rows
\end{verbatim}

It looks like nothing much happened, but if you look at the second line of the output you can read
the following:

\begin{Shaded}
\begin{Highlighting}[]
\DocumentationTok{\#\# \# Groups:   country [18]}
\end{Highlighting}
\end{Shaded}

this means that the data is grouped, and every computation you will do now will take these groups
into account. It is also possible to group by more than one variable:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country, year)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 6
## # Groups:   country, year [342]
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77
##  2 austria  1961     4.10    -6.43 -0.351    -9.61
##  3 austria  1962     4.07    -6.41 -0.380    -9.46
##  4 austria  1963     4.06    -6.37 -0.414    -9.34
##  5 austria  1964     4.04    -6.32 -0.445    -9.24
##  6 austria  1965     4.03    -6.29 -0.497    -9.12
##  7 austria  1966     4.05    -6.25 -0.467    -9.02
##  8 austria  1967     4.05    -6.23 -0.506    -8.93
##  9 austria  1968     4.05    -6.21 -0.522    -8.85
## 10 austria  1969     4.05    -6.15 -0.559    -8.79
## # ... with 332 more rows
\end{verbatim}

and so on. You can then also ungroup:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country, year) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ungroup}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 6
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77
##  2 austria  1961     4.10    -6.43 -0.351    -9.61
##  3 austria  1962     4.07    -6.41 -0.380    -9.46
##  4 austria  1963     4.06    -6.37 -0.414    -9.34
##  5 austria  1964     4.04    -6.32 -0.445    -9.24
##  6 austria  1965     4.03    -6.29 -0.497    -9.12
##  7 austria  1966     4.05    -6.25 -0.467    -9.02
##  8 austria  1967     4.05    -6.23 -0.506    -8.93
##  9 austria  1968     4.05    -6.21 -0.522    -8.85
## 10 austria  1969     4.05    -6.15 -0.559    -8.79
## # ... with 332 more rows
\end{verbatim}

Once your data is grouped, the operations that will follow will be executed inside each group.

\hypertarget{get-summary-statistics-with-summarise}{%
\subsubsection{\texorpdfstring{Get summary statistics with \texttt{summarise()}}{Get summary statistics with summarise()}}\label{get-summary-statistics-with-summarise}}

Ok, now that we have learned the basic verbs, we can start to do more interesting stuff. For
example, one might want to compute the average gasoline consumption in each country, for
the whole period:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\FunctionTok{mean}\NormalTok{(lgaspcar))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 2
##    country  `mean(lgaspcar)`
##    <chr>               <dbl>
##  1 austria              4.06
##  2 belgium              3.92
##  3 canada               4.86
##  4 denmark              4.19
##  5 france               3.82
##  6 germany              3.89
##  7 greece               4.88
##  8 ireland              4.23
##  9 italy                3.73
## 10 japan                4.70
## 11 netherla             4.08
## 12 norway               4.11
## 13 spain                4.06
## 14 sweden               4.01
## 15 switzerl             4.24
## 16 turkey               5.77
## 17 u.k.                 3.98
## 18 u.s.a.               4.82
\end{verbatim}

\texttt{mean()} was given as an argument to \texttt{summarise()}, which is a \texttt{\{dplyr\}} verb. What we get is
another \texttt{tibble}, that contains the variable we used to group, as well as the average per country.
We can also rename this column:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_gaspcar =} \FunctionTok{mean}\NormalTok{(lgaspcar))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 2
##    country  mean_gaspcar
##    <chr>           <dbl>
##  1 austria          4.06
##  2 belgium          3.92
##  3 canada           4.86
##  4 denmark          4.19
##  5 france           3.82
##  6 germany          3.89
##  7 greece           4.88
##  8 ireland          4.23
##  9 italy            3.73
## 10 japan            4.70
## 11 netherla         4.08
## 12 norway           4.11
## 13 spain            4.06
## 14 sweden           4.01
## 15 switzerl         4.24
## 16 turkey           5.77
## 17 u.k.             3.98
## 18 u.s.a.           4.82
\end{verbatim}

and because the output is a \texttt{tibble}, we can continue to use \texttt{\{dplyr\}} verbs on it:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_gaspcar =} \FunctionTok{mean}\NormalTok{(lgaspcar)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(country }\SpecialCharTok{==} \StringTok{"france"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 2
##   country mean_gaspcar
##   <chr>          <dbl>
## 1 france          3.82
\end{verbatim}

\texttt{summarise()} is a very useful verb. For example, we can compute several descriptive statistics at once:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_gaspcar =} \FunctionTok{mean}\NormalTok{(lgaspcar),}
            \AttributeTok{sd\_gaspcar =} \FunctionTok{sd}\NormalTok{(lgaspcar),}
            \AttributeTok{max\_gaspcar =} \FunctionTok{max}\NormalTok{(lgaspcar),}
            \AttributeTok{min\_gaspcar =} \FunctionTok{min}\NormalTok{(lgaspcar))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 5
##    country  mean_gaspcar sd_gaspcar max_gaspcar min_gaspcar
##    <chr>           <dbl>      <dbl>       <dbl>       <dbl>
##  1 austria          4.06     0.0693        4.20        3.92
##  2 belgium          3.92     0.103         4.16        3.82
##  3 canada           4.86     0.0262        4.90        4.81
##  4 denmark          4.19     0.158         4.50        4.00
##  5 france           3.82     0.0499        3.91        3.75
##  6 germany          3.89     0.0239        3.93        3.85
##  7 greece           4.88     0.255         5.38        4.48
##  8 ireland          4.23     0.0437        4.33        4.16
##  9 italy            3.73     0.220         4.05        3.38
## 10 japan            4.70     0.684         6.00        3.95
## 11 netherla         4.08     0.286         4.65        3.71
## 12 norway           4.11     0.123         4.44        3.96
## 13 spain            4.06     0.317         4.75        3.62
## 14 sweden           4.01     0.0364        4.07        3.91
## 15 switzerl         4.24     0.102         4.44        4.05
## 16 turkey           5.77     0.329         6.16        5.14
## 17 u.k.             3.98     0.0479        4.10        3.91
## 18 u.s.a.           4.82     0.0219        4.86        4.79
\end{verbatim}

Because the output is a \texttt{tibble}, you can save it in a variable of course:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{desc\_gasoline }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_gaspcar =} \FunctionTok{mean}\NormalTok{(lgaspcar),}
            \AttributeTok{sd\_gaspcar =} \FunctionTok{sd}\NormalTok{(lgaspcar),}
            \AttributeTok{max\_gaspcar =} \FunctionTok{max}\NormalTok{(lgaspcar),}
            \AttributeTok{min\_gaspcar =} \FunctionTok{min}\NormalTok{(lgaspcar))}
\end{Highlighting}
\end{Shaded}

And then you can answer questions such as, \emph{which country has the maximum average gasoline
consumption?}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{desc\_gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{max}\NormalTok{(mean\_gaspcar) }\SpecialCharTok{==}\NormalTok{ mean\_gaspcar)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 5
##   country mean_gaspcar sd_gaspcar max_gaspcar min_gaspcar
##   <chr>          <dbl>      <dbl>       <dbl>       <dbl>
## 1 turkey          5.77      0.329        6.16        5.14
\end{verbatim}

Turns out it's Turkey. What about the minimum consumption?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{desc\_gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{min}\NormalTok{(mean\_gaspcar) }\SpecialCharTok{==}\NormalTok{ mean\_gaspcar)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 5
##   country mean_gaspcar sd_gaspcar max_gaspcar min_gaspcar
##   <chr>          <dbl>      <dbl>       <dbl>       <dbl>
## 1 italy           3.73      0.220        4.05        3.38
\end{verbatim}

Because the output of \texttt{\{dplyr\}} verbs is a tibble, it is possible to continue working with it. This
is one shortcoming of using the base \texttt{summary()} function. The object returned by that function is
not very easy to manipulate.

\hypertarget{adding-columns-with-mutate-and-transmute}{%
\subsubsection{\texorpdfstring{Adding columns with \texttt{mutate()} and \texttt{transmute()}}{Adding columns with mutate() and transmute()}}\label{adding-columns-with-mutate-and-transmute}}

\texttt{mutate()} adds a column to the \texttt{tibble}, which can contain any transformation of any other
variable:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\FunctionTok{n}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 7
## # Groups:   country [18]
##    country  year lgaspcar lincomep  lrpmg lcarpcap `n()`
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl> <int>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77    19
##  2 austria  1961     4.10    -6.43 -0.351    -9.61    19
##  3 austria  1962     4.07    -6.41 -0.380    -9.46    19
##  4 austria  1963     4.06    -6.37 -0.414    -9.34    19
##  5 austria  1964     4.04    -6.32 -0.445    -9.24    19
##  6 austria  1965     4.03    -6.29 -0.497    -9.12    19
##  7 austria  1966     4.05    -6.25 -0.467    -9.02    19
##  8 austria  1967     4.05    -6.23 -0.506    -8.93    19
##  9 austria  1968     4.05    -6.21 -0.522    -8.85    19
## 10 austria  1969     4.05    -6.15 -0.559    -8.79    19
## # ... with 332 more rows
\end{verbatim}

Using \texttt{mutate()} I've added a column that counts how many times the country appears in the \texttt{tibble},
using \texttt{n()}, another \texttt{\{dplyr\}} function. There's also \texttt{count()} and \texttt{tally()}, which we are going to
see further down. It is also possible to rename the column on the fly:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{count =} \FunctionTok{n}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 7
## # Groups:   country [18]
##    country  year lgaspcar lincomep  lrpmg lcarpcap count
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl> <int>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77    19
##  2 austria  1961     4.10    -6.43 -0.351    -9.61    19
##  3 austria  1962     4.07    -6.41 -0.380    -9.46    19
##  4 austria  1963     4.06    -6.37 -0.414    -9.34    19
##  5 austria  1964     4.04    -6.32 -0.445    -9.24    19
##  6 austria  1965     4.03    -6.29 -0.497    -9.12    19
##  7 austria  1966     4.05    -6.25 -0.467    -9.02    19
##  8 austria  1967     4.05    -6.23 -0.506    -8.93    19
##  9 austria  1968     4.05    -6.21 -0.522    -8.85    19
## 10 austria  1969     4.05    -6.15 -0.559    -8.79    19
## # ... with 332 more rows
\end{verbatim}

It is possible to do any arbitrary operation:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{spam =} \FunctionTok{exp}\NormalTok{(lgaspcar }\SpecialCharTok{+}\NormalTok{ lincomep))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 7
## # Groups:   country [18]
##    country  year lgaspcar lincomep  lrpmg lcarpcap   spam
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>  <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77 0.100 
##  2 austria  1961     4.10    -6.43 -0.351    -9.61 0.0978
##  3 austria  1962     4.07    -6.41 -0.380    -9.46 0.0969
##  4 austria  1963     4.06    -6.37 -0.414    -9.34 0.0991
##  5 austria  1964     4.04    -6.32 -0.445    -9.24 0.102 
##  6 austria  1965     4.03    -6.29 -0.497    -9.12 0.104 
##  7 austria  1966     4.05    -6.25 -0.467    -9.02 0.110 
##  8 austria  1967     4.05    -6.23 -0.506    -8.93 0.113 
##  9 austria  1968     4.05    -6.21 -0.522    -8.85 0.115 
## 10 austria  1969     4.05    -6.15 -0.559    -8.79 0.122 
## # ... with 332 more rows
\end{verbatim}

\texttt{transmute()} is the same as \texttt{mutate()}, but only returns the created variable:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{transmute}\NormalTok{(}\AttributeTok{spam =} \FunctionTok{exp}\NormalTok{(lgaspcar }\SpecialCharTok{+}\NormalTok{ lincomep))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 2
## # Groups:   country [18]
##    country   spam
##    <chr>    <dbl>
##  1 austria 0.100 
##  2 austria 0.0978
##  3 austria 0.0969
##  4 austria 0.0991
##  5 austria 0.102 
##  6 austria 0.104 
##  7 austria 0.110 
##  8 austria 0.113 
##  9 austria 0.115 
## 10 austria 0.122 
## # ... with 332 more rows
\end{verbatim}

\hypertarget{joining-tibbles-with-full_join-left_join-right_join-and-all-the-others}{%
\subsubsection{\texorpdfstring{Joining \texttt{tibble}s with \texttt{full\_join()}, \texttt{left\_join()}, \texttt{right\_join()} and all the others}{Joining tibbles with full\_join(), left\_join(), right\_join() and all the others}}\label{joining-tibbles-with-full_join-left_join-right_join-and-all-the-others}}

I will end this section on \texttt{\{dplyr\}} with the very useful verbs: the \texttt{*\_join()} verbs. Let's first
start by loading another dataset from the \texttt{plm} package. \texttt{SumHes} and let's convert it to \texttt{tibble}
and rename it:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(SumHes, }\AttributeTok{package =} \StringTok{"plm"}\NormalTok{)}

\NormalTok{pwt }\OtherTok{\textless{}{-}}\NormalTok{ SumHes }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{as\_tibble}\NormalTok{() }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{country =} \FunctionTok{tolower}\NormalTok{(country))}
\end{Highlighting}
\end{Shaded}

Let's take a quick look at the data:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{glimpse}\NormalTok{(pwt)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 3,250
## Columns: 7
## $ year    <int> 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 19~
## $ country <chr> "algeria", "algeria", "algeria", "algeria", "algeria", "algeri~
## $ opec    <fct> no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no~
## $ com     <fct> no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no~
## $ pop     <int> 10800, 11016, 11236, 11460, 11690, 11923, 12267, 12622, 12986,~
## $ gdp     <int> 1723, 1599, 1275, 1517, 1589, 1584, 1548, 1600, 1758, 1835, 18~
## $ sr      <dbl> 19.9, 21.1, 15.0, 13.9, 10.6, 11.0, 8.3, 11.3, 15.1, 18.2, 19.~
\end{verbatim}

We can merge both \texttt{gasoline} and \texttt{pwt} by country and year, as these two variables are common to
both datasets. There are more countries and years in the \texttt{pwt} dataset, so when merging both, and
depending on which function you use, you will either have \texttt{NA}'s for the variables where there is
no match, or rows that will be dropped. Let's start with \texttt{full\_join}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_full }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{full\_join}\NormalTok{(pwt, }\AttributeTok{by =} \FunctionTok{c}\NormalTok{(}\StringTok{"country"}\NormalTok{, }\StringTok{"year"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

Let's see which countries and years are included:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_full }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{count}\NormalTok{(country, year)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 3,307 x 3
##    country  year     n
##    <chr>   <int> <int>
##  1 algeria  1960     1
##  2 algeria  1961     1
##  3 algeria  1962     1
##  4 algeria  1963     1
##  5 algeria  1964     1
##  6 algeria  1965     1
##  7 algeria  1966     1
##  8 algeria  1967     1
##  9 algeria  1968     1
## 10 algeria  1969     1
## # ... with 3,297 more rows
\end{verbatim}

As you see, every country and year was included, but what happened for, say, the U.S.S.R? This country
is in \texttt{pwt} but not in \texttt{gasoline} at all:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_full }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(country }\SpecialCharTok{==} \StringTok{"u.s.s.r."}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 26 x 11
##    country   year lgaspcar lincomep lrpmg lcarp~1 opec  com      pop   gdp    sr
##    <chr>    <int>    <dbl>    <dbl> <dbl>   <dbl> <fct> <fct>  <int> <int> <dbl>
##  1 u.s.s.r.  1960       NA       NA    NA      NA no    yes   214400  2397  37.9
##  2 u.s.s.r.  1961       NA       NA    NA      NA no    yes   217896  2542  39.4
##  3 u.s.s.r.  1962       NA       NA    NA      NA no    yes   221449  2656  38.4
##  4 u.s.s.r.  1963       NA       NA    NA      NA no    yes   225060  2681  38.4
##  5 u.s.s.r.  1964       NA       NA    NA      NA no    yes   227571  2854  39.5
##  6 u.s.s.r.  1965       NA       NA    NA      NA no    yes   230109  3049  39.9
##  7 u.s.s.r.  1966       NA       NA    NA      NA no    yes   232676  3247  39.9
##  8 u.s.s.r.  1967       NA       NA    NA      NA no    yes   235272  3454  40.2
##  9 u.s.s.r.  1968       NA       NA    NA      NA no    yes   237896  3730  40.6
## 10 u.s.s.r.  1969       NA       NA    NA      NA no    yes   240550  3808  37.9
## # ... with 16 more rows, and abbreviated variable name 1: lcarpcap
\end{verbatim}

As you probably guessed, the variables from \texttt{gasoline} that are not included in \texttt{pwt} are filled with
\texttt{NA}s. One could remove all these lines and only keep countries for which these variables are not
\texttt{NA} everywhere with \texttt{filter()}, but there is a simpler solution:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_inner }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{inner\_join}\NormalTok{(pwt, }\AttributeTok{by =} \FunctionTok{c}\NormalTok{(}\StringTok{"country"}\NormalTok{, }\StringTok{"year"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

Let's use the \texttt{tabyl()} from the \texttt{janitor} packages which is a very nice alternative to the \texttt{table()}
function from base R:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(janitor)}

\NormalTok{gas\_pwt\_inner }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{tabyl}\NormalTok{(country)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  country  n    percent
##  austria 19 0.06666667
##  belgium 19 0.06666667
##   canada 19 0.06666667
##  denmark 19 0.06666667
##   france 19 0.06666667
##   greece 19 0.06666667
##  ireland 19 0.06666667
##    italy 19 0.06666667
##    japan 19 0.06666667
##   norway 19 0.06666667
##    spain 19 0.06666667
##   sweden 19 0.06666667
##   turkey 19 0.06666667
##     u.k. 19 0.06666667
##   u.s.a. 19 0.06666667
\end{verbatim}

Only countries with values in both datasets were returned. It's almost every country from \texttt{gasoline},
apart from Germany (called ``germany west'' in \texttt{pwt} and ``germany'' in \texttt{gasoline}. I left it as is to
provide an example of a country not in \texttt{pwt}). Let's also look at the variables:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{glimpse}\NormalTok{(gas\_pwt\_inner)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 285
## Columns: 11
## $ country  <chr> "austria", "austria", "austria", "austria", "austria", "austr~
## $ year     <int> 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1~
## $ lgaspcar <dbl> 4.173244, 4.100989, 4.073177, 4.059509, 4.037689, 4.033983, 4~
## $ lincomep <dbl> -6.474277, -6.426006, -6.407308, -6.370679, -6.322247, -6.294~
## $ lrpmg    <dbl> -0.3345476, -0.3513276, -0.3795177, -0.4142514, -0.4453354, -~
## $ lcarpcap <dbl> -9.766840, -9.608622, -9.457257, -9.343155, -9.237739, -9.123~
## $ opec     <fct> no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, n~
## $ com      <fct> no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, n~
## $ pop      <int> 7048, 7087, 7130, 7172, 7215, 7255, 7308, 7338, 7362, 7384, 7~
## $ gdp      <int> 5143, 5388, 5481, 5688, 5978, 6144, 6437, 6596, 6847, 7162, 7~
## $ sr       <dbl> 24.3, 24.5, 23.3, 22.9, 25.2, 25.2, 26.7, 25.6, 25.7, 26.1, 2~
\end{verbatim}

The variables from both datasets are in the joined data.

Contrast this to \texttt{semi\_join()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_semi }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{semi\_join}\NormalTok{(pwt, }\AttributeTok{by =} \FunctionTok{c}\NormalTok{(}\StringTok{"country"}\NormalTok{, }\StringTok{"year"}\NormalTok{))}

\FunctionTok{glimpse}\NormalTok{(gas\_pwt\_semi)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 285
## Columns: 6
## $ country  <chr> "austria", "austria", "austria", "austria", "austria", "austr~
## $ year     <int> 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1~
## $ lgaspcar <dbl> 4.173244, 4.100989, 4.073177, 4.059509, 4.037689, 4.033983, 4~
## $ lincomep <dbl> -6.474277, -6.426006, -6.407308, -6.370679, -6.322247, -6.294~
## $ lrpmg    <dbl> -0.3345476, -0.3513276, -0.3795177, -0.4142514, -0.4453354, -~
## $ lcarpcap <dbl> -9.766840, -9.608622, -9.457257, -9.343155, -9.237739, -9.123~
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_semi }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{tabyl}\NormalTok{(country)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  country  n    percent
##  austria 19 0.06666667
##  belgium 19 0.06666667
##   canada 19 0.06666667
##  denmark 19 0.06666667
##   france 19 0.06666667
##   greece 19 0.06666667
##  ireland 19 0.06666667
##    italy 19 0.06666667
##    japan 19 0.06666667
##   norway 19 0.06666667
##    spain 19 0.06666667
##   sweden 19 0.06666667
##   turkey 19 0.06666667
##     u.k. 19 0.06666667
##   u.s.a. 19 0.06666667
\end{verbatim}

Only columns of \texttt{gasoline} are returned, and only rows of \texttt{gasoline} that were matched with rows
from \texttt{pwt}. \texttt{semi\_join()} is not a commutative operation:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{pwt\_gas\_semi }\OtherTok{\textless{}{-}}\NormalTok{ pwt }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{semi\_join}\NormalTok{(gasoline, }\AttributeTok{by =} \FunctionTok{c}\NormalTok{(}\StringTok{"country"}\NormalTok{, }\StringTok{"year"}\NormalTok{))}

\FunctionTok{glimpse}\NormalTok{(pwt\_gas\_semi)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 285
## Columns: 7
## $ year    <int> 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 19~
## $ country <chr> "canada", "canada", "canada", "canada", "canada", "canada", "c~
## $ opec    <fct> no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no~
## $ com     <fct> no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no~
## $ pop     <int> 17910, 18270, 18614, 18963, 19326, 19678, 20049, 20411, 20744,~
## $ gdp     <int> 7258, 7261, 7605, 7876, 8244, 8664, 9093, 9231, 9582, 9975, 10~
## $ sr      <dbl> 22.7, 21.5, 22.1, 21.9, 22.9, 24.8, 25.4, 23.1, 22.6, 23.4, 21~
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_semi }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{tabyl}\NormalTok{(country)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  country  n    percent
##  austria 19 0.06666667
##  belgium 19 0.06666667
##   canada 19 0.06666667
##  denmark 19 0.06666667
##   france 19 0.06666667
##   greece 19 0.06666667
##  ireland 19 0.06666667
##    italy 19 0.06666667
##    japan 19 0.06666667
##   norway 19 0.06666667
##    spain 19 0.06666667
##   sweden 19 0.06666667
##   turkey 19 0.06666667
##     u.k. 19 0.06666667
##   u.s.a. 19 0.06666667
\end{verbatim}

The rows are the same, but not the columns.

\texttt{left\_join()} and \texttt{right\_join()} return all the rows from either the dataset that is on the
``left'' (the first argument of the fonction) or on the ``right'' (the second argument of the
function) but all columns from both datasets. So depending on which countries you're interested in,
you're going to use either one of these functions:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_left }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{left\_join}\NormalTok{(pwt, }\AttributeTok{by =} \FunctionTok{c}\NormalTok{(}\StringTok{"country"}\NormalTok{, }\StringTok{"year"}\NormalTok{))}

\NormalTok{gas\_pwt\_left }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{tabyl}\NormalTok{(country)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   country  n    percent
##   austria 19 0.05555556
##   belgium 19 0.05555556
##    canada 19 0.05555556
##   denmark 19 0.05555556
##    france 19 0.05555556
##   germany 19 0.05555556
##    greece 19 0.05555556
##   ireland 19 0.05555556
##     italy 19 0.05555556
##     japan 19 0.05555556
##  netherla 19 0.05555556
##    norway 19 0.05555556
##     spain 19 0.05555556
##    sweden 19 0.05555556
##  switzerl 19 0.05555556
##    turkey 19 0.05555556
##      u.k. 19 0.05555556
##    u.s.a. 19 0.05555556
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_right }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{right\_join}\NormalTok{(pwt, }\AttributeTok{by =} \FunctionTok{c}\NormalTok{(}\StringTok{"country"}\NormalTok{, }\StringTok{"year"}\NormalTok{))}

\NormalTok{gas\_pwt\_right }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{tabyl}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{head}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##     country  n percent
##     algeria 26   0.008
##      angola 26   0.008
##   argentina 26   0.008
##   australia 26   0.008
##     austria 26   0.008
##  bangladesh 26   0.008
\end{verbatim}

The last merge function is \texttt{anti\_join()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_anti }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{anti\_join}\NormalTok{(pwt, }\AttributeTok{by =} \FunctionTok{c}\NormalTok{(}\StringTok{"country"}\NormalTok{, }\StringTok{"year"}\NormalTok{))}

\FunctionTok{glimpse}\NormalTok{(gas\_pwt\_anti)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 57
## Columns: 6
## $ country  <chr> "germany", "germany", "germany", "germany", "germany", "germa~
## $ year     <int> 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1~
## $ lgaspcar <dbl> 3.916953, 3.885345, 3.871484, 3.848782, 3.868993, 3.861049, 3~
## $ lincomep <dbl> -6.159837, -6.120923, -6.094258, -6.068361, -6.013442, -5.966~
## $ lrpmg    <dbl> -0.1859108, -0.2309538, -0.3438417, -0.3746467, -0.3996526, -~
## $ lcarpcap <dbl> -9.342481, -9.183841, -9.037280, -8.913630, -8.811013, -8.711~
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gas\_pwt\_anti }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{tabyl}\NormalTok{(country)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   country  n   percent
##   germany 19 0.3333333
##  netherla 19 0.3333333
##  switzerl 19 0.3333333
\end{verbatim}

\texttt{gas\_pwt\_anti} has the columns the \texttt{gasoline} dataset as well as the only country from \texttt{gasoline}
that is not in \texttt{pwt}: ``germany''.

That was it for the basic \texttt{\{dplyr\}} verbs. Next, we're going to learn about \texttt{\{tidyr\}}.

\hypertarget{reshaping-and-sprucing-up-data-with-tidyr}{%
\subsection{\texorpdfstring{Reshaping and sprucing up data with \texttt{\{tidyr\}}}{Reshaping and sprucing up data with \{tidyr\}}}\label{reshaping-and-sprucing-up-data-with-tidyr}}

Note: this section is going to be a lot harder than anything you've seen until now. Reshaping
data is tricky, and to really grok it, you need time, and you need to run each line, and see what
happens. Take your time, and don't be discouraged.

Another important package from the \texttt{\{tidyverse\}} that goes hand in hand with \texttt{\{dplyr\}} is \texttt{\{tidyr\}}.
\texttt{\{tidyr\}} is the package you need when it's time to reshape data.

I will start by presenting \texttt{pivot\_wider()} and \texttt{pivot\_longer()}.

\hypertarget{pivot_wider-and-pivot_longer}{%
\subsubsection{\texorpdfstring{\texttt{pivot\_wider()} and \texttt{pivot\_longer()}}{pivot\_wider() and pivot\_longer()}}\label{pivot_wider-and-pivot_longer}}

Let's first create a fake dataset:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(tidyr)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data }\OtherTok{\textless{}{-}} \FunctionTok{tribble}\NormalTok{(}
  \SpecialCharTok{\textasciitilde{}}\NormalTok{id, }\SpecialCharTok{\textasciitilde{}}\NormalTok{variable, }\SpecialCharTok{\textasciitilde{}}\NormalTok{value,}
  \DecValTok{1}\NormalTok{, }\StringTok{"var1"}\NormalTok{, }\DecValTok{1}\NormalTok{,}
  \DecValTok{1}\NormalTok{, }\StringTok{"var2"}\NormalTok{, }\FloatTok{0.2}\NormalTok{,}
  \ConstantTok{NA}\NormalTok{, }\StringTok{"var3"}\NormalTok{, }\FloatTok{0.3}\NormalTok{,}
  \DecValTok{2}\NormalTok{, }\StringTok{"var1"}\NormalTok{, }\FloatTok{1.4}\NormalTok{,}
  \DecValTok{2}\NormalTok{, }\StringTok{"var2"}\NormalTok{, }\FloatTok{1.9}\NormalTok{,}
  \DecValTok{2}\NormalTok{, }\StringTok{"var3"}\NormalTok{, }\FloatTok{4.1}\NormalTok{,}
  \DecValTok{3}\NormalTok{, }\StringTok{"var1"}\NormalTok{, }\FloatTok{0.1}\NormalTok{,}
  \DecValTok{3}\NormalTok{, }\StringTok{"var2"}\NormalTok{, }\FloatTok{2.8}\NormalTok{,}
  \DecValTok{3}\NormalTok{, }\StringTok{"var3"}\NormalTok{, }\FloatTok{8.9}\NormalTok{,}
  \DecValTok{4}\NormalTok{, }\StringTok{"var1"}\NormalTok{, }\FloatTok{1.7}\NormalTok{,}
  \ConstantTok{NA}\NormalTok{, }\StringTok{"var2"}\NormalTok{, }\FloatTok{1.9}\NormalTok{,}
  \DecValTok{4}\NormalTok{, }\StringTok{"var3"}\NormalTok{, }\FloatTok{7.6}
\NormalTok{)}

\FunctionTok{head}\NormalTok{(survey\_data)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 3
##      id variable value
##   <dbl> <chr>    <dbl>
## 1     1 var1       1  
## 2     1 var2       0.2
## 3    NA var3       0.3
## 4     2 var1       1.4
## 5     2 var2       1.9
## 6     2 var3       4.1
\end{verbatim}

I used the \texttt{tribble()} function from the \texttt{\{tibble\}} package to create this fake dataset.
I'll discuss this package later, for now, let's focus on \texttt{\{tidyr\}.}

Let's suppose that we need the data to be in the wide format which means \texttt{var1}, \texttt{var2} and \texttt{var3}
need to be their own columns. To do this, we need to use the \texttt{pivot\_wider()} function. Why \emph{wide}?
Because the data set will be wide, meaning, having more columns than rows.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_wider}\NormalTok{(}\AttributeTok{id\_cols =}\NormalTok{ id,}
              \AttributeTok{names\_from =}\NormalTok{ variable,}
              \AttributeTok{values\_from =}\NormalTok{ value)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 5 x 4
##      id  var1  var2  var3
##   <dbl> <dbl> <dbl> <dbl>
## 1     1   1     0.2  NA  
## 2    NA  NA     1.9   0.3
## 3     2   1.4   1.9   4.1
## 4     3   0.1   2.8   8.9
## 5     4   1.7  NA     7.6
\end{verbatim}

Let's go through \texttt{pivot\_wider()}'s arguments: the first is \texttt{id\_cols\ =} which requires the variable
that uniquely identifies the rows to be supplied. \texttt{names\_from\ =} is where you input the variable that will
generate the names of the new columns. In our case, the \texttt{variable} colmuns has three values; \texttt{var1},
\texttt{var2} and \texttt{var3}, and these are now the names of the new columns. Finally, \texttt{values\_from\ =} is where
you can specify the column containing the values that will fill the data frame.
I find the argument names \texttt{names\_from\ =} and \texttt{values\_from\ =} quite explicit.

As you can see, there are some missing values. Let's suppose that we know that these missing values
are true 0's. \texttt{pivot\_wider()} has an argument called \texttt{values\_fill\ =} that makes it easy to replace
the missing values:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_wider}\NormalTok{(}\AttributeTok{id\_cols =}\NormalTok{ id,}
              \AttributeTok{names\_from =}\NormalTok{ variable,}
              \AttributeTok{values\_from =}\NormalTok{ value,}
              \AttributeTok{values\_fill =} \FunctionTok{list}\NormalTok{(}\AttributeTok{value =} \DecValTok{0}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 5 x 4
##      id  var1  var2  var3
##   <dbl> <dbl> <dbl> <dbl>
## 1     1   1     0.2   0  
## 2    NA   0     1.9   0.3
## 3     2   1.4   1.9   4.1
## 4     3   0.1   2.8   8.9
## 5     4   1.7   0     7.6
\end{verbatim}

A list of variables and their respective values to replace NA's with must be supplied to \texttt{values\_fill}.

Let's now use another dataset, which you can get from
\href{https://github.com/b-rodrigues/modern_R/tree/master/datasets/unemployment/all}{here}
(downloaded from: \url{http://www.statistiques.public.lu/stat/TableViewer/tableView.aspx?ReportId=12950\&IF_Language=eng\&MainTheme=2\&FldrName=3\&RFPath=91}). This data set gives the unemployment rate for each Luxembourguish
canton from 2001 to 2015. We will come back to this data later on to learn how to plot it. For now,
let's use it to learn more about \texttt{\{tidyr\}}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\OtherTok{\textless{}{-}}\NormalTok{ rio}\SpecialCharTok{::}\FunctionTok{import}\NormalTok{(}
      \StringTok{"https://raw.githubusercontent.com/b{-}rodrigues/modern\_R/master/datasets/unemployment/all/unemployment\_lux\_all.csv"}
\NormalTok{                       )}

\FunctionTok{head}\NormalTok{(unemp\_lux\_data)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   division year active_population of_which_non_wage_earners
## 1 Beaufort 2001               688                        85
## 2 Beaufort 2002               742                        85
## 3 Beaufort 2003               773                        85
## 4 Beaufort 2004               828                        80
## 5 Beaufort 2005               866                        96
## 6 Beaufort 2006               893                        87
##   of_which_wage_earners total_employed_population unemployed
## 1                   568                       653         35
## 2                   631                       716         26
## 3                   648                       733         40
## 4                   706                       786         42
## 5                   719                       815         51
## 6                   746                       833         60
##   unemployment_rate_in_percent
## 1                         5.09
## 2                         3.50
## 3                         5.17
## 4                         5.07
## 5                         5.89
## 6                         6.72
\end{verbatim}

Now, let's suppose that for our purposes, it would make more sense to have the data in a wide format,
where columns are ``divison times year'' and the value is the unemployment rate. This can be easily done
with providing more columns to \texttt{names\_from\ =}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data2 }\OtherTok{\textless{}{-}}\NormalTok{ unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{\%in\%} \FunctionTok{seq}\NormalTok{(}\DecValTok{2013}\NormalTok{, }\DecValTok{2017}\NormalTok{),}
         \FunctionTok{str\_detect}\NormalTok{(division, }\StringTok{".*ange$"}\NormalTok{),}
         \SpecialCharTok{!}\FunctionTok{str\_detect}\NormalTok{(division, }\StringTok{".*Canton.*"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(division, year, unemployment\_rate\_in\_percent) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{rowid\_to\_column}\NormalTok{()}

\NormalTok{unemp\_lux\_data2 }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_wider}\NormalTok{(}\AttributeTok{names\_from =} \FunctionTok{c}\NormalTok{(division, year),}
              \AttributeTok{values\_from =}\NormalTok{ unemployment\_rate\_in\_percent)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 48 x 49
##    rowid Bertr~1 Bertr~2 Bertr~3 Diffe~4 Diffe~5 Diffe~6 Dudel~7 Dudel~8 Dudel~9
##    <int>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
##  1     1    5.69   NA      NA       NA      NA      NA     NA      NA      NA   
##  2     2   NA       5.65   NA       NA      NA      NA     NA      NA      NA   
##  3     3   NA      NA       5.35    NA      NA      NA     NA      NA      NA   
##  4     4   NA      NA      NA       13.2    NA      NA     NA      NA      NA   
##  5     5   NA      NA      NA       NA      12.6    NA     NA      NA      NA   
##  6     6   NA      NA      NA       NA      NA      11.4   NA      NA      NA   
##  7     7   NA      NA      NA       NA      NA      NA      9.35   NA      NA   
##  8     8   NA      NA      NA       NA      NA      NA     NA       9.37   NA   
##  9     9   NA      NA      NA       NA      NA      NA     NA      NA       8.53
## 10    10   NA      NA      NA       NA      NA      NA     NA      NA      NA   
## # ... with 38 more rows, 39 more variables: Frisange_2013 <dbl>,
## #   Frisange_2014 <dbl>, Frisange_2015 <dbl>, Hesperange_2013 <dbl>,
## #   Hesperange_2014 <dbl>, Hesperange_2015 <dbl>, Leudelange_2013 <dbl>,
## #   Leudelange_2014 <dbl>, Leudelange_2015 <dbl>, Mondercange_2013 <dbl>,
## #   Mondercange_2014 <dbl>, Mondercange_2015 <dbl>, Pétange_2013 <dbl>,
## #   Pétange_2014 <dbl>, Pétange_2015 <dbl>, Rumelange_2013 <dbl>,
## #   Rumelange_2014 <dbl>, Rumelange_2015 <dbl>, Schifflange_2013 <dbl>, ...
\end{verbatim}

In the \texttt{filter()} statement, I only kept data from 2013 to 2017, ``division''s ending with the string
``ange'' (``division'' can be a canton or a commune, for example ``Canton Redange'', a canton, or
``Hesperange'' a commune), and removed the cantons as I'm only interested in communes. If you don't
understand this \texttt{filter()} statement, don't fret; this is not important for what follows. I then
only kept the columns I'm interested in and pivoted the data to a wide format. Also, I needed to
add a unique identifier to the data frame. For this, I used \texttt{rowid\_to\_column()} function, from the
\texttt{\{tibble\}} package, which adds a new column to the data frame with an id, going from 1 to the
number of rows in the data frame. If I did not add this identifier, the statement would work still:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data3 }\OtherTok{\textless{}{-}}\NormalTok{ unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{\%in\%} \FunctionTok{seq}\NormalTok{(}\DecValTok{2013}\NormalTok{, }\DecValTok{2017}\NormalTok{), }\FunctionTok{str\_detect}\NormalTok{(division, }\StringTok{".*ange$"}\NormalTok{), }\SpecialCharTok{!}\FunctionTok{str\_detect}\NormalTok{(division, }\StringTok{".*Canton.*"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(division, year, unemployment\_rate\_in\_percent)}

\NormalTok{unemp\_lux\_data3 }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_wider}\NormalTok{(}\AttributeTok{names\_from =} \FunctionTok{c}\NormalTok{(division, year), }\AttributeTok{values\_from =}\NormalTok{ unemployment\_rate\_in\_percent)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 48
##   Bertrange_2013 Bertr~1 Bertr~2 Diffe~3 Diffe~4 Diffe~5 Dudel~6 Dudel~7 Dudel~8
##            <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
## 1           5.69    5.65    5.35    13.2    12.6    11.4    9.35    9.37    8.53
## # ... with 39 more variables: Frisange_2013 <dbl>, Frisange_2014 <dbl>,
## #   Frisange_2015 <dbl>, Hesperange_2013 <dbl>, Hesperange_2014 <dbl>,
## #   Hesperange_2015 <dbl>, Leudelange_2013 <dbl>, Leudelange_2014 <dbl>,
## #   Leudelange_2015 <dbl>, Mondercange_2013 <dbl>, Mondercange_2014 <dbl>,
## #   Mondercange_2015 <dbl>, Pétange_2013 <dbl>, Pétange_2014 <dbl>,
## #   Pétange_2015 <dbl>, Rumelange_2013 <dbl>, Rumelange_2014 <dbl>,
## #   Rumelange_2015 <dbl>, Schifflange_2013 <dbl>, Schifflange_2014 <dbl>, ...
\end{verbatim}

and actually look even better, but only because there are no repeated values; there is only one
unemployment rate for each ``commune times year''. I will come back to this later on, with another
example that might be clearer. These last two code blocks are intense; make sure you go through
each lien step by step and understand what is going on.

You might have noticed that because there is no data for the years 2016 and 2017, these columns do
not appear in the data. But suppose that we need to have these columns, so that a colleague from
another department can fill in the values. This is possible by providing a data frame with the
detailed specifications of the result data frame. This optional data frame must have at least two
columns, \texttt{.name}, which are the column names you want, and \texttt{.value} which contains the values.
Also, the function that uses this spec is a \texttt{pivot\_wider\_spec()}, and not \texttt{pivot\_wider()}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_spec }\OtherTok{\textless{}{-}}\NormalTok{ unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%} 
\NormalTok{  tidyr}\SpecialCharTok{::}\FunctionTok{expand}\NormalTok{(division,}
         \AttributeTok{year =} \FunctionTok{c}\NormalTok{(year, }\DecValTok{2016}\NormalTok{, }\DecValTok{2017}\NormalTok{),}
         \AttributeTok{.value =} \StringTok{"unemployment\_rate\_in\_percent"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{unite}\NormalTok{(}\StringTok{".name"}\NormalTok{, division, year, }\AttributeTok{remove =} \ConstantTok{FALSE}\NormalTok{)}

\NormalTok{unemp\_spec}
\end{Highlighting}
\end{Shaded}

Here, I use another function, \texttt{tidyr::expand()}, which returns every combinations (cartesian product)
of every variable from a dataset.

To make it work, we still need to create a column that uniquely identifies each row in the data:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data4 }\OtherTok{\textless{}{-}}\NormalTok{ unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(division, year, unemployment\_rate\_in\_percent) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{rowid\_to\_column}\NormalTok{() }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_wider\_spec}\NormalTok{(}\AttributeTok{spec =}\NormalTok{ unemp\_spec) }

\NormalTok{unemp\_lux\_data4}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1,770 x 2,007
##    rowid Beauf~1 Beauf~2 Beauf~3 Beauf~4 Beauf~5 Beauf~6 Beauf~7 Beauf~8 Beauf~9
##    <int>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
##  1     1    5.09    NA     NA      NA      NA      NA       NA     NA      NA   
##  2     2   NA        3.5   NA      NA      NA      NA       NA     NA      NA   
##  3     3   NA       NA      5.17   NA      NA      NA       NA     NA      NA   
##  4     4   NA       NA     NA       5.07   NA      NA       NA     NA      NA   
##  5     5   NA       NA     NA      NA       5.89   NA       NA     NA      NA   
##  6     6   NA       NA     NA      NA      NA       6.72    NA     NA      NA   
##  7     7   NA       NA     NA      NA      NA      NA        4.3   NA      NA   
##  8     8   NA       NA     NA      NA      NA      NA       NA      7.08   NA   
##  9     9   NA       NA     NA      NA      NA      NA       NA     NA       8.52
## 10    10   NA       NA     NA      NA      NA      NA       NA     NA      NA   
## # ... with 1,760 more rows, 1,997 more variables: Beaufort_2010 <dbl>,
## #   Beaufort_2011 <dbl>, Beaufort_2012 <dbl>, Beaufort_2013 <dbl>,
## #   Beaufort_2014 <dbl>, Beaufort_2015 <dbl>, Beaufort_2016 <dbl>,
## #   Beaufort_2017 <dbl>, Bech_2001 <dbl>, Bech_2002 <dbl>, Bech_2003 <dbl>,
## #   Bech_2004 <dbl>, Bech_2005 <dbl>, Bech_2006 <dbl>, Bech_2007 <dbl>,
## #   Bech_2008 <dbl>, Bech_2009 <dbl>, Bech_2010 <dbl>, Bech_2011 <dbl>,
## #   Bech_2012 <dbl>, Bech_2013 <dbl>, Bech_2014 <dbl>, Bech_2015 <dbl>, ...
\end{verbatim}

You can notice that now we have columns for 2016 and 2017 too. Let's clean the data a little bit more:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data4 }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{rowid) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{fill}\NormalTok{(}\FunctionTok{matches}\NormalTok{(}\StringTok{".*"}\NormalTok{), }\AttributeTok{.direction =} \StringTok{"down"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{slice}\NormalTok{(}\FunctionTok{n}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 2,006
##   Beaufort_2001 Beaufo~1 Beauf~2 Beauf~3 Beauf~4 Beauf~5 Beauf~6 Beauf~7 Beauf~8
##           <dbl>    <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
## 1          5.09      3.5    5.17    5.07    5.89    6.72     4.3    7.08    8.52
## # ... with 1,997 more variables: Beaufort_2010 <dbl>, Beaufort_2011 <dbl>,
## #   Beaufort_2012 <dbl>, Beaufort_2013 <dbl>, Beaufort_2014 <dbl>,
## #   Beaufort_2015 <dbl>, Beaufort_2016 <dbl>, Beaufort_2017 <dbl>,
## #   Bech_2001 <dbl>, Bech_2002 <dbl>, Bech_2003 <dbl>, Bech_2004 <dbl>,
## #   Bech_2005 <dbl>, Bech_2006 <dbl>, Bech_2007 <dbl>, Bech_2008 <dbl>,
## #   Bech_2009 <dbl>, Bech_2010 <dbl>, Bech_2011 <dbl>, Bech_2012 <dbl>,
## #   Bech_2013 <dbl>, Bech_2014 <dbl>, Bech_2015 <dbl>, Bech_2016 <dbl>, ...
\end{verbatim}

We will learn about \texttt{fill()}, anoher \texttt{\{tidyr\}} function a bit later in this chapter, but its basic
purpose is to fill rows with whatever value comes before or after the missing values. \texttt{slice(n())}
then only keeps the last row of the data frame, which is the row that contains all the values (expect
for 2016 and 2017, which has missing values, as we wanted).

Here is another example of the importance of having an identifier column when using a spec:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(mtcars)}
\NormalTok{mtcars\_spec }\OtherTok{\textless{}{-}}\NormalTok{ mtcars }\SpecialCharTok{\%\textgreater{}\%} 
\NormalTok{    tidyr}\SpecialCharTok{::}\FunctionTok{expand}\NormalTok{(am, cyl, }\AttributeTok{.value =} \StringTok{"mpg"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{unite}\NormalTok{(}\StringTok{".name"}\NormalTok{, am, cyl, }\AttributeTok{remove =} \ConstantTok{FALSE}\NormalTok{)}

\NormalTok{mtcars\_spec}
\end{Highlighting}
\end{Shaded}

We can now transform the data:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{pivot\_wider\_spec}\NormalTok{(}\AttributeTok{spec =}\NormalTok{ mtcars\_spec)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 32 x 14
##     disp    hp  drat    wt  qsec    vs  gear  carb `0_4` `0_6` `0_8` `1_4` `1_6`
##    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1  160    110  3.9   2.62  16.5     0     4     4  NA    NA    NA    NA      21
##  2  160    110  3.9   2.88  17.0     0     4     4  NA    NA    NA    NA      21
##  3  108     93  3.85  2.32  18.6     1     4     1  NA    NA    NA    22.8    NA
##  4  258    110  3.08  3.22  19.4     1     3     1  NA    21.4  NA    NA      NA
##  5  360    175  3.15  3.44  17.0     0     3     2  NA    NA    18.7  NA      NA
##  6  225    105  2.76  3.46  20.2     1     3     1  NA    18.1  NA    NA      NA
##  7  360    245  3.21  3.57  15.8     0     3     4  NA    NA    14.3  NA      NA
##  8  147.    62  3.69  3.19  20       1     4     2  24.4  NA    NA    NA      NA
##  9  141.    95  3.92  3.15  22.9     1     4     2  22.8  NA    NA    NA      NA
## 10  168.   123  3.92  3.44  18.3     1     4     4  NA    19.2  NA    NA      NA
## # ... with 22 more rows, and 1 more variable: `1_8` <dbl>
\end{verbatim}

As you can see, there are several values of ``mpg'' for some combinations of ``am'' times ``cyl''. If
we remove the other columns, each row will not be uniquely identified anymore. This results in a
warning message, and a tibble that contains list-columns:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(am, cyl, mpg) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_wider\_spec}\NormalTok{(}\AttributeTok{spec =}\NormalTok{ mtcars\_spec)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: Values from `mpg` are not uniquely identified; output will contain list-cols.
## * Use `values_fn = list` to suppress this warning.
## * Use `values_fn = {summary_fun}` to summarise duplicates.
## * Use the following dplyr code to identify duplicates.
##   {data} %>%
##     dplyr::group_by(am, cyl) %>%
##     dplyr::summarise(n = dplyr::n(), .groups = "drop") %>%
##     dplyr::filter(n > 1L)
\end{verbatim}

\begin{verbatim}
## # A tibble: 1 x 6
##   `0_4`     `0_6`     `0_8`      `1_4`     `1_6`     `1_8`    
##   <list>    <list>    <list>     <list>    <list>    <list>   
## 1 <dbl [3]> <dbl [4]> <dbl [12]> <dbl [8]> <dbl [3]> <dbl [2]>
\end{verbatim}

We are going to learn about list-columns in the next section. List-columns are very powerful, and
mastering them will be important. But generally speaking, when reshaping data, if you get list-columns
back it often means that something went wrong.

So you have to be careful with this.

\texttt{pivot\_longer()} is used when you need to go from a wide to a long dataset, meaning, a dataset
where there are some columns that should not be columns, but rather, the levels of a factor
variable. Let's suppose that the ``am'' column is split into two columns, \texttt{1} for automatic and \texttt{0}
for manual transmissions, and that the values filling these colums are miles per gallon, ``mpg'':

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_wide\_am }\OtherTok{\textless{}{-}}\NormalTok{ mtcars }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_wider}\NormalTok{(}\AttributeTok{names\_from =}\NormalTok{ am, }\AttributeTok{values\_from =}\NormalTok{ mpg)}

\NormalTok{mtcars\_wide\_am }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(}\StringTok{\textasciigrave{}}\AttributeTok{0}\StringTok{\textasciigrave{}}\NormalTok{, }\StringTok{\textasciigrave{}}\AttributeTok{1}\StringTok{\textasciigrave{}}\NormalTok{, }\FunctionTok{everything}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 32 x 11
##      `0`   `1`   cyl  disp    hp  drat    wt  qsec    vs  gear  carb
##    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1  NA    21       6  160    110  3.9   2.62  16.5     0     4     4
##  2  NA    21       6  160    110  3.9   2.88  17.0     0     4     4
##  3  NA    22.8     4  108     93  3.85  2.32  18.6     1     4     1
##  4  21.4  NA       6  258    110  3.08  3.22  19.4     1     3     1
##  5  18.7  NA       8  360    175  3.15  3.44  17.0     0     3     2
##  6  18.1  NA       6  225    105  2.76  3.46  20.2     1     3     1
##  7  14.3  NA       8  360    245  3.21  3.57  15.8     0     3     4
##  8  24.4  NA       4  147.    62  3.69  3.19  20       1     4     2
##  9  22.8  NA       4  141.    95  3.92  3.15  22.9     1     4     2
## 10  19.2  NA       6  168.   123  3.92  3.44  18.3     1     4     4
## # ... with 22 more rows
\end{verbatim}

As you can see, the ``0'' and ``1'' columns should not be their own columns, unless there is a very
specific and good reason they should\ldots{} but rather, they should be the levels of another column (in
our case, ``am'').

We can go back to a long dataset like so:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_wide\_am }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_longer}\NormalTok{(}\AttributeTok{cols =} \FunctionTok{c}\NormalTok{(}\StringTok{\textasciigrave{}}\AttributeTok{1}\StringTok{\textasciigrave{}}\NormalTok{, }\StringTok{\textasciigrave{}}\AttributeTok{0}\StringTok{\textasciigrave{}}\NormalTok{), }\AttributeTok{names\_to =} \StringTok{"am"}\NormalTok{, }\AttributeTok{values\_to =} \StringTok{"mpg"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(am, mpg, }\FunctionTok{everything}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 64 x 11
##    am      mpg   cyl  disp    hp  drat    wt  qsec    vs  gear  carb
##    <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1 1      21       6   160   110  3.9   2.62  16.5     0     4     4
##  2 0      NA       6   160   110  3.9   2.62  16.5     0     4     4
##  3 1      21       6   160   110  3.9   2.88  17.0     0     4     4
##  4 0      NA       6   160   110  3.9   2.88  17.0     0     4     4
##  5 1      22.8     4   108    93  3.85  2.32  18.6     1     4     1
##  6 0      NA       4   108    93  3.85  2.32  18.6     1     4     1
##  7 1      NA       6   258   110  3.08  3.22  19.4     1     3     1
##  8 0      21.4     6   258   110  3.08  3.22  19.4     1     3     1
##  9 1      NA       8   360   175  3.15  3.44  17.0     0     3     2
## 10 0      18.7     8   360   175  3.15  3.44  17.0     0     3     2
## # ... with 54 more rows
\end{verbatim}

In the cols argument, you need to list all the variables that need to be transformed. Only \texttt{1} and
\texttt{0} must be pivoted, so I list them. Just for illustration purposes, imagine that we would need
to pivot 50 columns. It would be faster to list the columns that do not need to be pivoted. This
can be achieved by listing the columns that must be excluded with \texttt{-} in front, and maybe using
\texttt{match()} with a regular expression:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_wide\_am }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_longer}\NormalTok{(}\AttributeTok{cols =} \SpecialCharTok{{-}}\FunctionTok{matches}\NormalTok{(}\StringTok{"\^{}[[:alpha:]]"}\NormalTok{),}
               \AttributeTok{names\_to =} \StringTok{"am"}\NormalTok{,}
               \AttributeTok{values\_to =} \StringTok{"mpg"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(am, mpg, }\FunctionTok{everything}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 64 x 11
##    am      mpg   cyl  disp    hp  drat    wt  qsec    vs  gear  carb
##    <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1 1      21       6   160   110  3.9   2.62  16.5     0     4     4
##  2 0      NA       6   160   110  3.9   2.62  16.5     0     4     4
##  3 1      21       6   160   110  3.9   2.88  17.0     0     4     4
##  4 0      NA       6   160   110  3.9   2.88  17.0     0     4     4
##  5 1      22.8     4   108    93  3.85  2.32  18.6     1     4     1
##  6 0      NA       4   108    93  3.85  2.32  18.6     1     4     1
##  7 1      NA       6   258   110  3.08  3.22  19.4     1     3     1
##  8 0      21.4     6   258   110  3.08  3.22  19.4     1     3     1
##  9 1      NA       8   360   175  3.15  3.44  17.0     0     3     2
## 10 0      18.7     8   360   175  3.15  3.44  17.0     0     3     2
## # ... with 54 more rows
\end{verbatim}

Every column that starts with a letter is ok, so there is no need to pivot them. I use the \texttt{match()}
function with a regular expression so that I don't have to type the names of all the columns. \texttt{select()}
is used to re-order the columns, only for viewing purposes

\texttt{names\_to\ =} takes a string as argument, which will be the name of the name column containing the
levels \texttt{0} and \texttt{1}, and \texttt{values\_to\ =} also takes a string as argument, which will be the name of
the column containing the values. Finally, you can see that there are a lot of \texttt{NA}s in the
output. These can be removed easily:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_wide\_am }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_longer}\NormalTok{(}\AttributeTok{cols =} \FunctionTok{c}\NormalTok{(}\StringTok{\textasciigrave{}}\AttributeTok{1}\StringTok{\textasciigrave{}}\NormalTok{, }\StringTok{\textasciigrave{}}\AttributeTok{0}\StringTok{\textasciigrave{}}\NormalTok{), }\AttributeTok{names\_to =} \StringTok{"am"}\NormalTok{, }\AttributeTok{values\_to =} \StringTok{"mpg"}\NormalTok{, }\AttributeTok{values\_drop\_na =} \ConstantTok{TRUE}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(am, mpg, }\FunctionTok{everything}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 32 x 11
##    am      mpg   cyl  disp    hp  drat    wt  qsec    vs  gear  carb
##    <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1 1      21       6  160    110  3.9   2.62  16.5     0     4     4
##  2 1      21       6  160    110  3.9   2.88  17.0     0     4     4
##  3 1      22.8     4  108     93  3.85  2.32  18.6     1     4     1
##  4 0      21.4     6  258    110  3.08  3.22  19.4     1     3     1
##  5 0      18.7     8  360    175  3.15  3.44  17.0     0     3     2
##  6 0      18.1     6  225    105  2.76  3.46  20.2     1     3     1
##  7 0      14.3     8  360    245  3.21  3.57  15.8     0     3     4
##  8 0      24.4     4  147.    62  3.69  3.19  20       1     4     2
##  9 0      22.8     4  141.    95  3.92  3.15  22.9     1     4     2
## 10 0      19.2     6  168.   123  3.92  3.44  18.3     1     4     4
## # ... with 22 more rows
\end{verbatim}

Now for a more advanced example, let's suppose that we are dealing with the following wide dataset:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_wide }\OtherTok{\textless{}{-}}\NormalTok{ mtcars }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{pivot\_wider\_spec}\NormalTok{(}\AttributeTok{spec =}\NormalTok{ mtcars\_spec)}

\NormalTok{mtcars\_wide}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 32 x 14
##     disp    hp  drat    wt  qsec    vs  gear  carb `0_4` `0_6` `0_8` `1_4` `1_6`
##    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1  160    110  3.9   2.62  16.5     0     4     4  NA    NA    NA    NA      21
##  2  160    110  3.9   2.88  17.0     0     4     4  NA    NA    NA    NA      21
##  3  108     93  3.85  2.32  18.6     1     4     1  NA    NA    NA    22.8    NA
##  4  258    110  3.08  3.22  19.4     1     3     1  NA    21.4  NA    NA      NA
##  5  360    175  3.15  3.44  17.0     0     3     2  NA    NA    18.7  NA      NA
##  6  225    105  2.76  3.46  20.2     1     3     1  NA    18.1  NA    NA      NA
##  7  360    245  3.21  3.57  15.8     0     3     4  NA    NA    14.3  NA      NA
##  8  147.    62  3.69  3.19  20       1     4     2  24.4  NA    NA    NA      NA
##  9  141.    95  3.92  3.15  22.9     1     4     2  22.8  NA    NA    NA      NA
## 10  168.   123  3.92  3.44  18.3     1     4     4  NA    19.2  NA    NA      NA
## # ... with 22 more rows, and 1 more variable: `1_8` <dbl>
\end{verbatim}

The difficulty here is that we have columns with two levels of information. For instance, the
column ``0\_4'' contains the miles per gallon values for manual cars (\texttt{0}) with \texttt{4} cylinders.
The first step is to first pivot the columns:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_wide }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_longer}\NormalTok{(}\AttributeTok{cols =} \FunctionTok{matches}\NormalTok{(}\StringTok{"0|1"}\NormalTok{),}
               \AttributeTok{names\_to =} \StringTok{"am\_cyl"}\NormalTok{,}
               \AttributeTok{values\_to =} \StringTok{"mpg"}\NormalTok{,}
               \AttributeTok{values\_drop\_na =} \ConstantTok{TRUE}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(am\_cyl, mpg, }\FunctionTok{everything}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 32 x 10
##    am_cyl   mpg  disp    hp  drat    wt  qsec    vs  gear  carb
##    <chr>  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1 1_6     21    160    110  3.9   2.62  16.5     0     4     4
##  2 1_6     21    160    110  3.9   2.88  17.0     0     4     4
##  3 1_4     22.8  108     93  3.85  2.32  18.6     1     4     1
##  4 0_6     21.4  258    110  3.08  3.22  19.4     1     3     1
##  5 0_8     18.7  360    175  3.15  3.44  17.0     0     3     2
##  6 0_6     18.1  225    105  2.76  3.46  20.2     1     3     1
##  7 0_8     14.3  360    245  3.21  3.57  15.8     0     3     4
##  8 0_4     24.4  147.    62  3.69  3.19  20       1     4     2
##  9 0_4     22.8  141.    95  3.92  3.15  22.9     1     4     2
## 10 0_6     19.2  168.   123  3.92  3.44  18.3     1     4     4
## # ... with 22 more rows
\end{verbatim}

Now we only need to separate the ``am\_cyl'' column into two new columns, ``am'' and ``cyl'':

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_wide }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pivot\_longer}\NormalTok{(}\AttributeTok{cols =} \FunctionTok{matches}\NormalTok{(}\StringTok{"0|1"}\NormalTok{),}
               \AttributeTok{names\_to =} \StringTok{"am\_cyl"}\NormalTok{,}
               \AttributeTok{values\_to =} \StringTok{"mpg"}\NormalTok{,}
               \AttributeTok{values\_drop\_na =} \ConstantTok{TRUE}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{separate}\NormalTok{(am\_cyl, }\AttributeTok{into =} \FunctionTok{c}\NormalTok{(}\StringTok{"am"}\NormalTok{, }\StringTok{"cyl"}\NormalTok{), }\AttributeTok{sep =} \StringTok{"\_"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(am, cyl, mpg, }\FunctionTok{everything}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 32 x 11
##    am    cyl     mpg  disp    hp  drat    wt  qsec    vs  gear  carb
##    <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1 1     6      21    160    110  3.9   2.62  16.5     0     4     4
##  2 1     6      21    160    110  3.9   2.88  17.0     0     4     4
##  3 1     4      22.8  108     93  3.85  2.32  18.6     1     4     1
##  4 0     6      21.4  258    110  3.08  3.22  19.4     1     3     1
##  5 0     8      18.7  360    175  3.15  3.44  17.0     0     3     2
##  6 0     6      18.1  225    105  2.76  3.46  20.2     1     3     1
##  7 0     8      14.3  360    245  3.21  3.57  15.8     0     3     4
##  8 0     4      24.4  147.    62  3.69  3.19  20       1     4     2
##  9 0     4      22.8  141.    95  3.92  3.15  22.9     1     4     2
## 10 0     6      19.2  168.   123  3.92  3.44  18.3     1     4     4
## # ... with 22 more rows
\end{verbatim}

It is also possible to construct a specification data frame, just like for \texttt{pivot\_wider\_spec()}.
This time, I'm using the \texttt{build\_longer\_spec()} function that makes it easy to build specifications:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_spec\_long }\OtherTok{\textless{}{-}}\NormalTok{ mtcars\_wide }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{build\_longer\_spec}\NormalTok{(}\FunctionTok{matches}\NormalTok{(}\StringTok{"0|1"}\NormalTok{),}
                    \AttributeTok{values\_to =} \StringTok{"mpg"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{separate}\NormalTok{(name, }\FunctionTok{c}\NormalTok{(}\StringTok{"am"}\NormalTok{, }\StringTok{"cyl"}\NormalTok{), }\AttributeTok{sep =} \StringTok{"\_"}\NormalTok{)}

\NormalTok{mtcars\_spec\_long}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 4
##   .name .value am    cyl  
##   <chr> <chr>  <chr> <chr>
## 1 0_4   mpg    0     4    
## 2 0_6   mpg    0     6    
## 3 0_8   mpg    0     8    
## 4 1_4   mpg    1     4    
## 5 1_6   mpg    1     6    
## 6 1_8   mpg    1     8
\end{verbatim}

This spec can now be specified to \texttt{pivot\_longer()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars\_wide }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{pivot\_longer\_spec}\NormalTok{(}\AttributeTok{spec =}\NormalTok{ mtcars\_spec\_long,}
                    \AttributeTok{values\_drop\_na =} \ConstantTok{TRUE}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(am, cyl, mpg, }\FunctionTok{everything}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 32 x 11
##    am    cyl     mpg  disp    hp  drat    wt  qsec    vs  gear  carb
##    <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1 1     6      21    160    110  3.9   2.62  16.5     0     4     4
##  2 1     6      21    160    110  3.9   2.88  17.0     0     4     4
##  3 1     4      22.8  108     93  3.85  2.32  18.6     1     4     1
##  4 0     6      21.4  258    110  3.08  3.22  19.4     1     3     1
##  5 0     8      18.7  360    175  3.15  3.44  17.0     0     3     2
##  6 0     6      18.1  225    105  2.76  3.46  20.2     1     3     1
##  7 0     8      14.3  360    245  3.21  3.57  15.8     0     3     4
##  8 0     4      24.4  147.    62  3.69  3.19  20       1     4     2
##  9 0     4      22.8  141.    95  3.92  3.15  22.9     1     4     2
## 10 0     6      19.2  168.   123  3.92  3.44  18.3     1     4     4
## # ... with 22 more rows
\end{verbatim}

Defining specifications give a lot of flexibility and in some complicated cases are the way to go.

\hypertarget{fill-and-full_seq}{%
\subsubsection{\texorpdfstring{\texttt{fill()} and \texttt{full\_seq()}}{fill() and full\_seq()}}\label{fill-and-full_seq}}

\texttt{fill()} is pretty useful to\ldots{} fill in missing values. For instance, in \texttt{survey\_data}, some ``id''s
are missing:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 3
##       id variable value
##    <dbl> <chr>    <dbl>
##  1     1 var1       1  
##  2     1 var2       0.2
##  3    NA var3       0.3
##  4     2 var1       1.4
##  5     2 var2       1.9
##  6     2 var3       4.1
##  7     3 var1       0.1
##  8     3 var2       2.8
##  9     3 var3       8.9
## 10     4 var1       1.7
## 11    NA var2       1.9
## 12     4 var3       7.6
\end{verbatim}

It seems pretty obvious that the first \texttt{NA} is supposed to be \texttt{1} and the second missing is supposed
to be \texttt{4}. With \texttt{fill()}, this is pretty easy to achieve:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{fill}\NormalTok{(}\AttributeTok{.direction =} \StringTok{"down"}\NormalTok{, id)}
\end{Highlighting}
\end{Shaded}

\texttt{full\_seq()} is similar:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{full\_seq}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\FunctionTok{as.Date}\NormalTok{(}\StringTok{"2018{-}08{-}01"}\NormalTok{), }\FunctionTok{as.Date}\NormalTok{(}\StringTok{"2018{-}08{-}03"}\NormalTok{)), }\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "2018-08-01" "2018-08-02" "2018-08-03"
\end{verbatim}

We can add this as the date column to our survey data:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{date =} \FunctionTok{rep}\NormalTok{(}\FunctionTok{full\_seq}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\FunctionTok{as.Date}\NormalTok{(}\StringTok{"2018{-}08{-}01"}\NormalTok{), }\FunctionTok{as.Date}\NormalTok{(}\StringTok{"2018{-}08{-}03"}\NormalTok{)), }\DecValTok{1}\NormalTok{), }\DecValTok{4}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 4
##       id variable value date      
##    <dbl> <chr>    <dbl> <date>    
##  1     1 var1       1   2018-08-01
##  2     1 var2       0.2 2018-08-02
##  3    NA var3       0.3 2018-08-03
##  4     2 var1       1.4 2018-08-01
##  5     2 var2       1.9 2018-08-02
##  6     2 var3       4.1 2018-08-03
##  7     3 var1       0.1 2018-08-01
##  8     3 var2       2.8 2018-08-02
##  9     3 var3       8.9 2018-08-03
## 10     4 var1       1.7 2018-08-01
## 11    NA var2       1.9 2018-08-02
## 12     4 var3       7.6 2018-08-03
\end{verbatim}

I use the base \texttt{rep()} function to repeat the date 4 times and then using \texttt{mutate()} I have added
it the data frame.

Putting all these operations together:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{fill}\NormalTok{(}\AttributeTok{.direction =} \StringTok{"down"}\NormalTok{, id) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{date =} \FunctionTok{rep}\NormalTok{(}\FunctionTok{full\_seq}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\FunctionTok{as.Date}\NormalTok{(}\StringTok{"2018{-}08{-}01"}\NormalTok{), }\FunctionTok{as.Date}\NormalTok{(}\StringTok{"2018{-}08{-}03"}\NormalTok{)), }\DecValTok{1}\NormalTok{), }\DecValTok{4}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 4
##       id variable value date      
##    <dbl> <chr>    <dbl> <date>    
##  1     1 var1       1   2018-08-01
##  2     1 var2       0.2 2018-08-02
##  3     1 var3       0.3 2018-08-03
##  4     2 var1       1.4 2018-08-01
##  5     2 var2       1.9 2018-08-02
##  6     2 var3       4.1 2018-08-03
##  7     3 var1       0.1 2018-08-01
##  8     3 var2       2.8 2018-08-02
##  9     3 var3       8.9 2018-08-03
## 10     4 var1       1.7 2018-08-01
## 11     4 var2       1.9 2018-08-02
## 12     4 var3       7.6 2018-08-03
\end{verbatim}

You should be careful when imputing missing values though. The method described above is called
\texttt{Last\ Observation\ Carried\ Forward}, and sometimes it makes sense, like here, but sometimes it doesn't and
doing this will introduce bias in your analysis. Discussing how to handle missing values in your analysis
is outside of the scope of this book, but there are many resources available. You may want to check
out the vignettes of the \texttt{\{mice\}} \href{https://amices.org/mice/articles/overview.html}{package}, which
lists many resources to get you started.

\hypertarget{put-order-in-your-columns-with-separate-unite-and-in-your-rows-with-separate_rows}{%
\subsubsection{\texorpdfstring{Put order in your columns with \texttt{separate()}, \texttt{unite()}, and in your rows with \texttt{separate\_rows()}}{Put order in your columns with separate(), unite(), and in your rows with separate\_rows()}}\label{put-order-in-your-columns-with-separate-unite-and-in-your-rows-with-separate_rows}}

Sometimes, data can be in a format that makes working with it needlessly painful. For example, you
get this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_not\_tidy}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 3
##       id variable_date   value
##    <dbl> <chr>           <dbl>
##  1     1 var1/2018-08-01   1  
##  2     1 var2/2018-08-02   0.2
##  3     1 var3/2018-08-03   0.3
##  4     2 var1/2018-08-01   1.4
##  5     2 var2/2018-08-02   1.9
##  6     2 var3/2018-08-03   4.1
##  7     3 var1/2018-08-01   0.1
##  8     3 var2/2018-08-02   2.8
##  9     3 var3/2018-08-03   8.9
## 10     4 var1/2018-08-01   1.7
## 11     4 var2/2018-08-02   1.9
## 12     4 var3/2018-08-03   7.6
\end{verbatim}

Dealing with this is simple, thanks to \texttt{separate()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_not\_tidy }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{separate}\NormalTok{(variable\_date, }\AttributeTok{into =} \FunctionTok{c}\NormalTok{(}\StringTok{"variable"}\NormalTok{, }\StringTok{"date"}\NormalTok{), }\AttributeTok{sep =} \StringTok{"/"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 4
##       id variable date       value
##    <dbl> <chr>    <chr>      <dbl>
##  1     1 var1     2018-08-01   1  
##  2     1 var2     2018-08-02   0.2
##  3     1 var3     2018-08-03   0.3
##  4     2 var1     2018-08-01   1.4
##  5     2 var2     2018-08-02   1.9
##  6     2 var3     2018-08-03   4.1
##  7     3 var1     2018-08-01   0.1
##  8     3 var2     2018-08-02   2.8
##  9     3 var3     2018-08-03   8.9
## 10     4 var1     2018-08-01   1.7
## 11     4 var2     2018-08-02   1.9
## 12     4 var3     2018-08-03   7.6
\end{verbatim}

The \texttt{variable\_date} column gets separated into two columns, \texttt{variable} and \texttt{date}. One also needs
to specify the separator, in this case ``/''.

\texttt{unite()} is the reverse operation, which can be useful when you are confronted to this situation:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data2}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 6
##       id variable year  month day   value
##    <dbl> <chr>    <chr> <chr> <chr> <dbl>
##  1     1 var1     2018  08    01      1  
##  2     1 var2     2018  08    02      0.2
##  3     1 var3     2018  08    03      0.3
##  4     2 var1     2018  08    01      1.4
##  5     2 var2     2018  08    02      1.9
##  6     2 var3     2018  08    03      4.1
##  7     3 var1     2018  08    01      0.1
##  8     3 var2     2018  08    02      2.8
##  9     3 var3     2018  08    03      8.9
## 10     4 var1     2018  08    01      1.7
## 11     4 var2     2018  08    02      1.9
## 12     4 var3     2018  08    03      7.6
\end{verbatim}

In some situation, it is better to have the date as a single column:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data2 }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{unite}\NormalTok{(date, year, month, day, }\AttributeTok{sep =} \StringTok{"{-}"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 4
##       id variable date       value
##    <dbl> <chr>    <chr>      <dbl>
##  1     1 var1     2018-08-01   1  
##  2     1 var2     2018-08-02   0.2
##  3     1 var3     2018-08-03   0.3
##  4     2 var1     2018-08-01   1.4
##  5     2 var2     2018-08-02   1.9
##  6     2 var3     2018-08-03   4.1
##  7     3 var1     2018-08-01   0.1
##  8     3 var2     2018-08-02   2.8
##  9     3 var3     2018-08-03   8.9
## 10     4 var1     2018-08-01   1.7
## 11     4 var2     2018-08-02   1.9
## 12     4 var3     2018-08-03   7.6
\end{verbatim}

Another awful situation is the following:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_from\_hell}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   id         variable         value
## 1  1             var1             1
## 2  1             var2           0.2
## 3 NA             var3           0.3
## 4  2 var1, var2, var3 1.4, 1.9, 4.1
## 5  3       var1, var2      0.1, 2.8
## 6  3             var3           8.9
## 7  4             var1           1.7
## 8 NA             var2           1.9
## 9  4             var3           7.6
\end{verbatim}

\texttt{separate\_rows()} saves the day:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_from\_hell }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{separate\_rows}\NormalTok{(variable, value)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 3
##       id variable value
##    <dbl> <chr>    <chr>
##  1     1 var1     1    
##  2     1 var2     0.2  
##  3    NA var3     0.3  
##  4     2 var1     1.4  
##  5     2 var2     1.9  
##  6     2 var3     4.1  
##  7     3 var1     0.1  
##  8     3 var2     2.8  
##  9     3 var3     8.9  
## 10     4 var1     1.7  
## 11    NA var2     1.9  
## 12     4 var3     7.6
\end{verbatim}

So to summarise\ldots{} you can go from this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_from\_hell}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   id         variable         value
## 1  1             var1             1
## 2  1             var2           0.2
## 3 NA             var3           0.3
## 4  2 var1, var2, var3 1.4, 1.9, 4.1
## 5  3       var1, var2      0.1, 2.8
## 6  3             var3           8.9
## 7  4             var1           1.7
## 8 NA             var2           1.9
## 9  4             var3           7.6
\end{verbatim}

to this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_clean}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 4
##       id variable date       value
##    <dbl> <chr>    <chr>      <dbl>
##  1     1 var1     2018-08-01   1  
##  2     1 var2     2018-08-02   0.2
##  3     1 var3     2018-08-03   0.3
##  4     2 var1     2018-08-01   1.4
##  5     2 var2     2018-08-02   1.9
##  6     2 var3     2018-08-03   4.1
##  7     3 var1     2018-08-01   0.1
##  8     3 var2     2018-08-02   2.8
##  9     3 var3     2018-08-03   8.9
## 10     4 var1     2018-08-01   1.7
## 11     4 var2     2018-08-02   1.9
## 12     4 var3     2018-08-03   7.6
\end{verbatim}

quite easily:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data\_from\_hell }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{separate\_rows}\NormalTok{(variable, value, }\AttributeTok{convert =} \ConstantTok{TRUE}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{fill}\NormalTok{(}\AttributeTok{.direction =} \StringTok{"down"}\NormalTok{, id) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{date =} \FunctionTok{rep}\NormalTok{(}\FunctionTok{full\_seq}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\FunctionTok{as.Date}\NormalTok{(}\StringTok{"2018{-}08{-}01"}\NormalTok{), }\FunctionTok{as.Date}\NormalTok{(}\StringTok{"2018{-}08{-}03"}\NormalTok{)), }\DecValTok{1}\NormalTok{), }\DecValTok{4}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\hypertarget{working-on-many-columns-with-if_any-if_all-and-across}{%
\subsection{\texorpdfstring{Working on many columns with \texttt{if\_any()}, \texttt{if\_all()} and \texttt{across()}}{Working on many columns with if\_any(), if\_all() and across()}}\label{working-on-many-columns-with-if_any-if_all-and-across}}

\hypertarget{filtering-rows-where-several-columns-verify-a-condition}{%
\subsubsection{Filtering rows where several columns verify a condition}\label{filtering-rows-where-several-columns-verify-a-condition}}

Let's go back to the \texttt{gasoline} data from the \texttt{\{Ecdat\}} package.

When using \texttt{filter()}, it is only possible to filter one column at a time. For example, you can
only filter rows where a column equals ``France'' for instance. But suppose that we have a condition that we want
to use to filter out a lot of columns at once. For example, for every column that is of type
\texttt{numeric}, keep only the lines where the condition \emph{value \textgreater{} -8} is satisfied. The next line does
that:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{if\_any}\NormalTok{(}\FunctionTok{where}\NormalTok{(is.numeric), \textbackslash{}(x)(}\StringTok{\textasciigrave{}}\AttributeTok{\textgreater{}}\StringTok{\textasciigrave{}}\NormalTok{(x, }\SpecialCharTok{{-}}\DecValTok{8}\NormalTok{))))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 6
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77
##  2 austria  1961     4.10    -6.43 -0.351    -9.61
##  3 austria  1962     4.07    -6.41 -0.380    -9.46
##  4 austria  1963     4.06    -6.37 -0.414    -9.34
##  5 austria  1964     4.04    -6.32 -0.445    -9.24
##  6 austria  1965     4.03    -6.29 -0.497    -9.12
##  7 austria  1966     4.05    -6.25 -0.467    -9.02
##  8 austria  1967     4.05    -6.23 -0.506    -8.93
##  9 austria  1968     4.05    -6.21 -0.522    -8.85
## 10 austria  1969     4.05    -6.15 -0.559    -8.79
## # ... with 332 more rows
\end{verbatim}

The above code is using the \texttt{if\_any()} function, included in \texttt{\{dplyr\}}. It also uses
\texttt{where()}, which must be used for predicate functions like \texttt{is.numeric()}, or \texttt{is.character()}, etc.
You can think of \texttt{if\_any()} as a function that helps you select the columns to which to apply the
function. You can read the code above like this:

\emph{Start with the gasoline data, then filter rows that are greater than -8 across the columns
which are numeric}

or similar. \texttt{if\_any()}, \texttt{if\_all()} and \texttt{across()} makes operations like these very easy to achieve.

Sometimes, you'd want to filter rows from columns that end their labels with a letter, for instance
\texttt{"p"}. This can again be achieved using another helper, \texttt{ends\_with()}, instead of \texttt{where()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{if\_any}\NormalTok{(}\FunctionTok{ends\_with}\NormalTok{(}\StringTok{"p"}\NormalTok{), \textbackslash{}(x)(}\StringTok{\textasciigrave{}}\AttributeTok{\textgreater{}}\StringTok{\textasciigrave{}}\NormalTok{(x, }\SpecialCharTok{{-}}\DecValTok{8}\NormalTok{))))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 340 x 6
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77
##  2 austria  1961     4.10    -6.43 -0.351    -9.61
##  3 austria  1962     4.07    -6.41 -0.380    -9.46
##  4 austria  1963     4.06    -6.37 -0.414    -9.34
##  5 austria  1964     4.04    -6.32 -0.445    -9.24
##  6 austria  1965     4.03    -6.29 -0.497    -9.12
##  7 austria  1966     4.05    -6.25 -0.467    -9.02
##  8 austria  1967     4.05    -6.23 -0.506    -8.93
##  9 austria  1968     4.05    -6.21 -0.522    -8.85
## 10 austria  1969     4.05    -6.15 -0.559    -8.79
## # ... with 330 more rows
\end{verbatim}

We already know about \texttt{ends\_with()} and \texttt{starts\_with()}. So the above line means ``for the columns
whose name end with a `p' only keep the lines where, for all the selected columns, the values are
strictly superior to \texttt{-8}''.

\texttt{if\_all()} works exactly the same way, but think of the \texttt{if} in \texttt{if\_all()} as having the conditions
separated by \texttt{and} while the \texttt{if} of \texttt{if\_any()} being separated by \texttt{or}. So for example, the
code above, where \texttt{if\_any()} is replaced by \texttt{if\_all()}, results in a much smaller data frame:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{if\_all}\NormalTok{(}\FunctionTok{ends\_with}\NormalTok{(}\StringTok{"p"}\NormalTok{), \textbackslash{}(x)(}\StringTok{\textasciigrave{}}\AttributeTok{\textgreater{}}\StringTok{\textasciigrave{}}\NormalTok{(x, }\SpecialCharTok{{-}}\DecValTok{8}\NormalTok{))))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 30 x 6
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 canada   1972     4.89    -5.44 -1.10     -7.99
##  2 canada   1973     4.90    -5.41 -1.13     -7.94
##  3 canada   1974     4.89    -5.42 -1.12     -7.90
##  4 canada   1975     4.89    -5.38 -1.19     -7.87
##  5 canada   1976     4.84    -5.36 -1.06     -7.81
##  6 canada   1977     4.81    -5.34 -1.07     -7.77
##  7 canada   1978     4.86    -5.31 -1.07     -7.79
##  8 germany  1978     3.88    -5.56 -0.628    -7.95
##  9 sweden   1975     3.97    -7.68 -2.77     -7.99
## 10 sweden   1976     3.98    -7.67 -2.82     -7.96
## # ... with 20 more rows
\end{verbatim}

because here, we only keep rows for columns that end with ``p'' where ALL of them are simultaneously
greater than 8.

\hypertarget{selecting-several-columns-at-once}{%
\subsubsection{Selecting several columns at once}\label{selecting-several-columns-at-once}}

In a previous section we already played around a little bit with \texttt{select()} and some helpers,
\texttt{everything()}, \texttt{starts\_with()} and \texttt{ends\_with()}. But there are many ways that you can use
helper functions to select several columns easily:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(}\FunctionTok{where}\NormalTok{(is.numeric))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 5
##     year lgaspcar lincomep  lrpmg lcarpcap
##    <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1  1960     4.17    -6.47 -0.335    -9.77
##  2  1961     4.10    -6.43 -0.351    -9.61
##  3  1962     4.07    -6.41 -0.380    -9.46
##  4  1963     4.06    -6.37 -0.414    -9.34
##  5  1964     4.04    -6.32 -0.445    -9.24
##  6  1965     4.03    -6.29 -0.497    -9.12
##  7  1966     4.05    -6.25 -0.467    -9.02
##  8  1967     4.05    -6.23 -0.506    -8.93
##  9  1968     4.05    -6.21 -0.522    -8.85
## 10  1969     4.05    -6.15 -0.559    -8.79
## # ... with 332 more rows
\end{verbatim}

Selecting by column position is also possible:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{5}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 3
##    country  year  lrpmg
##    <chr>   <int>  <dbl>
##  1 austria  1960 -0.335
##  2 austria  1961 -0.351
##  3 austria  1962 -0.380
##  4 austria  1963 -0.414
##  5 austria  1964 -0.445
##  6 austria  1965 -0.497
##  7 austria  1966 -0.467
##  8 austria  1967 -0.506
##  9 austria  1968 -0.522
## 10 austria  1969 -0.559
## # ... with 332 more rows
\end{verbatim}

As is selecting columns starting or ending with a certain string of characters, as discussed previously:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(}\FunctionTok{starts\_with}\NormalTok{(}\StringTok{"l"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 4
##    lgaspcar lincomep  lrpmg lcarpcap
##       <dbl>    <dbl>  <dbl>    <dbl>
##  1     4.17    -6.47 -0.335    -9.77
##  2     4.10    -6.43 -0.351    -9.61
##  3     4.07    -6.41 -0.380    -9.46
##  4     4.06    -6.37 -0.414    -9.34
##  5     4.04    -6.32 -0.445    -9.24
##  6     4.03    -6.29 -0.497    -9.12
##  7     4.05    -6.25 -0.467    -9.02
##  8     4.05    -6.23 -0.506    -8.93
##  9     4.05    -6.21 -0.522    -8.85
## 10     4.05    -6.15 -0.559    -8.79
## # ... with 332 more rows
\end{verbatim}

Another very neat trick is selecting columns that may or may not exist in your data frame. For this quick examples
let's use the \texttt{mtcars} dataset:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sort}\NormalTok{(}\FunctionTok{colnames}\NormalTok{(mtcars))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] "am"   "carb" "cyl"  "disp" "drat" "gear" "hp"   "mpg"  "qsec" "vs"  
## [11] "wt"
\end{verbatim}

Let's create a vector with some column names:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{cols\_to\_select }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\StringTok{"mpg"}\NormalTok{, }\StringTok{"cyl"}\NormalTok{, }\StringTok{"am"}\NormalTok{, }\StringTok{"nonsense"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

The following selects the columns that exist
in the data frame but shows a warning for the column that does not exist:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(}\FunctionTok{any\_of}\NormalTok{(cols\_to\_select))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                      mpg cyl am
## Mazda RX4           21.0   6  1
## Mazda RX4 Wag       21.0   6  1
## Datsun 710          22.8   4  1
## Hornet 4 Drive      21.4   6  0
## Hornet Sportabout   18.7   8  0
## Valiant             18.1   6  0
## Duster 360          14.3   8  0
## Merc 240D           24.4   4  0
## Merc 230            22.8   4  0
## Merc 280            19.2   6  0
## Merc 280C           17.8   6  0
## Merc 450SE          16.4   8  0
## Merc 450SL          17.3   8  0
## Merc 450SLC         15.2   8  0
## Cadillac Fleetwood  10.4   8  0
## Lincoln Continental 10.4   8  0
## Chrysler Imperial   14.7   8  0
## Fiat 128            32.4   4  1
## Honda Civic         30.4   4  1
## Toyota Corolla      33.9   4  1
## Toyota Corona       21.5   4  0
## Dodge Challenger    15.5   8  0
## AMC Javelin         15.2   8  0
## Camaro Z28          13.3   8  0
## Pontiac Firebird    19.2   8  0
## Fiat X1-9           27.3   4  1
## Porsche 914-2       26.0   4  1
## Lotus Europa        30.4   4  1
## Ford Pantera L      15.8   8  1
## Ferrari Dino        19.7   6  1
## Maserati Bora       15.0   8  1
## Volvo 142E          21.4   4  1
\end{verbatim}

and finally, if you want it to fail, don't use any helper:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(cols\_to\_select)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
Error: Can't subset columns that don't exist.
The column `nonsense` doesn't exist.
\end{verbatim}

or use \texttt{all\_of()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(}\FunctionTok{all\_of}\NormalTok{(cols\_to\_select))}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{✖ Column }\StringTok{\textasciigrave{}}\AttributeTok{nonsense}\StringTok{\textasciigrave{}}\NormalTok{ doesn}\StringTok{\textquotesingle{}t exist.}
\end{Highlighting}
\end{Shaded}

Bulk-renaming can be achieved using \texttt{rename\_with()}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{rename\_with}\NormalTok{(toupper, is.numeric)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 6
##    country  YEAR LGASPCAR LINCOMEP  LRPMG LCARPCAP
##    <chr>   <int>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77
##  2 austria  1961     4.10    -6.43 -0.351    -9.61
##  3 austria  1962     4.07    -6.41 -0.380    -9.46
##  4 austria  1963     4.06    -6.37 -0.414    -9.34
##  5 austria  1964     4.04    -6.32 -0.445    -9.24
##  6 austria  1965     4.03    -6.29 -0.497    -9.12
##  7 austria  1966     4.05    -6.25 -0.467    -9.02
##  8 austria  1967     4.05    -6.23 -0.506    -8.93
##  9 austria  1968     4.05    -6.21 -0.522    -8.85
## 10 austria  1969     4.05    -6.15 -0.559    -8.79
## # ... with 332 more rows
\end{verbatim}

you can also pass functions to \texttt{rename\_with()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{rename\_with}\NormalTok{(\textbackslash{}(x)(}\FunctionTok{paste0}\NormalTok{(}\StringTok{"new\_"}\NormalTok{, x)))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 6
##    new_country new_year new_lgaspcar new_lincomep new_lrpmg new_lcarpcap
##    <chr>          <int>        <dbl>        <dbl>     <dbl>        <dbl>
##  1 austria         1960         4.17        -6.47    -0.335        -9.77
##  2 austria         1961         4.10        -6.43    -0.351        -9.61
##  3 austria         1962         4.07        -6.41    -0.380        -9.46
##  4 austria         1963         4.06        -6.37    -0.414        -9.34
##  5 austria         1964         4.04        -6.32    -0.445        -9.24
##  6 austria         1965         4.03        -6.29    -0.497        -9.12
##  7 austria         1966         4.05        -6.25    -0.467        -9.02
##  8 austria         1967         4.05        -6.23    -0.506        -8.93
##  9 austria         1968         4.05        -6.21    -0.522        -8.85
## 10 austria         1969         4.05        -6.15    -0.559        -8.79
## # ... with 332 more rows
\end{verbatim}

The reason I'm talking about renaming in a section about selecting is because you can
also rename with select:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(}\AttributeTok{YEAR =}\NormalTok{ year)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 1
##     YEAR
##    <int>
##  1  1960
##  2  1961
##  3  1962
##  4  1963
##  5  1964
##  6  1965
##  7  1966
##  8  1967
##  9  1968
## 10  1969
## # ... with 332 more rows
\end{verbatim}

but of course here, you only keep that one column, and you can't rename with a function.

\hypertarget{summarising-with-across}{%
\subsubsection{\texorpdfstring{Summarising with \texttt{across()}}{Summarising with across()}}\label{summarising-with-across}}

\texttt{across()} is used for summarising data. It allows to aggregations\ldots{} \emph{across} several columns. It
is especially useful with \texttt{group\_by()}. To illustrate how \texttt{group\_by()} works with \texttt{across()} I have
to first modify the \texttt{gasoline} data a little bit. As you can see below, the \texttt{year} column is of
type \texttt{double}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{lapply}\NormalTok{(typeof)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## $country
## [1] "character"
## 
## $year
## [1] "integer"
## 
## $lgaspcar
## [1] "double"
## 
## $lincomep
## [1] "double"
## 
## $lrpmg
## [1] "double"
## 
## $lcarpcap
## [1] "double"
\end{verbatim}

(we'll discuss \texttt{lapply()} in a later chapter, but just to give you a little taste, \texttt{lapply()} applies
a function to each element of a list or of a data frame, in this case, \texttt{lapply()} applied the \texttt{typeof()}
function to each column of the \texttt{gasoline} data set, returning the type of each column)

Let's change that to character:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{year =} \FunctionTok{as.character}\NormalTok{(year),}
         \AttributeTok{country =} \FunctionTok{as.character}\NormalTok{(country))}
\end{Highlighting}
\end{Shaded}

This now allows me to group by type of columns for instance:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{where}\NormalTok{(is.character))) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_lincomep =} \FunctionTok{mean}\NormalTok{(lincomep))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `summarise()` has grouped output by 'country'. You can override using the
## `.groups` argument.
\end{verbatim}

\begin{verbatim}
## # A tibble: 342 x 3
## # Groups:   country [18]
##    country year  mean_lincomep
##    <chr>   <chr>         <dbl>
##  1 austria 1960          -6.47
##  2 austria 1961          -6.43
##  3 austria 1962          -6.41
##  4 austria 1963          -6.37
##  5 austria 1964          -6.32
##  6 austria 1965          -6.29
##  7 austria 1966          -6.25
##  8 austria 1967          -6.23
##  9 austria 1968          -6.21
## 10 austria 1969          -6.15
## # ... with 332 more rows
\end{verbatim}

This is faster than having to write:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(country, year) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_lincomep =} \FunctionTok{mean}\NormalTok{(lincomep))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `summarise()` has grouped output by 'country'. You can override using the
## `.groups` argument.
\end{verbatim}

\begin{verbatim}
## # A tibble: 342 x 3
## # Groups:   country [18]
##    country year  mean_lincomep
##    <chr>   <chr>         <dbl>
##  1 austria 1960          -6.47
##  2 austria 1961          -6.43
##  3 austria 1962          -6.41
##  4 austria 1963          -6.37
##  5 austria 1964          -6.32
##  6 austria 1965          -6.29
##  7 austria 1966          -6.25
##  8 austria 1967          -6.23
##  9 austria 1968          -6.21
## 10 austria 1969          -6.15
## # ... with 332 more rows
\end{verbatim}

You may think that having two write the name of two variables is not a huge deal, which is true.
But imagine that you have dozens of character columns that you want to group by.
With \texttt{across()} and the helper functions, it doesn't matter if the data frame has 2 columns
you need to group by or 2000. All that matters is that you can find some commonalities between
all these columns that make it easy to select them. It can be their type, as we have seen
before, or their label:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{contains}\NormalTok{(}\StringTok{"y"}\NormalTok{))) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_licomep =} \FunctionTok{mean}\NormalTok{(lincomep))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `summarise()` has grouped output by 'country'. You can override using the
## `.groups` argument.
\end{verbatim}

\begin{verbatim}
## # A tibble: 342 x 3
## # Groups:   country [18]
##    country year  mean_licomep
##    <chr>   <chr>        <dbl>
##  1 austria 1960         -6.47
##  2 austria 1961         -6.43
##  3 austria 1962         -6.41
##  4 austria 1963         -6.37
##  5 austria 1964         -6.32
##  6 austria 1965         -6.29
##  7 austria 1966         -6.25
##  8 austria 1967         -6.23
##  9 austria 1968         -6.21
## 10 austria 1969         -6.15
## # ... with 332 more rows
\end{verbatim}

but it's also possible to \texttt{group\_by()} position:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{))) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_licomep =} \FunctionTok{mean}\NormalTok{(lincomep))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `summarise()` has grouped output by 'country'. You can override using the
## `.groups` argument.
\end{verbatim}

\begin{verbatim}
## # A tibble: 342 x 3
## # Groups:   country [18]
##    country year  mean_licomep
##    <chr>   <chr>        <dbl>
##  1 austria 1960         -6.47
##  2 austria 1961         -6.43
##  3 austria 1962         -6.41
##  4 austria 1963         -6.37
##  5 austria 1964         -6.32
##  6 austria 1965         -6.29
##  7 austria 1966         -6.25
##  8 austria 1967         -6.23
##  9 austria 1968         -6.21
## 10 austria 1969         -6.15
## # ... with 332 more rows
\end{verbatim}

Using a sequence is also possible:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{seq}\NormalTok{(}\DecValTok{1}\SpecialCharTok{:}\DecValTok{2}\NormalTok{))) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_lincomep =} \FunctionTok{mean}\NormalTok{(lincomep))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `summarise()` has grouped output by 'country'. You can override using the
## `.groups` argument.
\end{verbatim}

\begin{verbatim}
## # A tibble: 342 x 3
## # Groups:   country [18]
##    country year  mean_lincomep
##    <chr>   <chr>         <dbl>
##  1 austria 1960          -6.47
##  2 austria 1961          -6.43
##  3 austria 1962          -6.41
##  4 austria 1963          -6.37
##  5 austria 1964          -6.32
##  6 austria 1965          -6.29
##  7 austria 1966          -6.25
##  8 austria 1967          -6.23
##  9 austria 1968          -6.21
## 10 austria 1969          -6.15
## # ... with 332 more rows
\end{verbatim}

but be careful, selecting by position is dangerous. If the position of columns changes, your code
will fail. Selecting by type or label is much more robust, especially by label, since types can
change as well (for example a date column can easily be exported as character column, etc).

\hypertarget{summarise-across-many-columns}{%
\subsubsection{\texorpdfstring{\texttt{summarise()} across many columns}{summarise() across many columns}}\label{summarise-across-many-columns}}

Summarising across many columns is really incredibly useful and in my opinion one of the best
arguments in favour of switching to a \texttt{\{tidyverse\}} only workflow:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{starts\_with}\NormalTok{(}\StringTok{"l"}\NormalTok{), mean))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 5
##    country  lgaspcar lincomep   lrpmg lcarpcap
##    <chr>       <dbl>    <dbl>   <dbl>    <dbl>
##  1 austria      4.06    -6.12 -0.486     -8.85
##  2 belgium      3.92    -5.85 -0.326     -8.63
##  3 canada       4.86    -5.58 -1.05      -8.08
##  4 denmark      4.19    -5.76 -0.358     -8.58
##  5 france       3.82    -5.87 -0.253     -8.45
##  6 germany      3.89    -5.85 -0.517     -8.51
##  7 greece       4.88    -6.61 -0.0339   -10.8 
##  8 ireland      4.23    -6.44 -0.348     -9.04
##  9 italy        3.73    -6.35 -0.152     -8.83
## 10 japan        4.70    -6.25 -0.287     -9.95
## 11 netherla     4.08    -5.92 -0.370     -8.82
## 12 norway       4.11    -5.75 -0.278     -8.77
## 13 spain        4.06    -5.63  0.739     -9.90
## 14 sweden       4.01    -7.82 -2.71      -8.25
## 15 switzerl     4.24    -5.93 -0.902     -8.54
## 16 turkey       5.77    -7.34 -0.422    -12.5 
## 17 u.k.         3.98    -6.02 -0.459     -8.55
## 18 u.s.a.       4.82    -5.45 -1.21      -7.78
\end{verbatim}

But where \texttt{summarise()} and \texttt{across()} really shine is when you want to apply several functions
to many columns at once:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{starts\_with}\NormalTok{(}\StringTok{"l"}\NormalTok{), tibble}\SpecialCharTok{::}\FunctionTok{lst}\NormalTok{(mean, sd, max, min), }\AttributeTok{.names =} \StringTok{"\{fn\}\_\{col\}"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 17
##    country  mean_lgasp~1 sd_lg~2 max_l~3 min_l~4 mean_~5 sd_li~6 max_l~7 min_l~8
##    <chr>           <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
##  1 austria          4.06  0.0693    4.20    3.92   -6.12   0.235   -5.76   -6.47
##  2 belgium          3.92  0.103     4.16    3.82   -5.85   0.227   -5.53   -6.22
##  3 canada           4.86  0.0262    4.90    4.81   -5.58   0.193   -5.31   -5.89
##  4 denmark          4.19  0.158     4.50    4.00   -5.76   0.176   -5.48   -6.06
##  5 france           3.82  0.0499    3.91    3.75   -5.87   0.241   -5.53   -6.26
##  6 germany          3.89  0.0239    3.93    3.85   -5.85   0.193   -5.56   -6.16
##  7 greece           4.88  0.255     5.38    4.48   -6.61   0.331   -6.15   -7.16
##  8 ireland          4.23  0.0437    4.33    4.16   -6.44   0.162   -6.19   -6.72
##  9 italy            3.73  0.220     4.05    3.38   -6.35   0.217   -6.08   -6.73
## 10 japan            4.70  0.684     6.00    3.95   -6.25   0.425   -5.71   -6.99
## 11 netherla         4.08  0.286     4.65    3.71   -5.92   0.193   -5.66   -6.22
## 12 norway           4.11  0.123     4.44    3.96   -5.75   0.201   -5.42   -6.09
## 13 spain            4.06  0.317     4.75    3.62   -5.63   0.278   -5.29   -6.17
## 14 sweden           4.01  0.0364    4.07    3.91   -7.82   0.126   -7.67   -8.07
## 15 switzerl         4.24  0.102     4.44    4.05   -5.93   0.124   -5.75   -6.16
## 16 turkey           5.77  0.329     6.16    5.14   -7.34   0.331   -6.89   -7.84
## 17 u.k.             3.98  0.0479    4.10    3.91   -6.02   0.107   -5.84   -6.19
## 18 u.s.a.           4.82  0.0219    4.86    4.79   -5.45   0.148   -5.22   -5.70
## # ... with 8 more variables: mean_lrpmg <dbl>, sd_lrpmg <dbl>, max_lrpmg <dbl>,
## #   min_lrpmg <dbl>, mean_lcarpcap <dbl>, sd_lcarpcap <dbl>,
## #   max_lcarpcap <dbl>, min_lcarpcap <dbl>, and abbreviated variable names
## #   1: mean_lgaspcar, 2: sd_lgaspcar, 3: max_lgaspcar, 4: min_lgaspcar,
## #   5: mean_lincomep, 6: sd_lincomep, 7: max_lincomep, 8: min_lincomep
\end{verbatim}

Here, I first started by grouping by \texttt{country}, then I applied the \texttt{mean()}, \texttt{sd()}, \texttt{max()} and
\texttt{min()} functions to every column starting with the character \texttt{"l"}. \texttt{tibble::lst()} allows you to
create a list just like with \texttt{list()} but names its arguments automatically. So the \texttt{mean()} function
gets name \texttt{"mean"}, and so on. Finally, I use the \texttt{.names\ =} argument to create the template for
the new column names. \texttt{\{fn\}\_\{col\}} creates new column names of the form \emph{function name \_ column name}.

As mentioned before, \texttt{across()} works with other helper functions:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{contains}\NormalTok{(}\StringTok{"car"}\NormalTok{), tibble}\SpecialCharTok{::}\FunctionTok{lst}\NormalTok{(mean, sd, max, min), }\AttributeTok{.names =} \StringTok{"\{fn\}\_\{col\}"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 9
##    country  mean_lgasp~1 sd_lg~2 max_l~3 min_l~4 mean_~5 sd_lc~6 max_l~7 min_l~8
##    <chr>           <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
##  1 austria          4.06  0.0693    4.20    3.92   -8.85   0.473   -8.21   -9.77
##  2 belgium          3.92  0.103     4.16    3.82   -8.63   0.417   -8.10   -9.41
##  3 canada           4.86  0.0262    4.90    4.81   -8.08   0.195   -7.77   -8.38
##  4 denmark          4.19  0.158     4.50    4.00   -8.58   0.349   -8.20   -9.33
##  5 france           3.82  0.0499    3.91    3.75   -8.45   0.344   -8.01   -9.15
##  6 germany          3.89  0.0239    3.93    3.85   -8.51   0.406   -7.95   -9.34
##  7 greece           4.88  0.255     5.38    4.48  -10.8    0.839   -9.57  -12.2 
##  8 ireland          4.23  0.0437    4.33    4.16   -9.04   0.345   -8.55   -9.70
##  9 italy            3.73  0.220     4.05    3.38   -8.83   0.639   -8.11  -10.1 
## 10 japan            4.70  0.684     6.00    3.95   -9.95   1.20    -8.59  -12.2 
## 11 netherla         4.08  0.286     4.65    3.71   -8.82   0.617   -8.16  -10.0 
## 12 norway           4.11  0.123     4.44    3.96   -8.77   0.438   -8.17   -9.68
## 13 spain            4.06  0.317     4.75    3.62   -9.90   0.960   -8.63  -11.6 
## 14 sweden           4.01  0.0364    4.07    3.91   -8.25   0.242   -7.96   -8.74
## 15 switzerl         4.24  0.102     4.44    4.05   -8.54   0.378   -8.03   -9.26
## 16 turkey           5.77  0.329     6.16    5.14  -12.5    0.751  -11.2   -13.5 
## 17 u.k.             3.98  0.0479    4.10    3.91   -8.55   0.281   -8.26   -9.12
## 18 u.s.a.           4.82  0.0219    4.86    4.79   -7.78   0.162   -7.54   -8.02
## # ... with abbreviated variable names 1: mean_lgaspcar, 2: sd_lgaspcar,
## #   3: max_lgaspcar, 4: min_lgaspcar, 5: mean_lcarpcap, 6: sd_lcarpcap,
## #   7: max_lcarpcap, 8: min_lcarpcap
\end{verbatim}

This is very likely the quickest, most elegant way to summarise that many columns.

There's also a way to \emph{summarise where}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{where}\NormalTok{(is.numeric), tibble}\SpecialCharTok{::}\FunctionTok{lst}\NormalTok{(mean, sd, min, max), }\AttributeTok{.names =} \StringTok{"\{fn\}\_\{col\}"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 17
##    country  mean_lgasp~1 sd_lg~2 min_l~3 max_l~4 mean_~5 sd_li~6 min_l~7 max_l~8
##    <chr>           <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
##  1 austria          4.06  0.0693    3.92    4.20   -6.12   0.235   -6.47   -5.76
##  2 belgium          3.92  0.103     3.82    4.16   -5.85   0.227   -6.22   -5.53
##  3 canada           4.86  0.0262    4.81    4.90   -5.58   0.193   -5.89   -5.31
##  4 denmark          4.19  0.158     4.00    4.50   -5.76   0.176   -6.06   -5.48
##  5 france           3.82  0.0499    3.75    3.91   -5.87   0.241   -6.26   -5.53
##  6 germany          3.89  0.0239    3.85    3.93   -5.85   0.193   -6.16   -5.56
##  7 greece           4.88  0.255     4.48    5.38   -6.61   0.331   -7.16   -6.15
##  8 ireland          4.23  0.0437    4.16    4.33   -6.44   0.162   -6.72   -6.19
##  9 italy            3.73  0.220     3.38    4.05   -6.35   0.217   -6.73   -6.08
## 10 japan            4.70  0.684     3.95    6.00   -6.25   0.425   -6.99   -5.71
## 11 netherla         4.08  0.286     3.71    4.65   -5.92   0.193   -6.22   -5.66
## 12 norway           4.11  0.123     3.96    4.44   -5.75   0.201   -6.09   -5.42
## 13 spain            4.06  0.317     3.62    4.75   -5.63   0.278   -6.17   -5.29
## 14 sweden           4.01  0.0364    3.91    4.07   -7.82   0.126   -8.07   -7.67
## 15 switzerl         4.24  0.102     4.05    4.44   -5.93   0.124   -6.16   -5.75
## 16 turkey           5.77  0.329     5.14    6.16   -7.34   0.331   -7.84   -6.89
## 17 u.k.             3.98  0.0479    3.91    4.10   -6.02   0.107   -6.19   -5.84
## 18 u.s.a.           4.82  0.0219    4.79    4.86   -5.45   0.148   -5.70   -5.22
## # ... with 8 more variables: mean_lrpmg <dbl>, sd_lrpmg <dbl>, min_lrpmg <dbl>,
## #   max_lrpmg <dbl>, mean_lcarpcap <dbl>, sd_lcarpcap <dbl>,
## #   min_lcarpcap <dbl>, max_lcarpcap <dbl>, and abbreviated variable names
## #   1: mean_lgaspcar, 2: sd_lgaspcar, 3: min_lgaspcar, 4: max_lgaspcar,
## #   5: mean_lincomep, 6: sd_lincomep, 7: min_lincomep, 8: max_lincomep
\end{verbatim}

This allows you to summarise every column that contains real numbers. The difference between
\texttt{is.double()} and \texttt{is.numeric()} is that \texttt{is.numeric()} returns \texttt{TRUE} for integers too, whereas
\texttt{is.double()} returns \texttt{TRUE} for real numbers only (integers are real numbers too, but you know
what I mean). It is also possible to summarise every column at once:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{year) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{everything}\NormalTok{(), tibble}\SpecialCharTok{::}\FunctionTok{lst}\NormalTok{(mean, sd, min, max), }\AttributeTok{.names =} \StringTok{"\{fn\}\_\{col\}"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 17
##    country  mean_lgasp~1 sd_lg~2 min_l~3 max_l~4 mean_~5 sd_li~6 min_l~7 max_l~8
##    <chr>           <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
##  1 austria          4.06  0.0693    3.92    4.20   -6.12   0.235   -6.47   -5.76
##  2 belgium          3.92  0.103     3.82    4.16   -5.85   0.227   -6.22   -5.53
##  3 canada           4.86  0.0262    4.81    4.90   -5.58   0.193   -5.89   -5.31
##  4 denmark          4.19  0.158     4.00    4.50   -5.76   0.176   -6.06   -5.48
##  5 france           3.82  0.0499    3.75    3.91   -5.87   0.241   -6.26   -5.53
##  6 germany          3.89  0.0239    3.85    3.93   -5.85   0.193   -6.16   -5.56
##  7 greece           4.88  0.255     4.48    5.38   -6.61   0.331   -7.16   -6.15
##  8 ireland          4.23  0.0437    4.16    4.33   -6.44   0.162   -6.72   -6.19
##  9 italy            3.73  0.220     3.38    4.05   -6.35   0.217   -6.73   -6.08
## 10 japan            4.70  0.684     3.95    6.00   -6.25   0.425   -6.99   -5.71
## 11 netherla         4.08  0.286     3.71    4.65   -5.92   0.193   -6.22   -5.66
## 12 norway           4.11  0.123     3.96    4.44   -5.75   0.201   -6.09   -5.42
## 13 spain            4.06  0.317     3.62    4.75   -5.63   0.278   -6.17   -5.29
## 14 sweden           4.01  0.0364    3.91    4.07   -7.82   0.126   -8.07   -7.67
## 15 switzerl         4.24  0.102     4.05    4.44   -5.93   0.124   -6.16   -5.75
## 16 turkey           5.77  0.329     5.14    6.16   -7.34   0.331   -7.84   -6.89
## 17 u.k.             3.98  0.0479    3.91    4.10   -6.02   0.107   -6.19   -5.84
## 18 u.s.a.           4.82  0.0219    4.79    4.86   -5.45   0.148   -5.70   -5.22
## # ... with 8 more variables: mean_lrpmg <dbl>, sd_lrpmg <dbl>, min_lrpmg <dbl>,
## #   max_lrpmg <dbl>, mean_lcarpcap <dbl>, sd_lcarpcap <dbl>,
## #   min_lcarpcap <dbl>, max_lcarpcap <dbl>, and abbreviated variable names
## #   1: mean_lgaspcar, 2: sd_lgaspcar, 3: min_lgaspcar, 4: max_lgaspcar,
## #   5: mean_lincomep, 6: sd_lincomep, 7: min_lincomep, 8: max_lincomep
\end{verbatim}

I removed the \texttt{year} variable because it's not a variable for which we want to have descriptive
statistics.

\hypertarget{other-useful-tidyverse-functions}{%
\subsection{\texorpdfstring{Other useful \texttt{\{tidyverse\}} functions}{Other useful \{tidyverse\} functions}}\label{other-useful-tidyverse-functions}}

\hypertarget{if_else-case_when-and-recode}{%
\subsubsection{\texorpdfstring{\texttt{if\_else()}, \texttt{case\_when()} and \texttt{recode()}}{if\_else(), case\_when() and recode()}}\label{if_else-case_when-and-recode}}

Some other very useful \texttt{\{tidyverse\}} functions are \texttt{if\_else()} and \texttt{case\_when}. These two
functions, combined with \texttt{mutate()} make it easy to create a new variable whose values must
respect certain conditions. For instance, we might want to have a dummy that equals \texttt{1} if a country
in the European Union (to simplify, say as of 2017) and \texttt{0} if not. First let's create a list of
countries that are in the EU:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{eu\_countries }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\StringTok{"austria"}\NormalTok{, }\StringTok{"belgium"}\NormalTok{, }\StringTok{"bulgaria"}\NormalTok{, }\StringTok{"croatia"}\NormalTok{, }\StringTok{"republic of cyprus"}\NormalTok{,}
                  \StringTok{"czech republic"}\NormalTok{, }\StringTok{"denmark"}\NormalTok{, }\StringTok{"estonia"}\NormalTok{, }\StringTok{"finland"}\NormalTok{, }\StringTok{"france"}\NormalTok{, }\StringTok{"germany"}\NormalTok{,}
                  \StringTok{"greece"}\NormalTok{, }\StringTok{"hungary"}\NormalTok{, }\StringTok{"ireland"}\NormalTok{, }\StringTok{"italy"}\NormalTok{, }\StringTok{"latvia"}\NormalTok{, }\StringTok{"lithuania"}\NormalTok{, }\StringTok{"luxembourg"}\NormalTok{,}
                  \StringTok{"malta"}\NormalTok{, }\StringTok{"netherla"}\NormalTok{, }\StringTok{"poland"}\NormalTok{, }\StringTok{"portugal"}\NormalTok{, }\StringTok{"romania"}\NormalTok{, }\StringTok{"slovakia"}\NormalTok{, }\StringTok{"slovenia"}\NormalTok{,}
                  \StringTok{"spain"}\NormalTok{, }\StringTok{"sweden"}\NormalTok{, }\StringTok{"u.k."}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

I've had to change ``netherlands'' to ``netherla'' because that's how the country is called in the
\texttt{gasoline} data. Now let's create a dummy variable that equals \texttt{1} for EU countries, and \texttt{0} for the others:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{country =} \FunctionTok{tolower}\NormalTok{(country)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{in\_eu =} \FunctionTok{if\_else}\NormalTok{(country }\SpecialCharTok{\%in\%}\NormalTok{ eu\_countries, }\DecValTok{1}\NormalTok{, }\DecValTok{0}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 7
##    country  year lgaspcar lincomep  lrpmg lcarpcap in_eu
##    <chr>   <dbl>    <dbl>    <dbl>  <dbl>    <dbl> <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77     1
##  2 austria  1961     4.10    -6.43 -0.351    -9.61     1
##  3 austria  1962     4.07    -6.41 -0.380    -9.46     1
##  4 austria  1963     4.06    -6.37 -0.414    -9.34     1
##  5 austria  1964     4.04    -6.32 -0.445    -9.24     1
##  6 austria  1965     4.03    -6.29 -0.497    -9.12     1
##  7 austria  1966     4.05    -6.25 -0.467    -9.02     1
##  8 austria  1967     4.05    -6.23 -0.506    -8.93     1
##  9 austria  1968     4.05    -6.21 -0.522    -8.85     1
## 10 austria  1969     4.05    -6.15 -0.559    -8.79     1
## # ... with 332 more rows
\end{verbatim}

Instead of \texttt{1} and \texttt{0}, we can of course use strings (I add \texttt{filter(year\ ==\ 1960)} at the end to
have a better view of what happened):

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{country =} \FunctionTok{tolower}\NormalTok{(country)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{in\_eu =} \FunctionTok{if\_else}\NormalTok{(country }\SpecialCharTok{\%in\%}\NormalTok{ eu\_countries, }\StringTok{"yes"}\NormalTok{, }\StringTok{"no"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{==} \DecValTok{1960}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 7
##    country   year lgaspcar lincomep   lrpmg lcarpcap in_eu
##    <chr>    <dbl>    <dbl>    <dbl>   <dbl>    <dbl> <chr>
##  1 austria   1960     4.17    -6.47 -0.335     -9.77 yes  
##  2 belgium   1960     4.16    -6.22 -0.166     -9.41 yes  
##  3 canada    1960     4.86    -5.89 -0.972     -8.38 no   
##  4 denmark   1960     4.50    -6.06 -0.196     -9.33 yes  
##  5 france    1960     3.91    -6.26 -0.0196    -9.15 yes  
##  6 germany   1960     3.92    -6.16 -0.186     -9.34 yes  
##  7 greece    1960     5.04    -7.16 -0.0835   -12.2  yes  
##  8 ireland   1960     4.27    -6.72 -0.0765    -9.70 yes  
##  9 italy     1960     4.05    -6.73  0.165    -10.1  yes  
## 10 japan     1960     6.00    -6.99 -0.145    -12.2  no   
## 11 netherla  1960     4.65    -6.22 -0.201    -10.0  yes  
## 12 norway    1960     4.44    -6.09 -0.140     -9.68 no   
## 13 spain     1960     4.75    -6.17  1.13     -11.6  yes  
## 14 sweden    1960     4.06    -8.07 -2.52      -8.74 yes  
## 15 switzerl  1960     4.40    -6.16 -0.823     -9.26 no   
## 16 turkey    1960     6.13    -7.80 -0.253    -13.5  no   
## 17 u.k.      1960     4.10    -6.19 -0.391     -9.12 yes  
## 18 u.s.a.    1960     4.82    -5.70 -1.12      -8.02 no
\end{verbatim}

I think that \texttt{if\_else()} is fairly straightforward, especially if you know \texttt{ifelse()} already. You
might be wondering what is the difference between these two. \texttt{if\_else()} is stricter than
\texttt{ifelse()} and does not do type conversion. Compare the two next lines:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ifelse}\NormalTok{(}\DecValTok{1} \SpecialCharTok{==} \DecValTok{1}\NormalTok{, }\StringTok{"0"}\NormalTok{, }\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "0"
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{if\_else}\NormalTok{(}\DecValTok{1} \SpecialCharTok{==} \DecValTok{1}\NormalTok{, }\StringTok{"0"}\NormalTok{, }\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{Error}\SpecialCharTok{:} \StringTok{\textasciigrave{}}\AttributeTok{false}\StringTok{\textasciigrave{}}\NormalTok{ must be type string, not double}
\end{Highlighting}
\end{Shaded}

Type conversion, especially without a warning is very dangerous. \texttt{if\_else()}'s behaviour which
consists in failing as soon as possble avoids a lot of pain and suffering, especially when
programming non-interactively.

\texttt{if\_else()} also accepts an optional argument, that allows you to specify what should be returned
in case of \texttt{NA}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{if\_else}\NormalTok{(}\DecValTok{1} \SpecialCharTok{\textless{}=} \ConstantTok{NA}\NormalTok{, }\DecValTok{0}\NormalTok{, }\DecValTok{1}\NormalTok{, }\DecValTok{999}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 999
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{\# Or}
\FunctionTok{if\_else}\NormalTok{(}\DecValTok{1} \SpecialCharTok{\textless{}=} \ConstantTok{NA}\NormalTok{, }\DecValTok{0}\NormalTok{, }\DecValTok{1}\NormalTok{, }\ConstantTok{NA\_real\_}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] NA
\end{verbatim}

\texttt{case\_when()} can be seen as a generalization of \texttt{if\_else()}. Whenever you want to use multiple
\texttt{if\_else()}s, that's when you know you should use \texttt{case\_when()} (I'm adding the filter at the end
for the same reason as before, to see the output better):

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{country =} \FunctionTok{tolower}\NormalTok{(country)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{region =} \FunctionTok{case\_when}\NormalTok{(}
\NormalTok{           country }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"france"}\NormalTok{, }\StringTok{"italy"}\NormalTok{, }\StringTok{"turkey"}\NormalTok{, }\StringTok{"greece"}\NormalTok{, }\StringTok{"spain"}\NormalTok{) }\SpecialCharTok{\textasciitilde{}} \StringTok{"mediterranean"}\NormalTok{,}
\NormalTok{           country }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"germany"}\NormalTok{, }\StringTok{"austria"}\NormalTok{, }\StringTok{"switzerl"}\NormalTok{, }\StringTok{"belgium"}\NormalTok{, }\StringTok{"netherla"}\NormalTok{) }\SpecialCharTok{\textasciitilde{}} \StringTok{"central europe"}\NormalTok{,}
\NormalTok{           country }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"canada"}\NormalTok{, }\StringTok{"u.s.a."}\NormalTok{, }\StringTok{"u.k."}\NormalTok{, }\StringTok{"ireland"}\NormalTok{) }\SpecialCharTok{\textasciitilde{}} \StringTok{"anglosphere"}\NormalTok{,}
\NormalTok{           country }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"denmark"}\NormalTok{, }\StringTok{"norway"}\NormalTok{, }\StringTok{"sweden"}\NormalTok{) }\SpecialCharTok{\textasciitilde{}} \StringTok{"nordic"}\NormalTok{,}
\NormalTok{           country }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"japan"}\NormalTok{) }\SpecialCharTok{\textasciitilde{}} \StringTok{"asia"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{==} \DecValTok{1960}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 7
##    country   year lgaspcar lincomep   lrpmg lcarpcap region        
##    <chr>    <dbl>    <dbl>    <dbl>   <dbl>    <dbl> <chr>         
##  1 austria   1960     4.17    -6.47 -0.335     -9.77 central europe
##  2 belgium   1960     4.16    -6.22 -0.166     -9.41 central europe
##  3 canada    1960     4.86    -5.89 -0.972     -8.38 anglosphere   
##  4 denmark   1960     4.50    -6.06 -0.196     -9.33 nordic        
##  5 france    1960     3.91    -6.26 -0.0196    -9.15 mediterranean 
##  6 germany   1960     3.92    -6.16 -0.186     -9.34 central europe
##  7 greece    1960     5.04    -7.16 -0.0835   -12.2  mediterranean 
##  8 ireland   1960     4.27    -6.72 -0.0765    -9.70 anglosphere   
##  9 italy     1960     4.05    -6.73  0.165    -10.1  mediterranean 
## 10 japan     1960     6.00    -6.99 -0.145    -12.2  asia          
## 11 netherla  1960     4.65    -6.22 -0.201    -10.0  central europe
## 12 norway    1960     4.44    -6.09 -0.140     -9.68 nordic        
## 13 spain     1960     4.75    -6.17  1.13     -11.6  mediterranean 
## 14 sweden    1960     4.06    -8.07 -2.52      -8.74 nordic        
## 15 switzerl  1960     4.40    -6.16 -0.823     -9.26 central europe
## 16 turkey    1960     6.13    -7.80 -0.253    -13.5  mediterranean 
## 17 u.k.      1960     4.10    -6.19 -0.391     -9.12 anglosphere   
## 18 u.s.a.    1960     4.82    -5.70 -1.12      -8.02 anglosphere
\end{verbatim}

If all you want is to recode values, you can use \texttt{recode()}. For example, the Netherlands is
written as ``NETHERLA'' in the \texttt{gasoline} data, which is quite ugly. Same for Switzerland:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{country =} \FunctionTok{tolower}\NormalTok{(country)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{country =} \FunctionTok{recode}\NormalTok{(country, }\StringTok{"netherla"} \OtherTok{=} \StringTok{"netherlands"}\NormalTok{, }\StringTok{"switzerl"} \OtherTok{=} \StringTok{"switzerland"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

I saved the data with these changes as they will become useful in the future. Let's take a look at
the data:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(country }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"netherlands"}\NormalTok{, }\StringTok{"switzerland"}\NormalTok{), year }\SpecialCharTok{==} \DecValTok{1960}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 2 x 6
##   country      year lgaspcar lincomep  lrpmg lcarpcap
##   <chr>       <dbl>    <dbl>    <dbl>  <dbl>    <dbl>
## 1 netherlands  1960     4.65    -6.22 -0.201   -10.0 
## 2 switzerland  1960     4.40    -6.16 -0.823    -9.26
\end{verbatim}

\hypertarget{lead-and-lag}{%
\subsubsection{\texorpdfstring{\texttt{lead()} and \texttt{lag()}}{lead() and lag()}}\label{lead-and-lag}}

\texttt{lead()} and \texttt{lag()} are especially useful in econometrics. When I was doing my masters, in 4 B.d.
(\emph{Before dplyr}) lagging variables in panel data was quite tricky. Now, with \texttt{\{dplyr\}} it's really
very easy:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lag\_lgaspcar =} \FunctionTok{lag}\NormalTok{(lgaspcar)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lead\_lgaspcar =} \FunctionTok{lead}\NormalTok{(lgaspcar)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{\%in\%} \FunctionTok{seq}\NormalTok{(}\DecValTok{1960}\NormalTok{, }\DecValTok{1963}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 72 x 8
## # Groups:   country [18]
##    country  year lgaspcar lincomep  lrpmg lcarpcap lag_lgaspcar lead_lgaspcar
##    <chr>   <dbl>    <dbl>    <dbl>  <dbl>    <dbl>        <dbl>         <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77        NA             4.10
##  2 austria  1961     4.10    -6.43 -0.351    -9.61         4.17          4.07
##  3 austria  1962     4.07    -6.41 -0.380    -9.46         4.10          4.06
##  4 austria  1963     4.06    -6.37 -0.414    -9.34         4.07          4.04
##  5 belgium  1960     4.16    -6.22 -0.166    -9.41        NA             4.12
##  6 belgium  1961     4.12    -6.18 -0.172    -9.30         4.16          4.08
##  7 belgium  1962     4.08    -6.13 -0.222    -9.22         4.12          4.00
##  8 belgium  1963     4.00    -6.09 -0.250    -9.11         4.08          3.99
##  9 canada   1960     4.86    -5.89 -0.972    -8.38        NA             4.83
## 10 canada   1961     4.83    -5.88 -0.972    -8.35         4.86          4.85
## # ... with 62 more rows
\end{verbatim}

To lag every variable, remember that you can use \texttt{mutate\_if()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate\_if}\NormalTok{(is.double, lag) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{\%in\%} \FunctionTok{seq}\NormalTok{(}\DecValTok{1960}\NormalTok{, }\DecValTok{1963}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `mutate_if()` ignored the following grouping variables:
## * Column `country`
\end{verbatim}

\begin{verbatim}
## # A tibble: 72 x 6
## # Groups:   country [18]
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <dbl>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77
##  2 austria  1961     4.10    -6.43 -0.351    -9.61
##  3 austria  1962     4.07    -6.41 -0.380    -9.46
##  4 austria  1963     4.06    -6.37 -0.414    -9.34
##  5 belgium  1960     4.16    -6.22 -0.166    -9.41
##  6 belgium  1961     4.12    -6.18 -0.172    -9.30
##  7 belgium  1962     4.08    -6.13 -0.222    -9.22
##  8 belgium  1963     4.00    -6.09 -0.250    -9.11
##  9 canada   1960     4.86    -5.89 -0.972    -8.38
## 10 canada   1961     4.83    -5.88 -0.972    -8.35
## # ... with 62 more rows
\end{verbatim}

you can replace \texttt{lag()} with \texttt{lead()}, but just keep in mind that the columns get transformed in
place.

\hypertarget{ntile}{%
\subsubsection{\texorpdfstring{\texttt{ntile()}}{ntile()}}\label{ntile}}

The last helper function I will discuss is \texttt{ntile()}. There are some other, so do read \texttt{mutate()}'s
documentation with \texttt{help(mutate)}!

If you need quantiles, you need \texttt{ntile()}. Let's see how it works:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{quintile =} \FunctionTok{ntile}\NormalTok{(lgaspcar, }\DecValTok{5}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{decile =} \FunctionTok{ntile}\NormalTok{(lgaspcar, }\DecValTok{10}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(country, year, lgaspcar, quintile, decile)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 5
##    country  year lgaspcar quintile decile
##    <chr>   <dbl>    <dbl>    <int>  <int>
##  1 austria  1960     4.17        3      6
##  2 austria  1961     4.10        3      6
##  3 austria  1962     4.07        3      5
##  4 austria  1963     4.06        3      5
##  5 austria  1964     4.04        3      5
##  6 austria  1965     4.03        3      5
##  7 austria  1966     4.05        3      5
##  8 austria  1967     4.05        3      5
##  9 austria  1968     4.05        3      5
## 10 austria  1969     4.05        3      5
## # ... with 332 more rows
\end{verbatim}

\texttt{quintile} and \texttt{decile} do not hold the values but the quantile the value lies in. If you want to
have a column that contains the median for instance, you can use good ol' \texttt{quantile()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{median =} \FunctionTok{quantile}\NormalTok{(lgaspcar, }\FloatTok{0.5}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%} \CommentTok{\# quantile(x, 0.5) is equivalent to median(x)}
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{==} \DecValTok{1960}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(country, year, median)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 3
## # Groups:   country [18]
##    country      year median
##    <chr>       <dbl>  <dbl>
##  1 austria      1960   4.05
##  2 belgium      1960   3.88
##  3 canada       1960   4.86
##  4 denmark      1960   4.16
##  5 france       1960   3.81
##  6 germany      1960   3.89
##  7 greece       1960   4.89
##  8 ireland      1960   4.22
##  9 italy        1960   3.74
## 10 japan        1960   4.52
## 11 netherlands  1960   3.99
## 12 norway       1960   4.08
## 13 spain        1960   3.99
## 14 sweden       1960   4.00
## 15 switzerland  1960   4.26
## 16 turkey       1960   5.72
## 17 u.k.         1960   3.98
## 18 u.s.a.       1960   4.81
\end{verbatim}

\hypertarget{arrange}{%
\subsubsection{\texorpdfstring{\texttt{arrange()}}{arrange()}}\label{arrange}}

\texttt{arrange()} re-orders the whole \texttt{tibble} according to values of the supplied variable:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{arrange}\NormalTok{(lgaspcar)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 6
##    country  year lgaspcar lincomep   lrpmg lcarpcap
##    <chr>   <dbl>    <dbl>    <dbl>   <dbl>    <dbl>
##  1 italy    1977     3.38    -6.10  0.164     -8.15
##  2 italy    1978     3.39    -6.08  0.0348    -8.11
##  3 italy    1976     3.43    -6.12  0.103     -8.17
##  4 italy    1974     3.50    -6.13 -0.223     -8.26
##  5 italy    1975     3.52    -6.17 -0.0327    -8.22
##  6 spain    1978     3.62    -5.29  0.621     -8.63
##  7 italy    1972     3.63    -6.21 -0.215     -8.38
##  8 italy    1971     3.65    -6.22 -0.148     -8.47
##  9 spain    1977     3.65    -5.30  0.526     -8.73
## 10 italy    1973     3.65    -6.16 -0.325     -8.32
## # ... with 332 more rows
\end{verbatim}

If you want to re-order the \texttt{tibble} in descending order of the variable:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{arrange}\NormalTok{(}\FunctionTok{desc}\NormalTok{(lgaspcar))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 6
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <dbl>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 turkey   1966     6.16    -7.51 -0.356    -13.0
##  2 turkey   1960     6.13    -7.80 -0.253    -13.5
##  3 turkey   1961     6.11    -7.79 -0.343    -13.4
##  4 turkey   1962     6.08    -7.84 -0.408    -13.2
##  5 turkey   1968     6.08    -7.42 -0.365    -12.8
##  6 turkey   1963     6.08    -7.63 -0.225    -13.3
##  7 turkey   1964     6.06    -7.63 -0.252    -13.2
##  8 turkey   1967     6.04    -7.46 -0.335    -12.8
##  9 japan    1960     6.00    -6.99 -0.145    -12.2
## 10 turkey   1965     5.82    -7.62 -0.293    -12.9
## # ... with 332 more rows
\end{verbatim}

\texttt{arrange}'s documentation alerts the user that re-ording by group is only possible by explicitely
specifying an option:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(year }\SpecialCharTok{\%in\%} \FunctionTok{seq}\NormalTok{(}\DecValTok{1960}\NormalTok{, }\DecValTok{1963}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{arrange}\NormalTok{(}\FunctionTok{desc}\NormalTok{(lgaspcar), }\AttributeTok{.by\_group =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 72 x 6
## # Groups:   country [18]
##    country  year lgaspcar lincomep  lrpmg lcarpcap
##    <chr>   <dbl>    <dbl>    <dbl>  <dbl>    <dbl>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77
##  2 austria  1961     4.10    -6.43 -0.351    -9.61
##  3 austria  1962     4.07    -6.41 -0.380    -9.46
##  4 austria  1963     4.06    -6.37 -0.414    -9.34
##  5 belgium  1960     4.16    -6.22 -0.166    -9.41
##  6 belgium  1961     4.12    -6.18 -0.172    -9.30
##  7 belgium  1962     4.08    -6.13 -0.222    -9.22
##  8 belgium  1963     4.00    -6.09 -0.250    -9.11
##  9 canada   1960     4.86    -5.89 -0.972    -8.38
## 10 canada   1962     4.85    -5.84 -0.979    -8.32
## # ... with 62 more rows
\end{verbatim}

This is especially useful for plotting. We'll see this in Chapter 6.

\hypertarget{tally-and-count}{%
\subsubsection{\texorpdfstring{\texttt{tally()} and \texttt{count()}}{tally() and count()}}\label{tally-and-count}}

\texttt{tally()} and \texttt{count()} count the number of observations in your data. I believe \texttt{count()} is the
more useful of the two, as it counts the number of observations within a group that you can provide:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{count}\NormalTok{(country)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 18 x 2
##    country         n
##    <chr>       <int>
##  1 austria        19
##  2 belgium        19
##  3 canada         19
##  4 denmark        19
##  5 france         19
##  6 germany        19
##  7 greece         19
##  8 ireland        19
##  9 italy          19
## 10 japan          19
## 11 netherlands    19
## 12 norway         19
## 13 spain          19
## 14 sweden         19
## 15 switzerland    19
## 16 turkey         19
## 17 u.k.           19
## 18 u.s.a.         19
\end{verbatim}

There's also \texttt{add\_count()} which adds the column to the data:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{add\_count}\NormalTok{(country)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 7
##    country  year lgaspcar lincomep  lrpmg lcarpcap     n
##    <chr>   <dbl>    <dbl>    <dbl>  <dbl>    <dbl> <int>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77    19
##  2 austria  1961     4.10    -6.43 -0.351    -9.61    19
##  3 austria  1962     4.07    -6.41 -0.380    -9.46    19
##  4 austria  1963     4.06    -6.37 -0.414    -9.34    19
##  5 austria  1964     4.04    -6.32 -0.445    -9.24    19
##  6 austria  1965     4.03    -6.29 -0.497    -9.12    19
##  7 austria  1966     4.05    -6.25 -0.467    -9.02    19
##  8 austria  1967     4.05    -6.23 -0.506    -8.93    19
##  9 austria  1968     4.05    -6.21 -0.522    -8.85    19
## 10 austria  1969     4.05    -6.15 -0.559    -8.79    19
## # ... with 332 more rows
\end{verbatim}

\texttt{add\_count()} is a shortcut for the following code:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(country) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{n =} \FunctionTok{n}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 342 x 7
## # Groups:   country [18]
##    country  year lgaspcar lincomep  lrpmg lcarpcap     n
##    <chr>   <dbl>    <dbl>    <dbl>  <dbl>    <dbl> <int>
##  1 austria  1960     4.17    -6.47 -0.335    -9.77    19
##  2 austria  1961     4.10    -6.43 -0.351    -9.61    19
##  3 austria  1962     4.07    -6.41 -0.380    -9.46    19
##  4 austria  1963     4.06    -6.37 -0.414    -9.34    19
##  5 austria  1964     4.04    -6.32 -0.445    -9.24    19
##  6 austria  1965     4.03    -6.29 -0.497    -9.12    19
##  7 austria  1966     4.05    -6.25 -0.467    -9.02    19
##  8 austria  1967     4.05    -6.23 -0.506    -8.93    19
##  9 austria  1968     4.05    -6.21 -0.522    -8.85    19
## 10 austria  1969     4.05    -6.15 -0.559    -8.79    19
## # ... with 332 more rows
\end{verbatim}

where \texttt{n()} is a \texttt{\{dplyr\}} function that can only be used within \texttt{summarise()}, \texttt{mutate()} and
\texttt{filter()}.

\hypertarget{special-packages-for-special-kinds-of-data-forcats-lubridate-and-stringr}{%
\subsection{\texorpdfstring{Special packages for special kinds of data: \texttt{\{forcats\}}, \texttt{\{lubridate\}}, and \texttt{\{stringr\}}}{Special packages for special kinds of data: \{forcats\}, \{lubridate\}, and \{stringr\}}}\label{special-packages-for-special-kinds-of-data-forcats-lubridate-and-stringr}}

\hypertarget{section}{%
\subsubsection{🐱🐱🐱🐱}\label{section}}

Factor variables are very useful but not very easy to manipulate. \texttt{forcats} contains very useful
functions that make working on factor variables painless. In my opinion, the four following functions, \texttt{fct\_recode()}, \texttt{fct\_relevel()}, \texttt{fct\_reorder()} and \texttt{fct\_relabel()}, are the ones you must
know, so that's what I'll be showing.

Remember in chapter 3 when I very quickly explained what were \texttt{factor} variables? In this section,
we are going to work a little bit with these type of variable. \texttt{factor}s are very useful, and the
\texttt{forcats} package includes some handy functions to work with them. First, let's load the \texttt{forcats} package:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(forcats)}
\end{Highlighting}
\end{Shaded}

as an example, we are going to work with the \texttt{gss\_cat} dataset that is included in \texttt{forcats}. Let's
load the data:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(gss\_cat)}

\FunctionTok{head}\NormalTok{(gss\_cat)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 9
##    year marital         age race  rincome        partyid     relig denom tvhours
##   <int> <fct>         <int> <fct> <fct>          <fct>       <fct> <fct>   <int>
## 1  2000 Never married    26 White $8000 to 9999  Ind,near r~ Prot~ Sout~      12
## 2  2000 Divorced         48 White $8000 to 9999  Not str re~ Prot~ Bapt~      NA
## 3  2000 Widowed          67 White Not applicable Independent Prot~ No d~       2
## 4  2000 Never married    39 White Not applicable Ind,near r~ Orth~ Not ~       4
## 5  2000 Divorced         25 White Not applicable Not str de~ None  Not ~       1
## 6  2000 Married          25 White $20000 - 24999 Strong dem~ Prot~ Sout~      NA
\end{verbatim}

as you can see, \texttt{marital}, \texttt{race}, \texttt{rincome} and \texttt{partyid} are all factor variables. Let's take a closer
look at \texttt{marital}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{str}\NormalTok{(gss\_cat}\SpecialCharTok{$}\NormalTok{marital)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  Factor w/ 6 levels "No answer","Never married",..: 2 4 5 2 4 6 2 4 6 6 ...
\end{verbatim}

and let's see \texttt{rincome}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{str}\NormalTok{(gss\_cat}\SpecialCharTok{$}\NormalTok{rincome)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  Factor w/ 16 levels "No answer","Don't know",..: 8 8 16 16 16 5 4 9 4 4 ...
\end{verbatim}

\texttt{factor} variables have different levels and the \texttt{forcats} package includes functions that allow
you to recode, collapse and do all sorts of things on these levels. For example , using
\texttt{forcats::fct\_recode()} you can recode levels:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gss\_cat }\OtherTok{\textless{}{-}}\NormalTok{ gss\_cat }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{marital =} \FunctionTok{fct\_recode}\NormalTok{(marital,}
                              \AttributeTok{refuse =} \StringTok{"No answer"}\NormalTok{,}
                              \AttributeTok{never\_married =} \StringTok{"Never married"}\NormalTok{,}
                              \AttributeTok{divorced =} \StringTok{"Separated"}\NormalTok{,}
                              \AttributeTok{divorced =} \StringTok{"Divorced"}\NormalTok{,}
                              \AttributeTok{widowed =} \StringTok{"Widowed"}\NormalTok{,}
                              \AttributeTok{married =} \StringTok{"Married"}\NormalTok{))}

\NormalTok{gss\_cat }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{tabyl}\NormalTok{(marital)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##        marital     n      percent
##         refuse    17 0.0007913234
##  never_married  5416 0.2521063166
##       divorced  4126 0.1920588372
##        widowed  1807 0.0841130196
##        married 10117 0.4709305032
\end{verbatim}

Using \texttt{fct\_recode()}, I was able to recode the levels and collapse \texttt{Separated} and \texttt{Divorced} to
a single category called \texttt{divorced}. As you can see, \texttt{refuse} and \texttt{widowed} are less than 10\%, so
maybe you'd want to lump these categories together:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gss\_cat }\OtherTok{\textless{}{-}}\NormalTok{ gss\_cat }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{marital =} \FunctionTok{fct\_lump}\NormalTok{(marital, }\AttributeTok{prop =} \FloatTok{0.10}\NormalTok{, }\AttributeTok{other\_level =} \StringTok{"other"}\NormalTok{))}

\NormalTok{gss\_cat }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{tabyl}\NormalTok{(marital)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##        marital     n    percent
##  never_married  5416 0.25210632
##       divorced  4126 0.19205884
##        married 10117 0.47093050
##          other  1824 0.08490434
\end{verbatim}

\texttt{fct\_reorder()} is especially useful for plotting. We will explore plotting in the next chapter,
but to show you why \texttt{fct\_reorder()} is so useful, I will create a barplot, first without
using \texttt{fct\_reorder()} to re-order the factors, then with reordering. Do not worry if you don't
understand all the code for now:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gss\_cat }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{tabyl}\NormalTok{(marital) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{ggplot}\NormalTok{() }\SpecialCharTok{+}
    \FunctionTok{geom\_col}\NormalTok{(}\FunctionTok{aes}\NormalTok{(}\AttributeTok{y =}\NormalTok{ n, }\AttributeTok{x =}\NormalTok{ marital)) }\SpecialCharTok{+}
    \FunctionTok{coord\_flip}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-328-1.pdf}

It would be much better if the categories were ordered by frequency. This is easy to do with
\texttt{fct\_reorder()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gss\_cat }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{tabyl}\NormalTok{(marital) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{marital =} \FunctionTok{fct\_reorder}\NormalTok{(marital, n, }\AttributeTok{.desc =} \ConstantTok{FALSE}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{ggplot}\NormalTok{() }\SpecialCharTok{+}
    \FunctionTok{geom\_col}\NormalTok{(}\FunctionTok{aes}\NormalTok{(}\AttributeTok{y =}\NormalTok{ n, }\AttributeTok{x =}\NormalTok{ marital)) }\SpecialCharTok{+}
    \FunctionTok{coord\_flip}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-329-1.pdf}

Much better! In Chapter 6, we are going to learn about \texttt{\{ggplot2\}}.

The last family of function I'd like to mention are the \texttt{fct\_lump*()} functions. These make it possible
to lump several levels of a factor into a new \emph{other} level:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{gss\_cat }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}
    \CommentTok{\# Description of the different functions taken from help(fct\_lump)}
    \AttributeTok{denom\_lowfreq =} \FunctionTok{fct\_lump\_lowfreq}\NormalTok{(denom), }\CommentTok{\# lumps together the least frequent levels, ensuring that "other" is still the smallest level.}
    \AttributeTok{denom\_min =} \FunctionTok{fct\_lump\_min}\NormalTok{(denom, }\AttributeTok{min =} \DecValTok{10}\NormalTok{), }\CommentTok{\# lumps levels that appear fewer than min times.}
    \AttributeTok{denom\_n =} \FunctionTok{fct\_lump\_n}\NormalTok{(denom, }\AttributeTok{n =} \DecValTok{3}\NormalTok{), }\CommentTok{\# lumps all levels except for the n most frequent (or least frequent if n \textless{} 0)}
    \AttributeTok{denom\_prop =} \FunctionTok{fct\_lump\_prop}\NormalTok{(denom, }\AttributeTok{prop =} \FloatTok{0.10}\NormalTok{) }\CommentTok{\# lumps levels that appear in fewer prop * n times.}
\NormalTok{         )}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 21,483 x 13
##     year marital   age race  rincome partyid relig denom tvhours denom~1 denom~2
##    <int> <fct>   <int> <fct> <fct>   <fct>   <fct> <fct>   <int> <fct>   <fct>  
##  1  2000 never_~    26 White $8000 ~ Ind,ne~ Prot~ Sout~      12 Southe~ Southe~
##  2  2000 divorc~    48 White $8000 ~ Not st~ Prot~ Bapt~      NA Baptis~ Baptis~
##  3  2000 other      67 White Not ap~ Indepe~ Prot~ No d~       2 No den~ No den~
##  4  2000 never_~    39 White Not ap~ Ind,ne~ Orth~ Not ~       4 Not ap~ Not ap~
##  5  2000 divorc~    25 White Not ap~ Not st~ None  Not ~       1 Not ap~ Not ap~
##  6  2000 married    25 White $20000~ Strong~ Prot~ Sout~      NA Southe~ Southe~
##  7  2000 never_~    36 White $25000~ Not st~ Chri~ Not ~       3 Not ap~ Not ap~
##  8  2000 divorc~    44 White $7000 ~ Ind,ne~ Prot~ Luth~      NA Luther~ Luther~
##  9  2000 married    44 White $25000~ Not st~ Prot~ Other       0 Other   Other  
## 10  2000 married    47 White $25000~ Strong~ Prot~ Sout~       3 Southe~ Southe~
## # ... with 21,473 more rows, 2 more variables: denom_n <fct>, denom_prop <fct>,
## #   and abbreviated variable names 1: denom_lowfreq, 2: denom_min
\end{verbatim}

There's many other, so I'd advise you go through the package's function \href{https://forcats.tidyverse.org/reference/index.html}{reference}.

\hypertarget{get-your-dates-right-with-lubridate}{%
\subsubsection{\texorpdfstring{Get your dates right with \texttt{\{lubridate\}}}{Get your dates right with \{lubridate\}}}\label{get-your-dates-right-with-lubridate}}

\texttt{\{lubridate\}} is yet another tidyverse package, that makes dealing with dates or durations (and intervals) as
painless as possible. I do not use every function contained in the package daily, and as such will
only focus on some of the functions. However, if you have to deal with dates often,
you might want to explore the package thouroughly.

\hypertarget{defining-dates-the-tidy-way}{%
\paragraph{Defining dates, the tidy way}\label{defining-dates-the-tidy-way}}

Let's load new dataset, called \emph{independence} from the datasets folder:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{independence }\OtherTok{\textless{}{-}} \FunctionTok{readRDS}\NormalTok{(}\StringTok{"datasets/independence.rds"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

This dataset was scraped from the following Wikipedia \href{https://en.wikipedia.org/wiki/Decolonisation_of_Africa\#Timeline}{page}.
It shows when African countries gained independence and from which colonial powers. In Chapter 10, I
will show you how to scrape Wikipedia pages using R. For now, let's take a look at the contents
of the dataset:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{independence}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 54 x 6
##    country      colonial_name                    colon~1 indep~2 first~3 indep~4
##    <chr>        <chr>                            <chr>   <chr>   <chr>   <chr>  
##  1 Liberia      Liberia                          United~ 26 Jul~ Joseph~ Liberi~
##  2 South Africa Cape Colony  Colony of Natal  O~ United~ 31 May~ Louis ~ South ~
##  3 Egypt        Sultanate of Egypt               United~ 28 Feb~ Fuad I  Egypti~
##  4 Eritrea      Italian Eritrea                  Italy   10 Feb~ Haile ~ -      
##  5 Libya        British Military Administration~ United~ 24 Dec~ Idris   -      
##  6 Sudan        Anglo-Egyptian Sudan             United~ 1 Janu~ Ismail~ -      
##  7 Tunisia      French Protectorate of Tunisia   France  20 Mar~ Muhamm~ -      
##  8 Morocco      French Protectorate in Morocco ~ France~ 2 Marc~ Mohamm~ Ifni W~
##  9 Ghana        Gold Coast                       United~ 6 Marc~ Kwame ~ Gold C~
## 10 Guinea       French West Africa               France  2 Octo~ Ahmed ~ Guinea~
## # ... with 44 more rows, and abbreviated variable names 1: colonial_power,
## #   2: independence_date, 3: first_head_of_state, 4: independence_won_through
\end{verbatim}

as you can see, the date of independence is in a format that might make it difficult to answer questions
such as \emph{Which African countries gained independence before 1960 ?} for two reasons. First of all,
the date uses the name of the month instead of the number of the month, and second of all the type of
the independence day column is \emph{character} and not ``date''. So our first task is to correctly define the column
as being of type date, while making sure that R understands that \emph{January} is supposed to be ``01'', and so
on. There are several helpful functions included in \texttt{\{lubridate\}} to convert columns to dates. For instance
if the column you want to convert is of the form ``2012-11-21'', then you would use the function \texttt{ymd()},
for ``year-month-day''. If, however the column is ``2012-21-11'', then you would use \texttt{ydm()}. There's
a few of these helper functions, and they can handle a lot of different formats for dates. In our case,
having the name of the month instead of the number might seem quite problematic, but it turns out
that this is a case that \texttt{\{lubridate\}} handles painfully:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(lubridate)}

\NormalTok{independence }\OtherTok{\textless{}{-}}\NormalTok{ independence }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{independence\_date =} \FunctionTok{dmy}\NormalTok{(independence\_date))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: 5 failed to parse.
\end{verbatim}

Some dates failed to parse, for instance for Morocco. This is because these countries have several
independence dates; this means that the string to convert looks like:

\begin{verbatim}
"2 March 1956
7 April 1956
10 April 1958
4 January 1969"
\end{verbatim}

which obviously cannot be converted by \texttt{\{lubridate\}} without further manipulation. I ignore these cases for
simplicity's sake.

\hypertarget{data-manipulation-with-dates}{%
\paragraph{Data manipulation with dates}\label{data-manipulation-with-dates}}

Let's take a look at the data now:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{independence}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 54 x 6
##    country      colonial_name                 colon~1 independ~2 first~3 indep~4
##    <chr>        <chr>                         <chr>   <date>     <chr>   <chr>  
##  1 Liberia      Liberia                       United~ 1847-07-26 Joseph~ Liberi~
##  2 South Africa Cape Colony  Colony of Natal~ United~ 1910-05-31 Louis ~ South ~
##  3 Egypt        Sultanate of Egypt            United~ 1922-02-28 Fuad I  Egypti~
##  4 Eritrea      Italian Eritrea               Italy   1947-02-10 Haile ~ -      
##  5 Libya        British Military Administrat~ United~ 1951-12-24 Idris   -      
##  6 Sudan        Anglo-Egyptian Sudan          United~ 1956-01-01 Ismail~ -      
##  7 Tunisia      French Protectorate of Tunis~ France  1956-03-20 Muhamm~ -      
##  8 Morocco      French Protectorate in Moroc~ France~ NA         Mohamm~ Ifni W~
##  9 Ghana        Gold Coast                    United~ 1957-03-06 Kwame ~ Gold C~
## 10 Guinea       French West Africa            France  1958-10-02 Ahmed ~ Guinea~
## # ... with 44 more rows, and abbreviated variable names 1: colonial_power,
## #   2: independence_date, 3: first_head_of_state, 4: independence_won_through
\end{verbatim}

As you can see, we now have a date column in the right format. We can now answer questions such as
\emph{Which countries gained independence before 1960?} quite easily, by using the functions \texttt{year()},
\texttt{month()} and \texttt{day()}. Let's see which countries gained independence before 1960:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{independence }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{year}\NormalTok{(independence\_date) }\SpecialCharTok{\textless{}=} \DecValTok{1960}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{pull}\NormalTok{(country)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] "Liberia"                          "South Africa"                    
##  [3] "Egypt"                            "Eritrea"                         
##  [5] "Libya"                            "Sudan"                           
##  [7] "Tunisia"                          "Ghana"                           
##  [9] "Guinea"                           "Cameroon"                        
## [11] "Togo"                             "Mali"                            
## [13] "Madagascar"                       "Democratic Republic of the Congo"
## [15] "Benin"                            "Niger"                           
## [17] "Burkina Faso"                     "Ivory Coast"                     
## [19] "Chad"                             "Central African Republic"        
## [21] "Republic of the Congo"            "Gabon"                           
## [23] "Mauritania"
\end{verbatim}

You guessed it, \texttt{year()} extracts the year of the date column and converts it as a \emph{numeric} so that we can work
on it. This is the same for \texttt{month()} or \texttt{day()}. Let's try to see if countries gained their independence on
Christmas Eve:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{independence }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{month}\NormalTok{(independence\_date) }\SpecialCharTok{==} \DecValTok{12}\NormalTok{,}
         \FunctionTok{day}\NormalTok{(independence\_date) }\SpecialCharTok{==} \DecValTok{24}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{pull}\NormalTok{(country)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "Libya"
\end{verbatim}

Seems like Libya was the only one! You can also operate on dates. For instance, let's compute the difference between
two dates, using the \texttt{interval()} column:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{independence }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{today =}\NormalTok{ lubridate}\SpecialCharTok{::}\FunctionTok{today}\NormalTok{()) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{independent\_since =} \FunctionTok{interval}\NormalTok{(independence\_date, today)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(country, independent\_since)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 54 x 2
##    country      independent_since             
##    <chr>        <Interval>                    
##  1 Liberia      1847-07-26 UTC--2022-10-13 UTC
##  2 South Africa 1910-05-31 UTC--2022-10-13 UTC
##  3 Egypt        1922-02-28 UTC--2022-10-13 UTC
##  4 Eritrea      1947-02-10 UTC--2022-10-13 UTC
##  5 Libya        1951-12-24 UTC--2022-10-13 UTC
##  6 Sudan        1956-01-01 UTC--2022-10-13 UTC
##  7 Tunisia      1956-03-20 UTC--2022-10-13 UTC
##  8 Morocco      NA--NA                        
##  9 Ghana        1957-03-06 UTC--2022-10-13 UTC
## 10 Guinea       1958-10-02 UTC--2022-10-13 UTC
## # ... with 44 more rows
\end{verbatim}

The \texttt{independent\_since} column now contains an \emph{interval} object that we can convert to years:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{independence }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{today =}\NormalTok{ lubridate}\SpecialCharTok{::}\FunctionTok{today}\NormalTok{()) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{independent\_since =} \FunctionTok{interval}\NormalTok{(independence\_date, today)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(country, independent\_since) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{years\_independent =} \FunctionTok{as.numeric}\NormalTok{(independent\_since, }\StringTok{"years"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 54 x 3
##    country      independent_since              years_independent
##    <chr>        <Interval>                                 <dbl>
##  1 Liberia      1847-07-26 UTC--2022-10-13 UTC             175. 
##  2 South Africa 1910-05-31 UTC--2022-10-13 UTC             112. 
##  3 Egypt        1922-02-28 UTC--2022-10-13 UTC             101. 
##  4 Eritrea      1947-02-10 UTC--2022-10-13 UTC              75.7
##  5 Libya        1951-12-24 UTC--2022-10-13 UTC              70.8
##  6 Sudan        1956-01-01 UTC--2022-10-13 UTC              66.8
##  7 Tunisia      1956-03-20 UTC--2022-10-13 UTC              66.6
##  8 Morocco      NA--NA                                      NA  
##  9 Ghana        1957-03-06 UTC--2022-10-13 UTC              65.6
## 10 Guinea       1958-10-02 UTC--2022-10-13 UTC              64.0
## # ... with 44 more rows
\end{verbatim}

We can now see for how long the last country to gain independence has been independent.
Because the data is not tidy (in some cases, an African country was colonized by two powers,
see Libya), I will only focus on 4 European colonial powers: Belgium, France, Portugal and the United Kingdom:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{independence }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(colonial\_power }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Belgium"}\NormalTok{, }\StringTok{"France"}\NormalTok{, }\StringTok{"Portugal"}\NormalTok{, }\StringTok{"United Kingdom"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{today =}\NormalTok{ lubridate}\SpecialCharTok{::}\FunctionTok{today}\NormalTok{()) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{independent\_since =} \FunctionTok{interval}\NormalTok{(independence\_date, today)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{years\_independent =} \FunctionTok{as.numeric}\NormalTok{(independent\_since, }\StringTok{"years"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(colonial\_power) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{last\_colony\_independent\_for =} \FunctionTok{min}\NormalTok{(years\_independent, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 4 x 2
##   colonial_power last_colony_independent_for
##   <chr>                                <dbl>
## 1 Belgium                               60.3
## 2 France                                45.3
## 3 Portugal                              46.9
## 4 United Kingdom                        46.3
\end{verbatim}

\hypertarget{arithmetic-with-dates}{%
\paragraph{Arithmetic with dates}\label{arithmetic-with-dates}}

Adding or substracting days to dates is quite easy:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ymd}\NormalTok{(}\StringTok{"2018{-}12{-}31"}\NormalTok{) }\SpecialCharTok{+} \DecValTok{16}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "2019-01-16"
\end{verbatim}

It is also possible to be more explicit and use \texttt{days()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ymd}\NormalTok{(}\StringTok{"2018{-}12{-}31"}\NormalTok{) }\SpecialCharTok{+} \FunctionTok{days}\NormalTok{(}\DecValTok{16}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "2019-01-16"
\end{verbatim}

To add years, you can use \texttt{years()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ymd}\NormalTok{(}\StringTok{"2018{-}12{-}31"}\NormalTok{) }\SpecialCharTok{+} \FunctionTok{years}\NormalTok{(}\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "2019-12-31"
\end{verbatim}

But you have to be careful with leap years:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ymd}\NormalTok{(}\StringTok{"2016{-}02{-}29"}\NormalTok{) }\SpecialCharTok{+} \FunctionTok{years}\NormalTok{(}\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] NA
\end{verbatim}

Because 2017 is not a leap year, the above computation returns \texttt{NA}. The same goes for months with
a different number of days:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ymd}\NormalTok{(}\StringTok{"2018{-}12{-}31"}\NormalTok{) }\SpecialCharTok{+} \FunctionTok{months}\NormalTok{(}\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] NA
\end{verbatim}

The way to solve these issues is to use the special \texttt{\%m+\%} infix operator:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ymd}\NormalTok{(}\StringTok{"2016{-}02{-}29"}\NormalTok{) }\SpecialCharTok{\%m+\%} \FunctionTok{years}\NormalTok{(}\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "2017-02-28"
\end{verbatim}

and for months:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ymd}\NormalTok{(}\StringTok{"2018{-}12{-}31"}\NormalTok{) }\SpecialCharTok{\%m+\%} \FunctionTok{months}\NormalTok{(}\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "2019-02-28"
\end{verbatim}

\texttt{\{lubridate\}} contains many more functions. If you often work with dates, duration or interval
data, \texttt{\{lubridate\}} is a package that you have to add to your toolbox.

\hypertarget{manipulate-strings-with-stringr}{%
\subsubsection{\texorpdfstring{Manipulate strings with \texttt{\{stringr\}}}{Manipulate strings with \{stringr\}}}\label{manipulate-strings-with-stringr}}

\texttt{\{stringr\}} contains functions to manipulate strings. In Chapter 10, I will teach you about regular
expressions, but the functions contained in \texttt{\{stringr\}} allow you to already do a lot of work on
strings, without needing to be a regular expression expert.

I will discuss the most common string operations: detecting, locating, matching, searching and
replacing, and exctracting/removing strings.

To introduce these operations, let us use an ALTO file of an issue of \emph{The Winchester News} from
October 31, 1910, which you can find on this
\href{https://gist.githubusercontent.com/b-rodrigues/5139560e7d0f2ecebe5da1df3629e015/raw/e3031d894ffb97217ddbad1ade1b307c9937d2c8/gistfile1.txt}{link} (to see
how the newspaper looked like,
\href{https://chroniclingamerica.loc.gov/lccn/sn86069133/1910-10-31/ed-1/seq-1/}{click here}). I re-hosted
the file on a public gist for archiving purposes. While working on the book, the original site went
down several times\ldots{}

ALTO is an XML schema for the description of text OCR and layout information of pages for digitzed
material, such as newspapers (source: \href{https://en.wikipedia.org/wiki/ALTO_(XML)}{ALTO Wikipedia page}).
For more details, you can read my
\href{https://www.brodrigues.co/blog/2019-01-13-newspapers_mets_alto/}{blogpost}
on the matter, but for our current purposes, it is enough to know that the file contains the text
of newspaper articles. The file looks like this:

\begin{verbatim}
<TextLine HEIGHT="138.0" WIDTH="2434.0" HPOS="4056.0" VPOS="5814.0">
<String STYLEREFS="ID7" HEIGHT="108.0" WIDTH="393.0" HPOS="4056.0" VPOS="5838.0" CONTENT="timore" WC="0.82539684">
<ALTERNATIVE>timole</ALTERNATIVE>
<ALTERNATIVE>tlnldre</ALTERNATIVE>
<ALTERNATIVE>timor</ALTERNATIVE>
<ALTERNATIVE>insole</ALTERNATIVE>
<ALTERNATIVE>landed</ALTERNATIVE>
</String>
<SP WIDTH="74.0" HPOS="4449.0" VPOS="5838.0"/>
<String STYLEREFS="ID7" HEIGHT="105.0" WIDTH="432.0" HPOS="4524.0" VPOS="5847.0" CONTENT="market" WC="0.95238096"/>
<SP WIDTH="116.0" HPOS="4956.0" VPOS="5847.0"/>
<String STYLEREFS="ID7" HEIGHT="69.0" WIDTH="138.0" HPOS="5073.0" VPOS="5883.0" CONTENT="as" WC="0.96825397"/>
<SP WIDTH="74.0" HPOS="5211.0" VPOS="5883.0"/>
<String STYLEREFS="ID7" HEIGHT="69.0" WIDTH="285.0" HPOS="5286.0" VPOS="5877.0" CONTENT="were" WC="1.0">
<ALTERNATIVE>verc</ALTERNATIVE>
<ALTERNATIVE>veer</ALTERNATIVE>
</String>
<SP WIDTH="68.0" HPOS="5571.0" VPOS="5877.0"/>
<String STYLEREFS="ID7" HEIGHT="111.0" WIDTH="147.0" HPOS="5640.0" VPOS="5838.0" CONTENT="all" WC="1.0"/>
<SP WIDTH="83.0" HPOS="5787.0" VPOS="5838.0"/>
<String STYLEREFS="ID7" HEIGHT="111.0" WIDTH="183.0" HPOS="5871.0" VPOS="5835.0" CONTENT="the" WC="0.95238096">
<ALTERNATIVE>tll</ALTERNATIVE>
<ALTERNATIVE>Cu</ALTERNATIVE>
<ALTERNATIVE>tall</ALTERNATIVE>
</String>
<SP WIDTH="75.0" HPOS="6054.0" VPOS="5835.0"/>
<String STYLEREFS="ID3" HEIGHT="132.0" WIDTH="351.0" HPOS="6129.0" VPOS="5814.0" CONTENT="cattle" WC="0.95238096"/>
</TextLine>
\end{verbatim}

We are interested in the strings after \texttt{CONTENT=}. We are going to use functions from the \texttt{\{stringr\}}
package to get the strings after \texttt{CONTENT=}. In Chapter 10, we are going to explore this file
again, but using complex regular expressions to get all the content in one go.

\hypertarget{getting-text-data-into-rstudio}{%
\paragraph{Getting text data into Rstudio}\label{getting-text-data-into-rstudio}}

First of all, let us read in the file:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester }\OtherTok{\textless{}{-}} \FunctionTok{read\_lines}\NormalTok{(}\StringTok{"https://gist.githubusercontent.com/b{-}rodrigues/5139560e7d0f2ecebe5da1df3629e015/raw/e3031d894ffb97217ddbad1ade1b307c9937d2c8/gistfile1.txt"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Even though the file is an XML file, I still read it in using \texttt{read\_lines()} and not \texttt{read\_xml()}
from the \texttt{\{xml2\}} package. This is for the purposes of the current exercise, and also because I
always have trouble with XML files, and prefer to treat them as simple text files, and use regular
expressions to get what I need.

Now that the ALTO file is read in and saved in the \texttt{winchester} variable, you might want to print
the whole thing in the console. Before that, take a look at the structure:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{str}\NormalTok{(winchester)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  chr [1:43] "" ...
\end{verbatim}

So the \texttt{winchester} variable is a character atomic vector with 43 elements. So first, we need to
understand what these elements are. Let's start with the first one:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester[}\DecValTok{1}\NormalTok{]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] ""
\end{verbatim}

Ok, so it seems like the first element is part of the header of the file. What about the second one?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester[}\DecValTok{2}\NormalTok{]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"><base href=\"https://chroniclingamerica.loc.gov/lccn/sn86069133/1910-10-31/ed-1/seq-1/ocr.xml\"><style>body{margin-left:0;margin-right:0;margin-top:0}#bN015htcoyT__google-cache-hdr{background:#f5f5f5;font:13px arial,sans-serif;text-align:left;color:#202020;border:0;margin:0;border-bottom:1px solid #cecece;line-height:16px;padding:16px 28px 24px 28px}#bN015htcoyT__google-cache-hdr *{display:inline;font:inherit;text-align:inherit;color:inherit;line-height:inherit;background:none;border:0;margin:0;padding:0;letter-spacing:0}#bN015htcoyT__google-cache-hdr a{text-decoration:none;color:#1a0dab}#bN015htcoyT__google-cache-hdr a:hover{text-decoration:underline}#bN015htcoyT__google-cache-hdr a:visited{color:#609}#bN015htcoyT__google-cache-hdr div{display:block;margin-top:4px}#bN015htcoyT__google-cache-hdr b{font-weight:bold;display:inline-block;direction:ltr}</style><div id=\"bN015htcoyT__google-cache-hdr\"><div><span>This is Google's cache of <a href=\"https://chroniclingamerica.loc.gov/lccn/sn86069133/1910-10-31/ed-1/seq-1/ocr.xml\">https://chroniclingamerica.loc.gov/lccn/sn86069133/1910-10-31/ed-1/seq-1/ocr.xml</a>.</span>&nbsp;<span>It is a snapshot of the page as it appeared on 21 Jan 2019 05:18:18 GMT.</span>&nbsp;<span>The <a href=\"https://chroniclingamerica.loc.gov/lccn/sn86069133/1910-10-31/ed-1/seq-1/ocr.xml\">current page</a> could have changed in the meantime.</span>&nbsp;<a href=\"http://support.google.com/websearch/bin/answer.py?hl=en&amp;p=cached&amp;answer=1687222\"><span>Learn more</span>.</a></div><div><span style=\"display:inline-block;margin-top:8px;margin-right:104px;white-space:nowrap\"><span style=\"margin-right:28px\"><span style=\"font-weight:bold\">Full version</span></span><span style=\"margin-right:28px\"><a href=\"http://webcache.googleusercontent.com/search?q=cache:2BVPV8QGj3oJ:https://chroniclingamerica.loc.gov/lccn/sn86069133/1910-10-31/ed-1/seq-1/ocr.xml&amp;hl=en&amp;gl=lu&amp;strip=1&amp;vwsrc=0\"><span>Text-only version</span></a></span><span style=\"margin-right:28px\"><a href=\"http://webcache.googleusercontent.com/search?q=cache:2BVPV8QGj3oJ:https://chroniclingamerica.loc.gov/lccn/sn86069133/1910-10-31/ed-1/seq-1/ocr.xml&amp;hl=en&amp;gl=lu&amp;strip=0&amp;vwsrc=1\"><span>View source</span></a></span></span></div><span style=\"display:inline-block;margin-top:8px;color:#717171\"><span>Tip: To quickly find your search term on this page, press <b>Ctrl+F</b> or <b>⌘-F</b> (Mac) and use the find bar.</span></span></div><div style=\"position:relative;\"><?xml version=\"1.0\" encoding=\"UTF-8\"?>"
\end{verbatim}

Same. So where is the content? The file is very large, so if you print it in the console, it will
take quite some time to print, and you will not really be able to make out anything. The best
way would be to try to detect the string \texttt{CONTENT} and work from there.

\hypertarget{detecting-getting-the-position-and-locating-strings}{%
\paragraph{Detecting, getting the position and locating strings}\label{detecting-getting-the-position-and-locating-strings}}

When confronted to an atomic vector of strings, you might want to know inside which elements you
can find certain strings. For example, to know which elements of \texttt{winchester} contain the string
\texttt{CONTENT}, use \texttt{str\_detect()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_detect}\NormalTok{(}\StringTok{"CONTENT"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [13] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [25] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [37] FALSE FALSE FALSE FALSE FALSE FALSE  TRUE
\end{verbatim}

This returns a boolean atomic vector of the same length as \texttt{winchester}. If the string \texttt{CONTENT} is
nowhere to be found, the result will equal \texttt{FALSE}, if not it will equal \texttt{TRUE}. Here it is easy to
see that the last element contains the string \texttt{CONTENT}. But what if instead of having 43 elements,
the vector had 24192 elements? And hundreds would contain the string \texttt{CONTENT}? It would be easier
to instead have the indices of the vector where one can find the word \texttt{CONTENT}. This is possible
with \texttt{str\_which()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_which}\NormalTok{(}\StringTok{"CONTENT"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 43
\end{verbatim}

Here, the result is 43, meaning that the 43rd element of \texttt{winchester} contains the string \texttt{CONTENT}
somewhere. If we need more precision, we can use \texttt{str\_locate()} and \texttt{str\_locate\_all()}. To explain
how both these functions work, let's create a very small example:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\StringTok{"aristotle"}\NormalTok{, }\StringTok{"plato"}\NormalTok{, }\StringTok{"epictetus"}\NormalTok{, }\StringTok{"seneca the younger"}\NormalTok{, }\StringTok{"epicurus"}\NormalTok{, }\StringTok{"marcus aurelius"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Now suppose I am interested in philosophers whose name ends in \texttt{us}. Let us use \texttt{str\_locate()} first:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_locate}\NormalTok{(}\StringTok{"us"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      start end
## [1,]    NA  NA
## [2,]    NA  NA
## [3,]     8   9
## [4,]    NA  NA
## [5,]     7   8
## [6,]     5   6
\end{verbatim}

You can interpret the result as follows: in the rows, the index of the vector where the
string \texttt{us} is found. So the 3rd, 5th and 6th philosopher have \texttt{us} somewhere in their name.
The result also has two columns: \texttt{start} and \texttt{end}. These give the position of the string. So the
string \texttt{us} can be found starting at position 8 of the 3rd element of the vector, and ends at position
9. Same goes for the other philisophers. However, consider Marcus Aurelius. He has two names, both
ending with \texttt{us}. However, \texttt{str\_locate()} only shows the position of the \texttt{us} in \texttt{Marcus}.

To get both \texttt{us} strings, you need to use \texttt{str\_locate\_all()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_locate\_all}\NormalTok{(}\StringTok{"us"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
##      start end
## 
## [[2]]
##      start end
## 
## [[3]]
##      start end
## [1,]     8   9
## 
## [[4]]
##      start end
## 
## [[5]]
##      start end
## [1,]     7   8
## 
## [[6]]
##      start end
## [1,]     5   6
## [2,]    14  15
\end{verbatim}

Now we get the position of the two \texttt{us} in Marcus Aurelius. Doing this on the \texttt{winchester} vector
will give use the position of the \texttt{CONTENT} string, but this is not really important right now. What
matters is that you know how \texttt{str\_locate()} and \texttt{str\_locate\_all()} work.

So now that we know what interests us in the 43nd element of \texttt{winchester}, let's take a closer
look at it:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester[}\DecValTok{43}\NormalTok{]}
\end{Highlighting}
\end{Shaded}

As you can see, it's a mess:

\begin{verbatim}
<TextLine HEIGHT=\"126.0\" WIDTH=\"1731.0\" HPOS=\"17160.0\" VPOS=\"21252.0\"><String HEIGHT=\"114.0\" WIDTH=\"354.0\" HPOS=\"17160.0\" VPOS=\"21264.0\" CONTENT=\"0tV\" WC=\"0.8095238\"/><SP WIDTH=\"131.0\" HPOS=\"17514.0\" VPOS=\"21264.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"111.0\" WIDTH=\"474.0\" HPOS=\"17646.0\" VPOS=\"21258.0\" CONTENT=\"BATES\" WC=\"1.0\"/><SP WIDTH=\"140.0\" HPOS=\"18120.0\" VPOS=\"21258.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"114.0\" WIDTH=\"630.0\" HPOS=\"18261.0\" VPOS=\"21252.0\" CONTENT=\"President\" WC=\"1.0\"><ALTERNATIVE>Prcideht</ALTERNATIVE><ALTERNATIVE>Pride</ALTERNATIVE></String></TextLine><TextLine HEIGHT=\"153.0\" WIDTH=\"1689.0\" HPOS=\"17145.0\" VPOS=\"21417.0\"><String STYLEREFS=\"ID7\" HEIGHT=\"105.0\" WIDTH=\"258.0\" HPOS=\"17145.0\" VPOS=\"21439.0\" CONTENT=\"WM\" WC=\"0.82539684\"><TextLine HEIGHT=\"120.0\" WIDTH=\"2211.0\" HPOS=\"16788.0\" VPOS=\"21870.0\"><String STYLEREFS=\"ID7\" HEIGHT=\"96.0\" WIDTH=\"102.0\" HPOS=\"16788.0\" VPOS=\"21894.0\" CONTENT=\"It\" WC=\"1.0\"/><SP WIDTH=\"72.0\" HPOS=\"16890.0\" VPOS=\"21894.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"96.0\" WIDTH=\"93.0\" HPOS=\"16962.0\" VPOS=\"21885.0\" CONTENT=\"is\" WC=\"1.0\"/><SP WIDTH=\"80.0\" HPOS=\"17055.0\" VPOS=\"21885.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"102.0\" WIDTH=\"417.0\" HPOS=\"17136.0\" VPOS=\"21879.0\" CONTENT=\"seldom\" WC=\"1.0\"/><SP WIDTH=\"80.0\" HPOS=\"17553.0\" VPOS=\"21879.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"96.0\" WIDTH=\"267.0\" HPOS=\"17634.0\" VPOS=\"21873.0\" CONTENT=\"hard\" WC=\"1.0\"/><SP WIDTH=\"81.0\" HPOS=\"17901.0\" VPOS=\"21873.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"87.0\" WIDTH=\"111.0\" HPOS=\"17982.0\" VPOS=\"21879.0\" CONTENT=\"to\" WC=\"1.0\"/><SP WIDTH=\"81.0\" HPOS=\"18093.0\" VPOS=\"21879.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"96.0\" WIDTH=\"219.0\" HPOS=\"18174.0\" VPOS=\"21870.0\" CONTENT=\"find\" WC=\"1.0\"/><SP WIDTH=\"77.0\" HPOS=\"18393.0\" VPOS=\"21870.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"69.0\" WIDTH=\"66.0\" HPOS=\"18471.0\" VPOS=\"21894.0\" CONTENT=\"a\" WC=\"1.0\"/><SP WIDTH=\"77.0\" HPOS=\"18537.0\" VPOS=\"21894.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"78.0\" WIDTH=\"384.0\" HPOS=\"18615.0\" VPOS=\"21888.0\" CONTENT=\"succes\" WC=\"0.82539684\"><ALTERNATIVE>success</ALTERNATIVE></String></TextLine><TextLine HEIGHT=\"126.0\" WIDTH=\"2316.0\" HPOS=\"16662.0\" VPOS=\"22008.0\"><String STYLEREFS=\"ID7\" HEIGHT=\"75.0\" WIDTH=\"183.0\" HPOS=\"16662.0\" VPOS=\"22059.0\" CONTENT=\"sor\" WC=\"1.0\"><ALTERNATIVE>soar</ALTERNATIVE></String><SP WIDTH=\"72.0\" HPOS=\"16845.0\" VPOS=\"22059.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"90.0\" WIDTH=\"168.0\" HPOS=\"16917.0\" VPOS=\"22035.0\" CONTENT=\"for\" WC=\"1.0\"/><SP WIDTH=\"72.0\" HPOS=\"17085.0\" VPOS=\"22035.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"69.0\" WIDTH=\"267.0\" HPOS=\"17157.0\" VPOS=\"22050.0\" CONTENT=\"even\" WC=\"1.0\"><ALTERNATIVE>cen</ALTERNATIVE><ALTERNATIVE>cent</ALTERNATIVE></String><SP WIDTH=\"77.0\" HPOS=\"17434.0\" VPOS=\"22050.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"66.0\" WIDTH=\"63.0\" HPOS=\"17502.0\" VPOS=\"22044.0\"
\end{verbatim}

The file was imported without any newlines. So we need to insert them ourselves, by splitting the
string in a clever way.

\hypertarget{splitting-strings}{%
\paragraph{Splitting strings}\label{splitting-strings}}

There are two functions included in \texttt{\{stringr\}} to split strings, \texttt{str\_split()} and \texttt{str\_split\_fixed()}.
Let's go back to our ancient philosophers. Two of them, Seneca the Younger and Marcus Aurelius have
something else in common than both being Roman Stoic philosophers. Their names are composed of several
words. If we want to split their names at the space character, we can use \texttt{str\_split()} like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_split}\NormalTok{(}\StringTok{" "}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] "aristotle"
## 
## [[2]]
## [1] "plato"
## 
## [[3]]
## [1] "epictetus"
## 
## [[4]]
## [1] "seneca"  "the"     "younger"
## 
## [[5]]
## [1] "epicurus"
## 
## [[6]]
## [1] "marcus"   "aurelius"
\end{verbatim}

\texttt{str\_split()} also has a \texttt{simplify\ =\ TRUE} option:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_split}\NormalTok{(}\StringTok{" "}\NormalTok{, }\AttributeTok{simplify =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      [,1]        [,2]       [,3]     
## [1,] "aristotle" ""         ""       
## [2,] "plato"     ""         ""       
## [3,] "epictetus" ""         ""       
## [4,] "seneca"    "the"      "younger"
## [5,] "epicurus"  ""         ""       
## [6,] "marcus"    "aurelius" ""
\end{verbatim}

This time, the returned object is a matrix.

What about \texttt{str\_split\_fixed()}? The difference is that here you can specify the number of pieces
to return. For example, you could consider the name ``Aurelius'' to be the middle name of Marcus Aurelius,
and the ``the younger'' to be the middle name of Seneca the younger. This means that you would want
to split the name only at the first space character, and not at all of them. This is easily achieved
with \texttt{str\_split\_fixed()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_split\_fixed}\NormalTok{(}\StringTok{" "}\NormalTok{, }\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      [,1]        [,2]         
## [1,] "aristotle" ""           
## [2,] "plato"     ""           
## [3,] "epictetus" ""           
## [4,] "seneca"    "the younger"
## [5,] "epicurus"  ""           
## [6,] "marcus"    "aurelius"
\end{verbatim}

This gives the expected result.

So how does this help in our case? Well, if you look at how the ALTO file looks like, at the beginning
of this section, you will notice that every line ends with the ``\textgreater{}'' character. So let's split at
that character!

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_text }\OtherTok{\textless{}{-}}\NormalTok{ winchester[}\DecValTok{43}\NormalTok{] }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_split}\NormalTok{(}\StringTok{"\textgreater{}"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Let's take a closer look at \texttt{winchester\_text}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{str}\NormalTok{(winchester\_text)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 1
##  $ : chr [1:19706] "</processingStepSettings" "<processingSoftware" "<softwareCreator" "iArchives</softwareCreator" ...
\end{verbatim}

So this is a list of length one, and the first, and only, element of that list is an atomic vector
with 19706 elements. Since this is a list of only one element, we can simplify it by saving the
atomic vector in a variable:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_text }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_text[[}\DecValTok{1}\NormalTok{]]}
\end{Highlighting}
\end{Shaded}

Let's now look at some lines:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_text[}\DecValTok{1232}\SpecialCharTok{:}\DecValTok{1245}\NormalTok{]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] "<SP WIDTH=\"66.0\" HPOS=\"5763.0\" VPOS=\"9696.0\"/"                                                                         
##  [2] "<String STYLEREFS=\"ID7\" HEIGHT=\"108.0\" WIDTH=\"612.0\" HPOS=\"5829.0\" VPOS=\"9693.0\" CONTENT=\"Louisville\" WC=\"1.0\""
##  [3] "<ALTERNATIVE"                                                                                                                
##  [4] "Loniile</ALTERNATIVE"                                                                                                        
##  [5] "<ALTERNATIVE"                                                                                                                
##  [6] "Lenities</ALTERNATIVE"                                                                                                       
##  [7] "</String"                                                                                                                    
##  [8] "</TextLine"                                                                                                                  
##  [9] "<TextLine HEIGHT=\"150.0\" WIDTH=\"2520.0\" HPOS=\"4032.0\" VPOS=\"9849.0\""                                                 
## [10] "<String STYLEREFS=\"ID7\" HEIGHT=\"108.0\" WIDTH=\"510.0\" HPOS=\"4032.0\" VPOS=\"9861.0\" CONTENT=\"Tobacco\" WC=\"1.0\"/"  
## [11] "<SP WIDTH=\"113.0\" HPOS=\"4542.0\" VPOS=\"9861.0\"/"                                                                        
## [12] "<String STYLEREFS=\"ID7\" HEIGHT=\"105.0\" WIDTH=\"696.0\" HPOS=\"4656.0\" VPOS=\"9861.0\" CONTENT=\"Warehouse\" WC=\"1.0\"" 
## [13] "<ALTERNATIVE"                                                                                                                
## [14] "WHrchons</ALTERNATIVE"
\end{verbatim}

This now looks easier to handle. We can narrow it down to the lines that only contain the string
we are interested in, ``CONTENT''. First, let's get the indices:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{content\_winchester\_index }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_text }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_which}\NormalTok{(}\StringTok{"CONTENT"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

How many lines contain the string ``CONTENT''?

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{length}\NormalTok{(content\_winchester\_index)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 4462
\end{verbatim}

As you can see, this reduces the amount of data we have to work with. Let us save this is a new
variable:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{content\_winchester }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_text[content\_winchester\_index]}
\end{Highlighting}
\end{Shaded}

\hypertarget{matching-strings}{%
\paragraph{Matching strings}\label{matching-strings}}

Matching strings is useful, but only in combination with regular expressions. As stated at the
beginning of this section, we are going to learn about regular expressions in Chapter 10, but in
order to make this section useful, we are going to learn the easiest, but perhaps the most useful
regular expression: \texttt{.*}.

Let's go back to our ancient philosophers, and use \texttt{str\_match()} and see what happens. Let's match
the ``us'' string:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_match}\NormalTok{(}\StringTok{"us"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      [,1]
## [1,] NA  
## [2,] NA  
## [3,] "us"
## [4,] NA  
## [5,] "us"
## [6,] "us"
\end{verbatim}

Not very useful, but what about the regular expression \texttt{.*}? How could it help?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_match}\NormalTok{(}\StringTok{".*us"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      [,1]             
## [1,] NA               
## [2,] NA               
## [3,] "epictetus"      
## [4,] NA               
## [5,] "epicurus"       
## [6,] "marcus aurelius"
\end{verbatim}

That's already very interesting! So how does \texttt{.*} work? To understand, let's first start by using
\texttt{.} alone:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_match}\NormalTok{(}\StringTok{".us"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      [,1] 
## [1,] NA   
## [2,] NA   
## [3,] "tus"
## [4,] NA   
## [5,] "rus"
## [6,] "cus"
\end{verbatim}

This also matched whatever symbol comes just before the ``u'' from ``us''. What if we use two \texttt{.} instead?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_match}\NormalTok{(}\StringTok{"..us"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      [,1]  
## [1,] NA    
## [2,] NA    
## [3,] "etus"
## [4,] NA    
## [5,] "urus"
## [6,] "rcus"
\end{verbatim}

This time, we get the two symbols that immediately precede ``us''. Instead of continuing like this
we now use the \texttt{*}, which matches zero or more of \texttt{.}. So by combining \texttt{*} and \texttt{.}, we can match
any symbol repeatedly, until there is nothing more to match. Note that there is also \texttt{+}, which works
similarly to \texttt{*}, but it matches one or more symbols.

There is also a \texttt{str\_match\_all()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_match\_all}\NormalTok{(}\StringTok{".*us"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
##      [,1]
## 
## [[2]]
##      [,1]
## 
## [[3]]
##      [,1]       
## [1,] "epictetus"
## 
## [[4]]
##      [,1]
## 
## [[5]]
##      [,1]      
## [1,] "epicurus"
## 
## [[6]]
##      [,1]             
## [1,] "marcus aurelius"
\end{verbatim}

In this particular case it does not change the end result, but keep it in mind for cases like this one:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{c}\NormalTok{(}\StringTok{"haha"}\NormalTok{, }\StringTok{"huhu"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_match}\NormalTok{(}\StringTok{"ha"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      [,1]
## [1,] "ha"
## [2,] NA
\end{verbatim}

and:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{c}\NormalTok{(}\StringTok{"haha"}\NormalTok{, }\StringTok{"huhu"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_match\_all}\NormalTok{(}\StringTok{"ha"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
##      [,1]
## [1,] "ha"
## [2,] "ha"
## 
## [[2]]
##      [,1]
\end{verbatim}

What if we want to match names containing the letter ``t''? Easy:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{ancient\_philosophers }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_match}\NormalTok{(}\StringTok{".*t.*"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      [,1]                
## [1,] "aristotle"         
## [2,] "plato"             
## [3,] "epictetus"         
## [4,] "seneca the younger"
## [5,] NA                  
## [6,] NA
\end{verbatim}

So how does this help us with our historical newspaper? Let's try to get the strings that come
after ``CONTENT'':

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_text }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{str\_match}\NormalTok{(}\StringTok{"CONTENT.*"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Let's use our faithful \texttt{str()} function to take a look:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\SpecialCharTok{\%\textgreater{}\%}
\NormalTok{  str}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  chr [1:19706, 1] NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA ...
\end{verbatim}

Hum, there's a lot of \texttt{NA} values! This is because a lot of the lines from the file did not have the
string ``CONTENT'', so there is no match possible. Let's us remove all these \texttt{NA}s. Because the
result is a matrix, we cannot use the \texttt{filter()} function from \texttt{\{dplyr\}}. So we need to convert it
to a tibble first:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_content }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{as.tibble}\NormalTok{() }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(V1))}
\end{Highlighting}
\end{Shaded}

Because matrix columns do not have names, when a matrix gets converted into a tibble, the firt column
gets automatically called \texttt{V1}. This is why I filter on this column. Let's take a look at the data:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{head}\NormalTok{(winchester\_content)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 1
##   V1                                  
##   <chr>                               
## 1 "CONTENT=\"J\" WC=\"0.8095238\"/"   
## 2 "CONTENT=\"a\" WC=\"0.8095238\"/"   
## 3 "CONTENT=\"Ira\" WC=\"0.95238096\"/"
## 4 "CONTENT=\"mj\" WC=\"0.8095238\"/"  
## 5 "CONTENT=\"iI\" WC=\"0.8095238\"/"  
## 6 "CONTENT=\"tE1r\" WC=\"0.8095238\"/"
\end{verbatim}

\hypertarget{searching-and-replacing-strings}{%
\paragraph{Searching and replacing strings}\label{searching-and-replacing-strings}}

We are getting close to the final result. We still need to do some cleaning however. Since our data
is inside a nice tibble, we might as well stick with it. So let's first rename the column and
change all the strings to lowercase:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_content }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{content =} \FunctionTok{tolower}\NormalTok{(V1)) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{V1)}
\end{Highlighting}
\end{Shaded}

Let's take a look at the result:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{head}\NormalTok{(winchester\_content)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 1
##   content                             
##   <chr>                               
## 1 "content=\"j\" wc=\"0.8095238\"/"   
## 2 "content=\"a\" wc=\"0.8095238\"/"   
## 3 "content=\"ira\" wc=\"0.95238096\"/"
## 4 "content=\"mj\" wc=\"0.8095238\"/"  
## 5 "content=\"ii\" wc=\"0.8095238\"/"  
## 6 "content=\"te1r\" wc=\"0.8095238\"/"
\end{verbatim}

The second part of the string, ``wc=\ldots.'' is not really interesting. Let's search and replace this
with an empty string, using \texttt{str\_replace()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_content }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{content =} \FunctionTok{str\_replace}\NormalTok{(content, }\StringTok{"wc.*"}\NormalTok{, }\StringTok{""}\NormalTok{))}

\FunctionTok{head}\NormalTok{(winchester\_content)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 1
##   content            
##   <chr>              
## 1 "content=\"j\" "   
## 2 "content=\"a\" "   
## 3 "content=\"ira\" " 
## 4 "content=\"mj\" "  
## 5 "content=\"ii\" "  
## 6 "content=\"te1r\" "
\end{verbatim}

We need to use the regular expression from before to replace ``wc'' and every character that follows.
The same can be use to remove ``content='':

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_content }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{content =} \FunctionTok{str\_replace}\NormalTok{(content, }\StringTok{"content="}\NormalTok{, }\StringTok{""}\NormalTok{))}

\FunctionTok{head}\NormalTok{(winchester\_content)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 1
##   content    
##   <chr>      
## 1 "\"j\" "   
## 2 "\"a\" "   
## 3 "\"ira\" " 
## 4 "\"mj\" "  
## 5 "\"ii\" "  
## 6 "\"te1r\" "
\end{verbatim}

We are almost done, but some cleaning is still necessary:

\hypertarget{exctracting-or-removing-strings}{%
\paragraph{Exctracting or removing strings}\label{exctracting-or-removing-strings}}

Now, because I now the ALTO spec, I know how to find words that are split between two sentences:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{str\_detect}\NormalTok{(content, }\StringTok{"hyppart"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 64 x 1
##    content                                                               
##    <chr>                                                                 
##  1 "\"aver\" subs_type=\"hyppart1\" subs_content=\"average\" "           
##  2 "\"age\" subs_type=\"hyppart2\" subs_content=\"average\" "            
##  3 "\"considera\" subs_type=\"hyppart1\" subs_content=\"consideration\" "
##  4 "\"tion\" subs_type=\"hyppart2\" subs_content=\"consideration\" "     
##  5 "\"re\" subs_type=\"hyppart1\" subs_content=\"resigned\" "            
##  6 "\"signed\" subs_type=\"hyppart2\" subs_content=\"resigned\" "        
##  7 "\"install\" subs_type=\"hyppart1\" subs_content=\"installed\" "      
##  8 "\"ed\" subs_type=\"hyppart2\" subs_content=\"installed\" "           
##  9 "\"be\" subs_type=\"hyppart1\" subs_content=\"before\" "              
## 10 "\"fore\" subs_type=\"hyppart2\" subs_content=\"before\" "            
## # ... with 54 more rows
\end{verbatim}

For instance, the word ``average'' was split over two lines, the first part of the word, ``aver'' on the
first line, and the second part of the word, ``age'', on the second line. We want to keep what comes
after ``subs\_content''. Let's extract the word ``average'' using \texttt{str\_extract()}. However, because only
some words were split between two lines, we first need to detect where the string ``hyppart1'' is
located, and only then can we extract what comes after ``subs\_content''. Thus, we need to combine
\texttt{str\_detect()} to first detect the string, and then \texttt{str\_extract()} to extract what comes after
``subs\_content'':

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_content }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{content =} \FunctionTok{if\_else}\NormalTok{(}\FunctionTok{str\_detect}\NormalTok{(content, }\StringTok{"hyppart1"}\NormalTok{), }
                           \FunctionTok{str\_extract\_all}\NormalTok{(content, }\StringTok{"content=.*"}\NormalTok{, }\AttributeTok{simplify =} \ConstantTok{TRUE}\NormalTok{), }
\NormalTok{                           content))}
\end{Highlighting}
\end{Shaded}

Let's take a look at the result:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{str\_detect}\NormalTok{(content, }\StringTok{"content"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 64 x 1
##    content                                                          
##    <chr>                                                            
##  1 "content=\"average\" "                                           
##  2 "\"age\" subs_type=\"hyppart2\" subs_content=\"average\" "       
##  3 "content=\"consideration\" "                                     
##  4 "\"tion\" subs_type=\"hyppart2\" subs_content=\"consideration\" "
##  5 "content=\"resigned\" "                                          
##  6 "\"signed\" subs_type=\"hyppart2\" subs_content=\"resigned\" "   
##  7 "content=\"installed\" "                                         
##  8 "\"ed\" subs_type=\"hyppart2\" subs_content=\"installed\" "      
##  9 "content=\"before\" "                                            
## 10 "\"fore\" subs_type=\"hyppart2\" subs_content=\"before\" "       
## # ... with 54 more rows
\end{verbatim}

We still need to get rid of the string ``content='' and then of all the strings that contain ``hyppart2'',
which are not needed now:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_content }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{content =} \FunctionTok{str\_replace}\NormalTok{(content, }\StringTok{"content="}\NormalTok{, }\StringTok{""}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{content =} \FunctionTok{if\_else}\NormalTok{(}\FunctionTok{str\_detect}\NormalTok{(content, }\StringTok{"hyppart2"}\NormalTok{), }\ConstantTok{NA\_character\_}\NormalTok{, content))}

\FunctionTok{head}\NormalTok{(winchester\_content)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 1
##   content    
##   <chr>      
## 1 "\"j\" "   
## 2 "\"a\" "   
## 3 "\"ira\" " 
## 4 "\"mj\" "  
## 5 "\"ii\" "  
## 6 "\"te1r\" "
\end{verbatim}

Almost done! We only need to remove the \texttt{"} characters:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_content }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{content =} \FunctionTok{str\_replace\_all}\NormalTok{(content, }\StringTok{"}\SpecialCharTok{\textbackslash{}"}\StringTok{"}\NormalTok{, }\StringTok{""}\NormalTok{)) }

\FunctionTok{head}\NormalTok{(winchester\_content)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 1
##   content
##   <chr>  
## 1 "j "   
## 2 "a "   
## 3 "ira " 
## 4 "mj "  
## 5 "ii "  
## 6 "te1r "
\end{verbatim}

Let's remove space characters with \texttt{str\_trim()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{winchester\_content }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_content }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{content =} \FunctionTok{str\_trim}\NormalTok{(content)) }

\FunctionTok{head}\NormalTok{(winchester\_content)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 1
##   content
##   <chr>  
## 1 j      
## 2 a      
## 3 ira    
## 4 mj     
## 5 ii     
## 6 te1r
\end{verbatim}

To finish off this section, let's remove stop words (words that do not add any meaning to a sentence,
such as ``as'', ``and''\ldots) and words that are composed of less than 3 characters. You can find a dataset
with stopwords inside the \texttt{\{stopwords\}} package:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(stopwords)}

\FunctionTok{data}\NormalTok{(data\_stopwords\_stopwordsiso)}

\NormalTok{eng\_stopwords }\OtherTok{\textless{}{-}} \FunctionTok{tibble}\NormalTok{(}\StringTok{"content"} \OtherTok{=}\NormalTok{ data\_stopwords\_stopwordsiso}\SpecialCharTok{$}\NormalTok{en)}

\NormalTok{winchester\_content }\OtherTok{\textless{}{-}}\NormalTok{ winchester\_content }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{anti\_join}\NormalTok{(eng\_stopwords) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{filter}\NormalTok{(}\FunctionTok{nchar}\NormalTok{(content) }\SpecialCharTok{\textgreater{}} \DecValTok{3}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Joining, by = "content"
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{head}\NormalTok{(winchester\_content)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 1
##   content   
##   <chr>     
## 1 te1r      
## 2 jilas     
## 3 edition   
## 4 winchester
## 5 news      
## 6 injuries
\end{verbatim}

That's it for this section! You now know how to work with strings, but in Chapter 10 we are going
one step further by learning about regular expressions, which offer much more power.

\hypertarget{tidy-data-frames-with-tibble}{%
\subsubsection{\texorpdfstring{Tidy data frames with \texttt{\{tibble\}}}{Tidy data frames with \{tibble\}}}\label{tidy-data-frames-with-tibble}}

We have already seen and used several functions from the \texttt{\{tibble\}} package. Let's now go through
some more useful functions.

\hypertarget{creating-tibbles}{%
\paragraph{Creating tibbles}\label{creating-tibbles}}

\texttt{tribble()} makes it easy to create tibble row by row, manually:

It is also possible to create a tibble from a named list:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{as\_tibble}\NormalTok{(}\FunctionTok{list}\NormalTok{(}\StringTok{"combustion"} \OtherTok{=} \FunctionTok{c}\NormalTok{(}\StringTok{"oil"}\NormalTok{, }\StringTok{"diesel"}\NormalTok{, }\StringTok{"oil"}\NormalTok{, }\StringTok{"electric"}\NormalTok{),}
               \StringTok{"doors"} \OtherTok{=} \FunctionTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{, }\DecValTok{5}\NormalTok{, }\DecValTok{5}\NormalTok{, }\DecValTok{5}\NormalTok{)))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 4 x 2
##   combustion doors
##   <chr>      <dbl>
## 1 oil            3
## 2 diesel         5
## 3 oil            5
## 4 electric       5
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{enframe}\NormalTok{(}\FunctionTok{list}\NormalTok{(}\StringTok{"combustion"} \OtherTok{=} \FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{), }\StringTok{"doors"} \OtherTok{=} \FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{4}\NormalTok{), }\StringTok{"cylinders"} \OtherTok{=} \FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{8}\NormalTok{,}\DecValTok{9}\NormalTok{,}\DecValTok{10}\NormalTok{)))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 3 x 2
##   name       value    
##   <chr>      <list>   
## 1 combustion <dbl [2]>
## 2 doors      <dbl [3]>
## 3 cylinders  <dbl [4]>
\end{verbatim}

\hypertarget{list-columns}{%
\subsection{List-columns}\label{list-columns}}

To learn about list-columns, let's first focus on a single character of the \texttt{starwars} dataset:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(starwars)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(name }\SpecialCharTok{==} \StringTok{"Luke Skywalker"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{glimpse}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 1
## Columns: 14
## $ name       <chr> "Luke Skywalker"
## $ height     <int> 172
## $ mass       <dbl> 77
## $ hair_color <chr> "blond"
## $ skin_color <chr> "fair"
## $ eye_color  <chr> "blue"
## $ birth_year <dbl> 19
## $ sex        <chr> "male"
## $ gender     <chr> "masculine"
## $ homeworld  <chr> "Tatooine"
## $ species    <chr> "Human"
## $ films      <list> <"The Empire Strikes Back", "Revenge of the Sith", "Return ~
## $ vehicles   <list> <"Snowspeeder", "Imperial Speeder Bike">
## $ starships  <list> <"X-wing", "Imperial shuttle">
\end{verbatim}

We see that the columns \texttt{films}, \texttt{vehicles} and \texttt{starships} (at the bottom) are all lists, and in
the case of \texttt{films}, it lists all the films where Luke Skywalker has appeared. What if you want to
take a closer look at films where Luke Skywalker appeared?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(name }\SpecialCharTok{==} \StringTok{"Luke Skywalker"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{pull}\NormalTok{(films)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] "The Empire Strikes Back" "Revenge of the Sith"    
## [3] "Return of the Jedi"      "A New Hope"             
## [5] "The Force Awakens"
\end{verbatim}

\texttt{pull()} is a \texttt{\{dplyr\}} function that extract (pulls) the column you're interested in. It is quite
useful when you want to inspect a column. Instead of just looking at Luke Skywalker's films,
let's pull the complete \texttt{films} column instead:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{head}\NormalTok{() }\SpecialCharTok{\%\textgreater{}\%}  \CommentTok{\# let\textquotesingle{}s just look at the first six rows}
  \FunctionTok{pull}\NormalTok{(films) }
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] "The Empire Strikes Back" "Revenge of the Sith"    
## [3] "Return of the Jedi"      "A New Hope"             
## [5] "The Force Awakens"      
## 
## [[2]]
## [1] "The Empire Strikes Back" "Attack of the Clones"   
## [3] "The Phantom Menace"      "Revenge of the Sith"    
## [5] "Return of the Jedi"      "A New Hope"             
## 
## [[3]]
## [1] "The Empire Strikes Back" "Attack of the Clones"   
## [3] "The Phantom Menace"      "Revenge of the Sith"    
## [5] "Return of the Jedi"      "A New Hope"             
## [7] "The Force Awakens"      
## 
## [[4]]
## [1] "The Empire Strikes Back" "Revenge of the Sith"    
## [3] "Return of the Jedi"      "A New Hope"             
## 
## [[5]]
## [1] "The Empire Strikes Back" "Revenge of the Sith"    
## [3] "Return of the Jedi"      "A New Hope"             
## [5] "The Force Awakens"      
## 
## [[6]]
## [1] "Attack of the Clones" "Revenge of the Sith"  "A New Hope"
\end{verbatim}

Let's stop here a moment. As you see, the \texttt{films} column contains several items in it. How is it
possible that a single \emph{cell} contains more than one film? This is because what is actually
contained in the cell is not the seven films, as seven separate characters, but an atomic vector
that happens to have seven elements. But it is still only one vector. \emph{Zooming} in into the data
frame helps understand:

\includegraphics[width=6.47in]{assets/zoom_list_columns}

In the picture above we see three columns. The first two, \texttt{name} and \texttt{sex} are what you're used
to see, just one element defining the characters \texttt{name} and \texttt{sex} respectively. The last one
also contains only one element for each character, it just so happens to be a complete
vector of characters. Because what is inside the \emph{cells} of a list-column can be very different
things (as list can contain anything), you have to think a bit about it in order to extract
insights from such columns. List-columns may seem arcane, but they are extremely powerful
once you master them.

As an example, suppose we want to create a numerical variable which counts the number of movies
in which the characters have appeared. For this we need to compute the length of the list, or count
the number of elements this list has. Let's try with \texttt{length()} a base R function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(name }\SpecialCharTok{==} \StringTok{"Luke Skywalker"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{pull}\NormalTok{(films) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{length}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1
\end{verbatim}

This might be surprising, but remember that a list with only one element, has a length of 1:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{length}\NormalTok{(}
  \FunctionTok{list}\NormalTok{(words) }\CommentTok{\# this creates a list which one element. This element is a list of 980 words.}
\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1
\end{verbatim}

Even though \texttt{words} contain a vector of 980 words, if we put this very long vector inside the
first element of list, \texttt{length(list(words))} will this compute the length of the list. Let's
see what happens if we create a more complex list:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{numbers }\OtherTok{\textless{}{-}} \FunctionTok{seq}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{5}\NormalTok{)}

\FunctionTok{length}\NormalTok{(}
  \FunctionTok{list}\NormalTok{(words, }\CommentTok{\# this creates a list which one element. This element is a list of 980 words.}
\NormalTok{       numbers) }\CommentTok{\# numbers contains numbers 1 through 5}
\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 2
\end{verbatim}

\texttt{list(words,\ numbers)} is now a list of two elements, \texttt{words} and \texttt{numbers}. If we want to compute
the length of \texttt{words} and \texttt{numbers}, we need to learn about another powerful concept called
\emph{higher-order functions}. We are going to learn about this in greater detail in Chapter 8. For now,
let's use the fact that our list \texttt{films} is contained inside of a data frame, and use a convenience
function included in \texttt{\{dplyr\}} to handle situations like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\OtherTok{\textless{}{-}}\NormalTok{ starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{rowwise}\NormalTok{() }\SpecialCharTok{\%\textgreater{}\%} \CommentTok{\# \textless{}{-} Apply the next steps for each row individually}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{n\_films =} \FunctionTok{length}\NormalTok{(films))}
\end{Highlighting}
\end{Shaded}

\texttt{dplyr::rowwise()} is useful when working with list-columns because whatever instructions follow
get run on the single element contained in the list. The picture below illustrates this:

\includegraphics[width=5.5in]{assets/rowwise}

Let's take a look at the characters and the number of films they have appeared in:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(name, films, n\_films)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 87 x 3
## # Rowwise: 
##    name               films     n_films
##    <chr>              <list>      <int>
##  1 Luke Skywalker     <chr [5]>       5
##  2 C-3PO              <chr [6]>       6
##  3 R2-D2              <chr [7]>       7
##  4 Darth Vader        <chr [4]>       4
##  5 Leia Organa        <chr [5]>       5
##  6 Owen Lars          <chr [3]>       3
##  7 Beru Whitesun lars <chr [3]>       3
##  8 R5-D4              <chr [1]>       1
##  9 Biggs Darklighter  <chr [1]>       1
## 10 Obi-Wan Kenobi     <chr [6]>       6
## # ... with 77 more rows
\end{verbatim}

Now we can, for example, create a factor variable that groups characters by asking whether they appeared only in
1 movie, or more:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\OtherTok{\textless{}{-}}\NormalTok{ starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{more\_1 =} \FunctionTok{case\_when}\NormalTok{(n\_films }\SpecialCharTok{==} \DecValTok{1} \SpecialCharTok{\textasciitilde{}} \StringTok{"Exactly one movie"}\NormalTok{,}
\NormalTok{                            n\_films }\SpecialCharTok{\textgreater{}=} \DecValTok{1} \SpecialCharTok{\textasciitilde{}} \StringTok{"More than 1 movie"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

You can also create list-columns with your own datasets, by using \texttt{tidyr::nest()}. Remember the
fake \texttt{survey\_data} I created to illustrate \texttt{pivot\_longer()} and \texttt{pivot\_wider()}? Let's go back to that dataset
again:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data }\OtherTok{\textless{}{-}} \FunctionTok{tribble}\NormalTok{(}
  \SpecialCharTok{\textasciitilde{}}\NormalTok{id, }\SpecialCharTok{\textasciitilde{}}\NormalTok{variable, }\SpecialCharTok{\textasciitilde{}}\NormalTok{value,}
  \DecValTok{1}\NormalTok{, }\StringTok{"var1"}\NormalTok{, }\DecValTok{1}\NormalTok{,}
  \DecValTok{1}\NormalTok{, }\StringTok{"var2"}\NormalTok{, }\FloatTok{0.2}\NormalTok{,}
  \ConstantTok{NA}\NormalTok{, }\StringTok{"var3"}\NormalTok{, }\FloatTok{0.3}\NormalTok{,}
  \DecValTok{2}\NormalTok{, }\StringTok{"var1"}\NormalTok{, }\FloatTok{1.4}\NormalTok{,}
  \DecValTok{2}\NormalTok{, }\StringTok{"var2"}\NormalTok{, }\FloatTok{1.9}\NormalTok{,}
  \DecValTok{2}\NormalTok{, }\StringTok{"var3"}\NormalTok{, }\FloatTok{4.1}\NormalTok{,}
  \DecValTok{3}\NormalTok{, }\StringTok{"var1"}\NormalTok{, }\FloatTok{0.1}\NormalTok{,}
  \DecValTok{3}\NormalTok{, }\StringTok{"var2"}\NormalTok{, }\FloatTok{2.8}\NormalTok{,}
  \DecValTok{3}\NormalTok{, }\StringTok{"var3"}\NormalTok{, }\FloatTok{8.9}\NormalTok{,}
  \DecValTok{4}\NormalTok{, }\StringTok{"var1"}\NormalTok{, }\FloatTok{1.7}\NormalTok{,}
  \ConstantTok{NA}\NormalTok{, }\StringTok{"var2"}\NormalTok{, }\FloatTok{1.9}\NormalTok{,}
  \DecValTok{4}\NormalTok{, }\StringTok{"var3"}\NormalTok{, }\FloatTok{7.6}
\NormalTok{)}

\FunctionTok{print}\NormalTok{(survey\_data)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 3
##       id variable value
##    <dbl> <chr>    <dbl>
##  1     1 var1       1  
##  2     1 var2       0.2
##  3    NA var3       0.3
##  4     2 var1       1.4
##  5     2 var2       1.9
##  6     2 var3       4.1
##  7     3 var1       0.1
##  8     3 var2       2.8
##  9     3 var3       8.9
## 10     4 var1       1.7
## 11    NA var2       1.9
## 12     4 var3       7.6
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{nested\_data }\OtherTok{\textless{}{-}}\NormalTok{ survey\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(id) }\SpecialCharTok{\%\textgreater{}\%}  
  \FunctionTok{nest}\NormalTok{()}

\FunctionTok{glimpse}\NormalTok{(nested\_data)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 5
## Columns: 2
## Groups: id [5]
## $ id   <dbl> 1, NA, 2, 3, 4
## $ data <list> [<tbl_df[2 x 2]>], [<tbl_df[2 x 2]>], [<tbl_df[3 x 2]>], [<tbl_df~
\end{verbatim}

This creates a new tibble, with columns \texttt{id} and \texttt{data}. \texttt{data} is a list-column that contains
tibbles; each tibble is the \texttt{variable} and \texttt{value} for each individual:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{nested\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(id }\SpecialCharTok{==} \StringTok{"1"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{pull}\NormalTok{(data)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## # A tibble: 2 x 2
##   variable value
##   <chr>    <dbl>
## 1 var1       1  
## 2 var2       0.2
\end{verbatim}

As you can see, for individual 1, the column data contains a 2x2 tibble with columns \texttt{variable} and
\texttt{value}. Because \texttt{group\_by()} followed by \texttt{nest()} is so useful, there is a wrapper around these two functions
called \texttt{group\_nest()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{survey\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_nest}\NormalTok{(id)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 5 x 2
##      id               data
##   <dbl> <list<tibble[,2]>>
## 1     1            [2 x 2]
## 2     2            [3 x 2]
## 3     3            [3 x 2]
## 4     4            [2 x 2]
## 5    NA            [2 x 2]
\end{verbatim}

You might be wondering why this is useful, because this seems to introduce an unnecessary
layer of complexity. The usefulness of list-columns will become apparent in the next chapters,
where we are going to learn how to repeat actions over, say, individuals. So if you've reached
the end of this section and still didn't really grok list-columns, go take some fresh air and
come back to this section again later on.

\hypertarget{going-beyond-descriptive-statistics-and-data-manipulation}{%
\subsection{Going beyond descriptive statistics and data manipulation}\label{going-beyond-descriptive-statistics-and-data-manipulation}}

The \texttt{\{tidyverse\}} collection of packages can do much more than simply data manipulation and
descriptive statisics. You can use the principles we have covered and the functions you now know
to do much more. For instance, you can use a few \texttt{\{tidyverse\}} functions to do Monte Carlo simulations,
for example to estimate \(\pi\).

Draw the unit circle inside the unit square, the ratio of the area of the circle to the area of the
square will be \(\pi/4\). Then shot K arrows at the square; roughly \(K*\pi/4\) should have fallen
inside the circle. So if now you shoot N arrows at the square, and M fall inside the circle, you have
the following relationship \(M = N*\pi/4\). You can thus compute \(\pi\) like so: \(\pi = 4*M/N\).

The more arrows N you throw at the square, the better approximation of \(\pi\) you'll have. Let's
try to do this with a tidy Monte Carlo simulation. First, let's randomly pick some points inside
the unit square:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(tidyverse)}

\NormalTok{n }\OtherTok{\textless{}{-}} \DecValTok{5000}

\FunctionTok{set.seed}\NormalTok{(}\DecValTok{2019}\NormalTok{)}
\NormalTok{points }\OtherTok{\textless{}{-}} \FunctionTok{tibble}\NormalTok{(}\StringTok{"x"} \OtherTok{=} \FunctionTok{runif}\NormalTok{(n), }\StringTok{"y"} \OtherTok{=} \FunctionTok{runif}\NormalTok{(n))}
\end{Highlighting}
\end{Shaded}

Now, to know if a point is inside the unit circle, we need to check wether \(x^2 + y^2 < 1\). Let's
add a new column to the \texttt{points} tibble, called \texttt{inside} equal to 1 if the point is inside the
unit circle and 0 if not:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{points }\OtherTok{\textless{}{-}}\NormalTok{ points }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{inside =} \FunctionTok{map2\_dbl}\NormalTok{(}\AttributeTok{.x =}\NormalTok{ x, }\AttributeTok{.y =}\NormalTok{ y, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{ifelse}\NormalTok{(.x}\SpecialCharTok{**}\DecValTok{2} \SpecialCharTok{+}\NormalTok{ .y}\SpecialCharTok{**}\DecValTok{2} \SpecialCharTok{\textless{}} \DecValTok{1}\NormalTok{, }\DecValTok{1}\NormalTok{, }\DecValTok{0}\NormalTok{))) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{rowid\_to\_column}\NormalTok{(}\StringTok{"N"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Let's take a look at \texttt{points}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{points}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 5,000 x 4
##        N       x      y inside
##    <int>   <dbl>  <dbl>  <dbl>
##  1     1 0.770   0.984       0
##  2     2 0.713   0.0107      1
##  3     3 0.303   0.133       1
##  4     4 0.618   0.0378      1
##  5     5 0.0505  0.677       1
##  6     6 0.0432  0.0846      1
##  7     7 0.820   0.727       0
##  8     8 0.00961 0.0758      1
##  9     9 0.102   0.373       1
## 10    10 0.609   0.676       1
## # ... with 4,990 more rows
\end{verbatim}

Now, I can compute the estimation
of \(\pi\) at each row, by computing the cumulative sum of the 1's in the \texttt{inside} column and dividing
that by the current value of \texttt{N} column:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{points }\OtherTok{\textless{}{-}}\NormalTok{ points }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{estimate =} \DecValTok{4}\SpecialCharTok{*}\FunctionTok{cumsum}\NormalTok{(inside)}\SpecialCharTok{/}\NormalTok{N)}
\end{Highlighting}
\end{Shaded}

\texttt{cumsum(inside)} is the \texttt{M} from the formula. Now, we can finish by plotting the result:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(points) }\SpecialCharTok{+}
    \FunctionTok{geom\_line}\NormalTok{(}\FunctionTok{aes}\NormalTok{(}\AttributeTok{y =}\NormalTok{ estimate, }\AttributeTok{x =}\NormalTok{ N)) }\SpecialCharTok{+}
    \FunctionTok{geom\_hline}\NormalTok{(}\AttributeTok{yintercept =}\NormalTok{ pi)}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-414-1.pdf}

In the next chapter, we are going to learn all about \texttt{\{ggplot2\}}, the package I used in the lines
above to create this plot.

As the number of tries grows, the estimation of \(\pi\) gets better.

Using a data frame as a structure to hold our simulated points and the results makes it very easy
to avoid loops, and thus write code that is more concise and easier to follow.
If you studied a quantitative field in university, you might have done a similar exercise at the
time, very likely by defining a matrix to hold your points, and an empty vector to hold whether a
particular point was inside the unit circle. Then you wrote a loop to compute whether
a point was inside the unit circle, save this result in the before-defined empty vector and then
compute the estimation of \(\pi\). Again, I take this opportunity here to stress that there is nothing
wrong with this approach per se, but R is better suited for a workflow where lists or data frames
are the central objects and where the analyst operates over them with functional programming techniques.

\hypertarget{exercises-2}{%
\subsection{Exercises}\label{exercises-2}}

\hypertarget{exercise-1-2}{%
\subsubsection*{Exercise 1}\label{exercise-1-2}}
\addcontentsline{toc}{subsubsection}{Exercise 1}

\begin{itemize}
\tightlist
\item
  Combine \texttt{mutate()} with \texttt{across()} to exponentiate every column of type \texttt{double} of the \texttt{gasoline} dataset.
\end{itemize}

To obtain the \texttt{gasoline} dataset, run the following lines:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(Gasoline, }\AttributeTok{package =} \StringTok{"plm"}\NormalTok{)}

\NormalTok{gasoline }\OtherTok{\textless{}{-}} \FunctionTok{as\_tibble}\NormalTok{(Gasoline)}

\NormalTok{gasoline }\OtherTok{\textless{}{-}}\NormalTok{ gasoline }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{country =} \FunctionTok{tolower}\NormalTok{(country))}
\end{Highlighting}
\end{Shaded}

\begin{itemize}
\item
  Exponeniate columns starting with the character \texttt{"l"} of the \texttt{gasoline} dataset.
\item
  Convert all columns' classes into the character class.
\end{itemize}

\hypertarget{exercise-2-1}{%
\subsubsection*{Exercise 2}\label{exercise-2-1}}
\addcontentsline{toc}{subsubsection}{Exercise 2}

Load the \texttt{LaborSupply} dataset from the \texttt{\{Ecdat\}} package and answer the following questions:

\begin{itemize}
\tightlist
\item
  Compute the average annual hours worked by year (plus standard deviation)
\item
  What age group worked the most hours in the year 1982?
\item
  Create a variable, \texttt{n\_years} that equals the number of years an individual stays in the panel. Is the panel balanced?
\item
  Which are the individuals that do not have any kids during the whole period? Create a variable, \texttt{no\_kids}, that flags these individuals (1 = no kids, 0 = kids)
\item
  Using the \texttt{no\_kids} variable from before compute the average wage, standard deviation and number of observations in each group for the year 1980 (no kids group vs kids group).
\item
  Create the lagged logarithm of hours worked and wages. Remember that this is a panel.
\end{itemize}

\hypertarget{exercise-3-1}{%
\subsubsection*{Exercise 3}\label{exercise-3-1}}
\addcontentsline{toc}{subsubsection}{Exercise 3}

\begin{itemize}
\tightlist
\item
  What does the following code do? Copy and paste it in an R interpreter to find out!
\end{itemize}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{LaborSupply }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(id) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{starts\_with}\NormalTok{(}\StringTok{"l"}\NormalTok{), tibble}\SpecialCharTok{::}\FunctionTok{lst}\NormalTok{(lag, lead)))}
\end{Highlighting}
\end{Shaded}

\begin{itemize}
\tightlist
\item
  Using \texttt{summarise()} and \texttt{across()}, compute the mean, standard deviation and number of individuals of \texttt{lnhr} and \texttt{lnwg} for each individual.
\end{itemize}

\hypertarget{exercise-4-1}{%
\subsubsection*{Exercise 4}\label{exercise-4-1}}
\addcontentsline{toc}{subsubsection}{Exercise 4}

\begin{itemize}
\item
  In the dataset folder you downloaded at the beginning of the chapter, there is a folder called
  ``unemployment''. I used the data in the section about working with lists of datasets. Using
  \texttt{rio::import\_list()}, read the 4 datasets into R.
\item
  Using \texttt{map()}, map the \texttt{janitor::clean\_names()} function to each dataset (just like in the example
  in the section on working with lists of datasets). Then, still with \texttt{map()} and \texttt{mutate()} convert
  all commune names in the \texttt{commune} column with the function \texttt{tolower()}, in a new column called \texttt{lcommune}.
  This is not an easy exercise; so here are some hints:

  \begin{itemize}
  \tightlist
  \item
    Remember that \texttt{all\_datasets} is a list of datasets. Which function do you use when you want to map a function to each element of a list?
  \item
    Each element of \texttt{all\_datasets} are \texttt{data.frame} objects. Which function do you use to add a column to a \texttt{data.frame}?
  \item
    What symbol can you use to access a column of a \texttt{data.frame}?
  \end{itemize}
\end{itemize}

\hypertarget{graphs}{%
\section{Graphs}\label{graphs}}

By default, it is possible to make a lot of graphs with R without the need of any external
packages. However, in this chapter, we are going to learn how to make graphs using \texttt{\{ggplot2\}} which
is a very powerful package that produces amazing graphs. There is an entry cost to \texttt{\{ggplot2\}} as it
works in a very different way than what you would expect, especially if you know how to make plots
with the basic R functions already. But the resulting graphs are well worth the effort and once
you will know more about \texttt{\{ggplot2\}} you will see that in a lot of situations it is actually faster
and easier. Another advantage is that making plots with \texttt{\{ggplot2\}} is consistent, so you do not need
to learn anything specific to make, say, density plots. There are a lot of extensions to \texttt{\{ggplot2\}},
such as \texttt{\{ggridges\}} to create so-called ridge plots and \texttt{\{gganimate\}} to create animated plots. By
the end of this chapter you will know how to do basic plots with \texttt{\{ggplot2\}} and also how to use these
two extensions.

\hypertarget{resources}{%
\subsection{Resources}\label{resources}}

Before showing some examples and the general functionality of \texttt{\{ggplot2\}}, I list here some online
resources that I keep coming back to:

\begin{itemize}
\item
  \href{http://socviz.co/}{Data Visualization for Social Science}
\item
  \href{http://www.cookbook-r.com/Graphs/}{R Graphics Cookbook}
\item
  \href{http://www.r-graph-gallery.com/portfolio/ggplot2-package/}{R graph gallery}
\item
  \href{http://motioninsocial.com/tufte/}{Tufte in R}
\item
  \href{https://exts.ggplot2.tidyverse.org/}{ggplot2 extensions}
\item
  \href{https://jrnold.github.io/ggthemes/reference/index.html}{ggthemes function reference}
\item
  \href{https://raw.githubusercontent.com/rstudio/cheatsheets/main/data-visualization.pdf}{ggplot2 cheatsheet}
\end{itemize}

When I first started using \texttt{\{ggplot2\}}, I had a cookbook approach to it; I tried findinge examples
online that looked like what I needed, copy and paste the code and then adapted it to my case. The above resources
are the ones I consulted and keep consulting in these situations (I also go back to past code I've written, of
course). Don't hesitate to skim these resources for inspiration and to learn more about some
extensions to \texttt{\{ggplot2\}}. In the next subsections I am going to show you how to draw the most common
plots, as well as show you how to customize your plots with \texttt{\{ggthemes\}}, a package that contains pre-defined
themes for \texttt{\{ggplot2\}}.

\hypertarget{examples}{%
\subsection{Examples}\label{examples}}

I think that the best way to learn how to use \texttt{\{ggplot2\}} is to jump right into it. Let's first start with
barplots.

\hypertarget{barplots}{%
\subsubsection{Barplots}\label{barplots}}

To follow the examples below, load the following libraries:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(ggplot2)}
\FunctionTok{library}\NormalTok{(ggthemes)}
\end{Highlighting}
\end{Shaded}

\texttt{\{ggplot2\}} is an implementation of the \emph{Grammar of Graphics} by \citet{wilkinson2006}, but you don't need
to read the books to start using it. If we go back to the Star Wars data (contained in \texttt{dplyr}),
and wish to draw a barplot of the gender, the following lines are enough:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(starwars, }\FunctionTok{aes}\NormalTok{(gender)) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-425-1.pdf}

The first argument of the function is the data (called \texttt{starwars} in this example), and then the
function \texttt{aes()}. This function is where you list the variables that you want to map to the aesthetics
of the \emph{geoms} functions. On the second line, you see that we use the \texttt{geom\_bar()} function. This
function creates a barplot of \texttt{gender} variable.
You can get different kind of plots by using different \texttt{geom\_} functions. You can also provide the
\texttt{aes()} argument to the \texttt{geom\_*()} function:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(starwars) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{(}\FunctionTok{aes}\NormalTok{(gender))}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-426-1.pdf}

The difference between these two approaches is that when you specify the aesthetics in the \texttt{ggplot()} function,
all the \texttt{geom\_*()} functions that follow will inherited these aesthetics. This is useful if you want to avoid
writing the same code over and over again, but can be problematic if you need to specify different aesthetics
to different \texttt{geom\_*()} functions. This will become clear in a later example.

You can add options to your plots, for instance, you can change the coordinate system in your barplot:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(starwars, }\FunctionTok{aes}\NormalTok{(gender)) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{coord\_flip}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-427-1.pdf}

This is the basic recipe to create plots using \texttt{\{ggplot2\}}: start with a call to \texttt{ggplot()} where you specify
the data you want to plot, and optionally the aesthetics. Then, use the \texttt{geom\_*()} function you need; if you
did not specify the aesthetics in the call to the \texttt{ggplot()} function, do it here. Then, you can add different
options, such as changing the coordinate system, changing the theme, the colour palette used, changing the
position of the legend and much, much more. This chapter will only give you an overview of the capabilities
of \texttt{\{ggplot2\}}.

\hypertarget{scatter-plots}{%
\subsubsection{Scatter plots}\label{scatter-plots}}

Scatter plots are very useful, especially if you are trying to figure out the relationship between two variables.
For instance, let's make a scatter plot of height vs weight of Star Wars characters:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(starwars) }\SpecialCharTok{+}
  \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass))}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-428-1.pdf}

As you can see there is an outlier; a very heavy character! Star Wars fans already guessed it, it's Jabba the Hut.
To make the plot easier to read, let's remove this outlier:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{str\_detect}\NormalTok{(name, }\StringTok{"Jabba"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{() }\SpecialCharTok{+}
    \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass))}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-429-1.pdf}

There is a positive correlation between height and mass, by adding \texttt{geom\_smooth()} with the option \texttt{method\ =\ "lm"}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{str\_detect}\NormalTok{(name, }\StringTok{"Jabba"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass)) }\SpecialCharTok{+}
  \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass)) }\SpecialCharTok{+}
  \FunctionTok{geom\_smooth}\NormalTok{(}\AttributeTok{method =} \StringTok{"lm"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `geom_smooth()` using formula 'y ~ x'
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-430-1.pdf}

I've moved the \texttt{aes(height,\ mass)} up to the \texttt{ggplot()} function because both \texttt{geom\_point()} and \texttt{geom\_smooth()}
need them, and as explained in the begging of this section, the aesthetics listed in \texttt{ggplot()} get passed down
to the other geoms.

If you omit \texttt{method\ =\ "lm}, you get a non-parametric curve:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{str\_detect}\NormalTok{(name, }\StringTok{"Jabba"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass)) }\SpecialCharTok{+}
  \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass)) }\SpecialCharTok{+}
  \FunctionTok{geom\_smooth}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `geom_smooth()` using method = 'loess' and formula 'y ~ x'
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-431-1.pdf}

\hypertarget{density}{%
\subsubsection{Density}\label{density}}

Use \texttt{geom\_density()} to get density plots:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(starwars, }\FunctionTok{aes}\NormalTok{(height)) }\SpecialCharTok{+}
  \FunctionTok{geom\_density}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: Removed 6 rows containing non-finite values (stat_density).
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-432-1.pdf}

Let's go into more detail now; what if you would like to plot the densities for feminines and masculines
only (removing the droids from the data first)? This can be done by first filtering the data using
\texttt{dplyr} and then separating the dataset by gender:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(gender }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"feminine"}\NormalTok{, }\StringTok{"masculine"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

The above lines do the filtering; only keep gender if gender is in the vector \texttt{"feminine",\ "masculine"}.
This is much easier than having to write \texttt{gender\ ==\ "feminine"\ \textbar{}\ gender\ ==\ "masculine"}. Then, we pipe
this dataset to \texttt{ggplot}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(gender }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"feminine"}\NormalTok{, }\StringTok{"masculine"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, }\AttributeTok{fill =}\NormalTok{ gender)) }\SpecialCharTok{+}
  \FunctionTok{geom\_density}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: Removed 5 rows containing non-finite values (stat_density).
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-434-1.pdf}

Let's take a closer look to the \texttt{aes()} function: I've added \texttt{fill\ =\ gender}. This means that
there will be one density plot for each gender in the data, and each will be coloured accordingly.
This is where \texttt{\{ggplot2\}} might be confusing; there is no need to write explicitly (even if it is
possible) that you want the \emph{feminine} density to be red and the \emph{masculine} density to be blue. You just
map the variable \texttt{gender} to this particular aesthetic. You conclude the plot by adding
\texttt{geom\_density()} which is this case is the plot you want. We will see later how to change the
colours of your plot.

An alternative way to write this code is first to save the filtered data in a variable, and define
the aesthetics inside the \texttt{geom\_density()} function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{filtered\_data }\OtherTok{\textless{}{-}}\NormalTok{ starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(gender }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"feminine"}\NormalTok{, }\StringTok{"masculine"}\NormalTok{))}

\FunctionTok{ggplot}\NormalTok{(filtered\_data) }\SpecialCharTok{+}
  \FunctionTok{geom\_density}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, }\AttributeTok{fill =}\NormalTok{ gender))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: Removed 5 rows containing non-finite values (stat_density).
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-435-1.pdf}

\hypertarget{line-plots}{%
\subsubsection{Line plots}\label{line-plots}}

For the line plots, we are going to use official unemployment data (the same as in the previous
chapter, but with all the available years). Get it from
\href{https://github.com/b-rodrigues/modern_R/tree/master/datasets/unemployment/all}{here}
(downloaded from the website of the \href{https://lustat.statec.lu/vis?pg=0\&df\%5Bds\%5D=release\&df\%5Bid\%5D=DF_X026\&df\%5Bag\%5D=LU1\&df\%5Bvs\%5D=1.0\&pd=2021\%2C\&dq=..A\&ly\%5Brw\%5D=SPECIFICATION\&ly\%5Bcl\%5D=VARIABLE\&lc=en}{Luxembourguish national statistical institute}.

Let's plot the unemployment for the canton of Luxembourg only:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\OtherTok{\textless{}{-}} \FunctionTok{import}\NormalTok{(}\StringTok{"datasets/unemployment/all/unemployment\_lux\_all.csv"}\NormalTok{)}

\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{==} \StringTok{"Luxembourg"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(}\AttributeTok{x =}\NormalTok{ year, }\AttributeTok{y =}\NormalTok{ unemployment\_rate\_in\_percent, }\AttributeTok{group =} \DecValTok{1}\NormalTok{)) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-436-1.pdf}

Because line plots are 2D, you need to specify the y and x axes. There is also another option you
need to add, \texttt{group\ =\ 1}. This is to tell \texttt{aes()} that the dots have to be connected with a single
line. What if you want to plot more than one commune?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(}\AttributeTok{x =}\NormalTok{ year, }\AttributeTok{y =}\NormalTok{ unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-437-1.pdf}

This time, I've specified \texttt{group\ =\ division} which means that there has to be one line per as many
communes as in the variable \texttt{division}. I do the same for colours. I think the next example
illustrates how \texttt{\{ggplot2\}} is actually brilliant; if you need to add a third commune, there is no
need to specify anything else; no need to add anything to the legend, no need to specify a third
colour etc:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(}\AttributeTok{x =}\NormalTok{ year, }\AttributeTok{y =}\NormalTok{ unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-438-1.pdf}

The three communes get mapped to the colour aesthetic so whatever the number of communes, as long
as there are enough colours, the communes will each get mapped to one of these colours.

\hypertarget{facets}{%
\subsubsection{Facets}\label{facets}}

In some case you have a factor variable that separates the data you wish to plot into different
categories. If you want to have a plot per category you can use the \texttt{facet\_grid()} function.
Careful though, this function does not take a variable as an argument, but a formula, hence the \texttt{\textasciitilde{}}
symbol in the code below:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{human =} \FunctionTok{case\_when}\NormalTok{(species }\SpecialCharTok{==} \StringTok{"Human"} \SpecialCharTok{\textasciitilde{}} \StringTok{"Human"}\NormalTok{,}
\NormalTok{                           species }\SpecialCharTok{!=} \StringTok{"Human"} \SpecialCharTok{\textasciitilde{}} \StringTok{"Not Human"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(gender }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"feminine"}\NormalTok{, }\StringTok{"masculine"}\NormalTok{), }\SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(human)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, }\AttributeTok{fill =}\NormalTok{ gender)) }\SpecialCharTok{+}
  \FunctionTok{facet\_grid}\NormalTok{(. }\SpecialCharTok{\textasciitilde{}}\NormalTok{ human) }\SpecialCharTok{+} \CommentTok{\#\textless{}{-}{-}{-} this is a formula}
  \FunctionTok{geom\_density}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: Removed 5 rows containing non-finite values (stat_density).
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-439-1.pdf}

I first created a factor variable that specifies if a Star Wars character is human or not, and then
use it for facetting. By changing the formula, you change how the facetting is done:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{human =} \FunctionTok{case\_when}\NormalTok{(species }\SpecialCharTok{==} \StringTok{"Human"} \SpecialCharTok{\textasciitilde{}} \StringTok{"Human"}\NormalTok{,}
\NormalTok{                           species }\SpecialCharTok{!=} \StringTok{"Human"} \SpecialCharTok{\textasciitilde{}} \StringTok{"Not Human"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(gender }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"feminine"}\NormalTok{, }\StringTok{"masculine"}\NormalTok{), }\SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(human)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, }\AttributeTok{fill =}\NormalTok{ gender)) }\SpecialCharTok{+}
  \FunctionTok{facet\_grid}\NormalTok{(human }\SpecialCharTok{\textasciitilde{}}\NormalTok{ .) }\SpecialCharTok{+}
  \FunctionTok{geom\_density}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: Removed 5 rows containing non-finite values (stat_density).
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-440-1.pdf}

Recall the categorical variable \texttt{more\_1} that we computed in the previous chapter? Let's use it as
a faceting variable:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{rowwise}\NormalTok{() }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{n\_films =} \FunctionTok{length}\NormalTok{(films)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{more\_1 =} \FunctionTok{case\_when}\NormalTok{(n\_films }\SpecialCharTok{==} \DecValTok{1} \SpecialCharTok{\textasciitilde{}} \StringTok{"Exactly one movie"}\NormalTok{,}
\NormalTok{                            n\_films }\SpecialCharTok{!=} \DecValTok{1} \SpecialCharTok{\textasciitilde{}} \StringTok{"More than 1 movie"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{human =} \FunctionTok{case\_when}\NormalTok{(species }\SpecialCharTok{==} \StringTok{"Human"} \SpecialCharTok{\textasciitilde{}} \StringTok{"Human"}\NormalTok{,}
\NormalTok{                           species }\SpecialCharTok{!=} \StringTok{"Human"} \SpecialCharTok{\textasciitilde{}} \StringTok{"Not Human"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(gender }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"feminine"}\NormalTok{, }\StringTok{"masculine"}\NormalTok{), }\SpecialCharTok{!}\FunctionTok{is.na}\NormalTok{(human)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, }\AttributeTok{fill =}\NormalTok{ gender)) }\SpecialCharTok{+}
  \FunctionTok{facet\_grid}\NormalTok{(human }\SpecialCharTok{\textasciitilde{}}\NormalTok{ more\_1) }\SpecialCharTok{+}
  \FunctionTok{geom\_density}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: Removed 5 rows containing non-finite values (stat_density).
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-441-1.pdf}

\hypertarget{pie-charts}{%
\subsubsection{Pie Charts}\label{pie-charts}}

I am not a huge fan of pie charts, but sometimes this is what you have to do. So let's see how you
can create pie charts.
First, let's create a mock dataset with the function \texttt{tibble::tribble()} which allows you to create a
dataset line by line:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{test\_data }\OtherTok{\textless{}{-}} \FunctionTok{tribble}\NormalTok{(}
  \SpecialCharTok{\textasciitilde{}}\NormalTok{id, }\SpecialCharTok{\textasciitilde{}}\NormalTok{var1, }\SpecialCharTok{\textasciitilde{}}\NormalTok{var2,  }\SpecialCharTok{\textasciitilde{}}\NormalTok{var3, }\SpecialCharTok{\textasciitilde{}}\NormalTok{var4, }\SpecialCharTok{\textasciitilde{}}\NormalTok{var5,}
  \StringTok{"a"}\NormalTok{, }\FloatTok{26.5}\NormalTok{, }\DecValTok{38}\NormalTok{, }\DecValTok{30}\NormalTok{, }\DecValTok{32}\NormalTok{, }\DecValTok{34}\NormalTok{,}
  \StringTok{"b"}\NormalTok{, }\DecValTok{30}\NormalTok{, }\DecValTok{30}\NormalTok{, }\DecValTok{28}\NormalTok{, }\DecValTok{32}\NormalTok{, }\DecValTok{30}\NormalTok{,}
  \StringTok{"c"}\NormalTok{, }\DecValTok{34}\NormalTok{, }\DecValTok{32}\NormalTok{, }\DecValTok{30}\NormalTok{, }\DecValTok{28}\NormalTok{, }\FloatTok{26.5}
\NormalTok{)}
\end{Highlighting}
\end{Shaded}

This data is not in the right format though, which is wide. We need to have it in the long format
for it to work with \texttt{\{ggplot2\}}. For this, let's use \texttt{tidyr::gather()} as seen in the previous chapter:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{test\_data\_long }\OtherTok{=}\NormalTok{ test\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{gather}\NormalTok{(variable, value, }\FunctionTok{starts\_with}\NormalTok{(}\StringTok{"var"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

Now, let's plot this data, first by creating 3 bar plots:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(test\_data\_long) }\SpecialCharTok{+}
  \FunctionTok{facet\_wrap}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\NormalTok{id) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value, }\AttributeTok{fill =}\NormalTok{ variable), }\AttributeTok{stat =} \StringTok{"identity"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-444-1.pdf}

In the code above, I introduce a new option, called \texttt{stat\ =\ "identity"}. By default, \texttt{geom\_bar()} counts
the number of observations of each category that is plotted, which is a statistical transformation.
By adding \texttt{stat\ =\ "identity"}, I force the statistical transformation to be the identity function, and
thus plot the data as is.

To create the pie chart, first we need to compute the share of each \texttt{id} to \texttt{var1}, \texttt{var2}, etc\ldots{}

To do this, we first group by \texttt{id}, then compute the total. Then we use a new function \texttt{ungroup()}.
After using \texttt{ungroup()} all the computations are done on the whole dataset instead of by group, which
is what we need to compute the share:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{test\_data\_long }\OtherTok{\textless{}{-}}\NormalTok{ test\_data\_long }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(id) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{total =} \FunctionTok{sum}\NormalTok{(value)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ungroup}\NormalTok{() }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{share =}\NormalTok{ value}\SpecialCharTok{/}\NormalTok{total)}
\end{Highlighting}
\end{Shaded}

Let's take a look to see if this is what we wanted:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{print}\NormalTok{(test\_data\_long)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 15 x 5
##    id    variable value total share
##    <chr> <chr>    <dbl> <dbl> <dbl>
##  1 a     var1      26.5  160. 0.165
##  2 b     var1      30    150  0.2  
##  3 c     var1      34    150. 0.226
##  4 a     var2      38    160. 0.237
##  5 b     var2      30    150  0.2  
##  6 c     var2      32    150. 0.213
##  7 a     var3      30    160. 0.187
##  8 b     var3      28    150  0.187
##  9 c     var3      30    150. 0.199
## 10 a     var4      32    160. 0.199
## 11 b     var4      32    150  0.213
## 12 c     var4      28    150. 0.186
## 13 a     var5      34    160. 0.212
## 14 b     var5      30    150  0.2  
## 15 c     var5      26.5  150. 0.176
\end{verbatim}

If you didn't understand what \texttt{ungroup()} did, rerun the last few lines with it and inspect the
output.

To plot the pie chart, we create a barplot again, but specify polar coordinates:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(test\_data\_long) }\SpecialCharTok{+}
  \FunctionTok{facet\_wrap}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\NormalTok{id) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{(}\FunctionTok{aes}\NormalTok{(}\AttributeTok{y =}\NormalTok{ share, }\AttributeTok{x =} \StringTok{""}\NormalTok{, }\AttributeTok{fill =}\NormalTok{ variable), }\AttributeTok{stat =} \StringTok{"identity"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{coord\_polar}\NormalTok{(}\StringTok{"y"}\NormalTok{, }\AttributeTok{start =} \DecValTok{0}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-447-1.pdf}

As you can see, this typical pie chart is not very easy to read; compared to the barplots above it
is not easy to distinguish if \texttt{a} has a higher share than \texttt{b} or \texttt{c}. You can change the look of the
pie chart, for example by specifying \texttt{variable} as the \texttt{x}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(test\_data\_long) }\SpecialCharTok{+}
  \FunctionTok{facet\_wrap}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\NormalTok{id) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{(}\FunctionTok{aes}\NormalTok{(}\AttributeTok{y =}\NormalTok{ share, }\AttributeTok{x =}\NormalTok{ variable, }\AttributeTok{fill =}\NormalTok{ variable), }\AttributeTok{stat =} \StringTok{"identity"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{coord\_polar}\NormalTok{(}\StringTok{"x"}\NormalTok{, }\AttributeTok{start =} \DecValTok{0}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-448-1.pdf}

But as a general rule, avoid pie charts if possible. I find that pie charts are only interesting if
you need to show proportions that are hugely unequal, to really emphasize the difference between
said proportions.

\hypertarget{adding-text-to-plots}{%
\subsubsection{Adding text to plots}\label{adding-text-to-plots}}

Sometimes you might want to add some text to your plots. This is possible with \texttt{geom\_text()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(test\_data\_long) }\SpecialCharTok{+}
  \FunctionTok{facet\_wrap}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\NormalTok{id) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value, }\AttributeTok{fill =}\NormalTok{ variable), }\AttributeTok{stat =} \StringTok{"identity"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_text}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value }\SpecialCharTok{+} \FloatTok{1.5}\NormalTok{, }\AttributeTok{label =}\NormalTok{ value))}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-449-1.pdf}

You can put anything after \texttt{label\ =} but in general what you want are the values, so that's what
I put there. But you can also refine it, imagine the values are actually in euros:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(test\_data\_long) }\SpecialCharTok{+}
  \FunctionTok{facet\_wrap}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\NormalTok{id) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value, }\AttributeTok{fill =}\NormalTok{ variable), }\AttributeTok{stat =} \StringTok{"identity"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_text}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value }\SpecialCharTok{+} \FloatTok{1.5}\NormalTok{, }\AttributeTok{label =} \FunctionTok{paste}\NormalTok{(value, }\StringTok{"€"}\NormalTok{)))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-450-1.pdf}

You can also achieve something similar with \texttt{geom\_label()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(test\_data\_long) }\SpecialCharTok{+}
  \FunctionTok{facet\_wrap}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\NormalTok{id) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value, }\AttributeTok{fill =}\NormalTok{ variable), }\AttributeTok{stat =} \StringTok{"identity"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_label}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value }\SpecialCharTok{+} \FloatTok{1.5}\NormalTok{, }\AttributeTok{label =} \FunctionTok{paste}\NormalTok{(value, }\StringTok{"€"}\NormalTok{)))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '38 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '34 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '32 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '30 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '28 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <e2>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <82>
\end{verbatim}

\begin{verbatim}
## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## conversion failure on '26.5 €' in 'mbcsToSbcs': dot substituted for <ac>
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-451-1.pdf}

\hypertarget{customization}{%
\subsection{Customization}\label{customization}}

Every plot you've seen until now was made with the default look of \texttt{\{ggplot2\}}. If you want to change
the look, you can apply a theme, and a colour scheme. Let's take a look at themes first by using the
ones found in the package \texttt{ggthemes}. But first, let's learn how to change the names of the axes
and how to title a plot.

\hypertarget{changing-titles-axes-labels-options-mixing-geoms-and-changing-themes}{%
\subsubsection{Changing titles, axes labels, options, mixing geoms and changing themes}\label{changing-titles-axes-labels-options-mixing-geoms-and-changing-themes}}

The name of this subsection is quite long, but this is because everything is kind of linked. Let's
start by learning what the \texttt{labs()} function does. To change the title of the plot, and of the axes,
you need to pass the names to the \texttt{labs()} function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-452-1.pdf}

What if you want to make the lines thicker?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{(}\AttributeTok{size =} \DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-453-1.pdf}

Each \texttt{geom\_*()} function has its own options. Notice that the \texttt{size=2} argument is not inside
an \texttt{aes()} function. This is because I do not want to map a variable of the data to the size
of the line, in other words, I do not want to make the size of the line proportional to a certain
variable in the data. Recall the scatter plot we did earlier, where we showed that height and mass of
star wars characters increased together? Let's take this plot again, but make the size of the dots proportional
to the birth year of the character:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{str\_detect}\NormalTok{(name, }\StringTok{"Jabba"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{() }\SpecialCharTok{+}
    \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass, }\AttributeTok{size =}\NormalTok{ birth\_year))}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-454-1.pdf}

Making the size proportional to the birth year (the age would have been more informative) allows
us to see a third dimension. It is also possible to ``see'' a fourth dimension, the gender for instance,
by changing the colour of the dots:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{str\_detect}\NormalTok{(name, }\StringTok{"Jabba"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{() }\SpecialCharTok{+}
    \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass, }\AttributeTok{size =}\NormalTok{ birth\_year, }\AttributeTok{colour =}\NormalTok{ gender))}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-455-1.pdf}

As I promised above, we are now going to learn how to add a regression line to this scatter plot:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{str\_detect}\NormalTok{(name, }\StringTok{"Jabba"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{() }\SpecialCharTok{+}
    \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass, }\AttributeTok{size =}\NormalTok{ birth\_year, }\AttributeTok{colour =}\NormalTok{ gender)) }\SpecialCharTok{+}
    \FunctionTok{geom\_smooth}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass), }\AttributeTok{method  =} \StringTok{"lm"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `geom_smooth()` using formula 'y ~ x'
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-456-1.pdf}

\texttt{geom\_smooth()} adds a regression line, but only if you specify \texttt{method\ =\ "lm"} (``lm'' stands for
``linear model''). What happens if you remove this option?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{str\_detect}\NormalTok{(name, }\StringTok{"Jabba"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{() }\SpecialCharTok{+}
    \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass, }\AttributeTok{size =}\NormalTok{ birth\_year, }\AttributeTok{colour =}\NormalTok{ gender)) }\SpecialCharTok{+}
    \FunctionTok{geom\_smooth}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `geom_smooth()` using method = 'loess' and formula 'y ~ x'
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-457-1.pdf}

By default, \texttt{geom\_smooth()} does a non-parametric regression called LOESS (locally estimated scatterplot smoothing),
which is more flexible. It is also possible to have one regression line by gender:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{starwars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!}\FunctionTok{str\_detect}\NormalTok{(name, }\StringTok{"Jabba"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{() }\SpecialCharTok{+}
    \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass, }\AttributeTok{size =}\NormalTok{ birth\_year, }\AttributeTok{colour =}\NormalTok{ gender)) }\SpecialCharTok{+}
    \FunctionTok{geom\_smooth}\NormalTok{(}\FunctionTok{aes}\NormalTok{(height, mass, }\AttributeTok{colour =}\NormalTok{ gender))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## `geom_smooth()` using method = 'loess' and formula 'y ~ x'
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-458-1.pdf}

Because there are only a few observations for feminines and \texttt{NA}s the regression lines are not very informative,
but this was only an example to show you some options of \texttt{geom\_smooth()}.

Let's go back to the unemployment line plots. For now, let's keep the base \texttt{\{ggplot2\}} theme, but
modify it a bit. For example, the legend placement is actually a feature of the theme. This means
that if you want to change where the legend is placed you need to modify this feature from the
theme. This is done with the function \texttt{theme()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{(}\AttributeTok{legend.position =} \StringTok{"bottom"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-459-1.pdf}

What I also like to do is remove the title of the legend, because it is often superfluous:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{(}\AttributeTok{legend.position =} \StringTok{"bottom"}\NormalTok{, }\AttributeTok{legend.title =} \FunctionTok{element\_blank}\NormalTok{()) }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-460-1.pdf}

The legend title has to be an \texttt{element\_text} object.\texttt{element\_text} objects are used with \texttt{theme} to
specify how text should be displayed. \texttt{element\_blank()} draws nothing and assigns no space (not
even blank space). If you want to keep the legend title but change it, you need to use \texttt{element\_text()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{(}\AttributeTok{legend.position =} \StringTok{"bottom"}\NormalTok{, }\AttributeTok{legend.title =} \FunctionTok{element\_text}\NormalTok{(}\AttributeTok{colour =} \StringTok{"red"}\NormalTok{)) }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-461-1.pdf}

If you want to change the word ``division'' to something else, you can do so by providing the \texttt{colour} argument
to the \texttt{labs()} function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{(}\AttributeTok{legend.position =} \StringTok{"bottom"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{, }\AttributeTok{colour =} \StringTok{"Administrative division"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-462-1.pdf}

You could modify every feature of the theme like that, but there are built-in themes that you can use:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme\_minimal}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{(}\AttributeTok{legend.position =} \StringTok{"bottom"}\NormalTok{, }\AttributeTok{legend.title =} \FunctionTok{element\_blank}\NormalTok{()) }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-463-1.pdf}

For example in the code above, I have used \texttt{theme\_minimal()} which I like quite a lot. You can also
use themes from the \texttt{ggthemes} package, which even contains a STATA theme, if you like it:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme\_stata}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-464-1.pdf}

As you can see, \texttt{theme\_stata()} has the legend on the bottom by default, because this is how the
legend position is defined within the theme. However the legend title is still there. Let's remove
it:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme\_stata}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{(}\AttributeTok{legend.title =} \FunctionTok{element\_blank}\NormalTok{()) }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-465-1.pdf}

\texttt{ggthemes} even features an Excel 2003 theme (don't use it though):

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme\_excel}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-466-1.pdf}

You can create your own theme by using a simple theme, such as \texttt{theme\_minimal()} as a base
and then add your options. We are going to create one theme after we learn how to create our
own functions, in Chapter 7. Then, we are going to create a package to share this theme with
the world, and we are going to learn how to make packages in Chapter 9.

\hypertarget{colour-schemes}{%
\subsubsection{Colour schemes}\label{colour-schemes}}

You can also change colour schemes, by specifying either \texttt{scale\_colour\_*()} or \texttt{scale\_fill\_*()}
functions. \texttt{scale\_colour\_*()} functions are used for continuous variables, while \texttt{scale\_fill\_*()}
functions for discrete variables (so for barplots for example). A colour scheme I like is the
\href{https://www.highcharts.com/}{Highcharts} colour scheme.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme\_minimal}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{scale\_colour\_hc}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{(}\AttributeTok{legend.position =} \StringTok{"bottom"}\NormalTok{, }\AttributeTok{legend.title =} \FunctionTok{element\_blank}\NormalTok{()) }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-467-1.pdf}

An example with a barplot:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(test\_data\_long) }\SpecialCharTok{+}
  \FunctionTok{facet\_wrap}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\NormalTok{id) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value, }\AttributeTok{fill =}\NormalTok{ variable), }\AttributeTok{stat =} \StringTok{"identity"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_text}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value }\SpecialCharTok{+} \FloatTok{1.5}\NormalTok{, }\AttributeTok{label =}\NormalTok{ value)) }\SpecialCharTok{+}
  \FunctionTok{theme\_minimal}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{scale\_fill\_hc}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-468-1.pdf}

It is also possible to define and use your own palette.

To use your own colours you can use \texttt{scale\_colour\_manual()} and \texttt{scale\_fill\_manual()} and specify
the html codes of the colours you want to use.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division, }\AttributeTok{colour =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme\_minimal}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{scale\_colour\_manual}\NormalTok{(}\AttributeTok{values =} \FunctionTok{c}\NormalTok{(}\StringTok{"\#FF336C"}\NormalTok{, }\StringTok{"\#334BFF"}\NormalTok{, }\StringTok{"\#2CAE00"}\NormalTok{)) }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{(}\AttributeTok{legend.position =} \StringTok{"bottom"}\NormalTok{, }\AttributeTok{legend.title =} \FunctionTok{element\_blank}\NormalTok{()) }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg, Esch/Alzette and Wiltz"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-469-1.pdf}

To get html codes of colours you can use \href{http://htmlcolorcodes.com/color-picker/}{this online
tool}.
There is also a very nice package, called \texttt{colourpicker} that allows you to
pick colours from with RStudio. Also, you do not even need to load it to use
it, since it comes with an Addin:

\includegraphics{pics/rstudio_colourpicker.gif}

For a barplot you would do the same:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(test\_data\_long) }\SpecialCharTok{+}
  \FunctionTok{facet\_wrap}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\NormalTok{id) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value, }\AttributeTok{fill =}\NormalTok{ variable), }\AttributeTok{stat =} \StringTok{"identity"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_text}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable, value }\SpecialCharTok{+} \FloatTok{1.5}\NormalTok{, }\AttributeTok{label =}\NormalTok{ value)) }\SpecialCharTok{+}
  \FunctionTok{theme\_minimal}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{theme}\NormalTok{(}\AttributeTok{legend.position =} \StringTok{"bottom"}\NormalTok{, }\AttributeTok{legend.title =} \FunctionTok{element\_blank}\NormalTok{()) }\SpecialCharTok{+}
  \FunctionTok{scale\_fill\_manual}\NormalTok{(}\AttributeTok{values =} \FunctionTok{c}\NormalTok{(}\StringTok{"\#FF336C"}\NormalTok{, }\StringTok{"\#334BFF"}\NormalTok{, }\StringTok{"\#2CAE00"}\NormalTok{, }\StringTok{"\#B3C9C6"}\NormalTok{, }\StringTok{"\#765234"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-471-1.pdf}

For countinuous variables, things are a bit different. Let's first create a plot where we map a continuous
variable to the colour argument of \texttt{aes()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(diamonds) }\SpecialCharTok{+}
  \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(carat, price, }\AttributeTok{colour =}\NormalTok{ depth))}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-472-1.pdf}

To change the colour, we need to use \texttt{scale\_color\_gradient()} and specify a value for low values of the variable,
and a value for high values of the variable. For example, using the colours of the theme I made for my blog:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(diamonds) }\SpecialCharTok{+}
  \FunctionTok{geom\_point}\NormalTok{(}\FunctionTok{aes}\NormalTok{(carat, price, }\AttributeTok{colour =}\NormalTok{ depth)) }\SpecialCharTok{+}
  \FunctionTok{scale\_color\_gradient}\NormalTok{(}\AttributeTok{low =} \StringTok{"\#bec3b8"}\NormalTok{, }\AttributeTok{high =} \StringTok{"\#ad2c6c"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-473-1.pdf}

\hypertarget{saving-plots-to-disk}{%
\subsection{Saving plots to disk}\label{saving-plots-to-disk}}

There are two ways to save plots on disk; one through the \emph{Plots} plane in RStudio and another using the
\texttt{ggsave()} function. Using RStudio, navigate to the \emph{Plots} pane and click on \emph{Export}. You can
then choose where to save the plot and other various options:

\includegraphics{pics/rstudio_save_plots.gif}

This is fine if you only generate one or two plots but if you generate a large number of them, it
is less tedious to use the \texttt{ggsave()} function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_plot1 }\OtherTok{\textless{}{-}} \FunctionTok{ggplot}\NormalTok{(my\_data) }\SpecialCharTok{+}
  \FunctionTok{geom\_bar}\NormalTok{(}\FunctionTok{aes}\NormalTok{(variable))}

\FunctionTok{ggsave}\NormalTok{(}\StringTok{"path/you/want/to/save/the/plot/to/my\_plot1.pdf"}\NormalTok{, my\_plot1)}
\end{Highlighting}
\end{Shaded}

There are other options that you can specify such as the width and height, resolution, units,
etc\ldots{}

\hypertarget{exercises-3}{%
\subsection{Exercises}\label{exercises-3}}

\hypertarget{exercise-1-3}{%
\subsubsection*{Exercise 1}\label{exercise-1-3}}
\addcontentsline{toc}{subsubsection}{Exercise 1}

Load the \texttt{Bwages} dataset from the \texttt{Ecdat} package. Your first task is to create a new variable,
\texttt{educ\_level}, which is a factor variable that equals:

\begin{itemize}
\tightlist
\item
  ``Primary school'' if \texttt{educ\ ==\ 1}
\item
  ``High school'' if \texttt{educ\ ==\ 2}
\item
  ``Some university'' if \texttt{educ\ ==\ 3}
\item
  ``Master's degree'' if \texttt{educ\ ==\ 4}
\item
  ``Doctoral degree'' if \texttt{educ\ ==\ 5}
\end{itemize}

Use \texttt{case\_when()} for this.

Then, plot a scatter plot of wages on experience, by education level. Add a theme that you like,
and remove the title of the legend.

The scatter plot is not very useful, because you cannot make anything out. Instead, use another
geom that shows you a non-parametric fit with confidence bands.

\hypertarget{statistical-models}{%
\section{Statistical models}\label{statistical-models}}

In this chapter, we will not learn about all the models out there that you may or may not need.
Instead, I will show you how can use what you have learned until now and how you can apply these
concepts to modeling. Also, as you read in the beginning of the book, R has many many packages. So
the model you need is most probably already implemented in some package and you will very likely
not need to write your own from scratch.

In the first section, I will discuss the terminology used in this book. Then I will discuss
linear regression; showing how linear regression works illsutrates very well how other models
work too, without loss of generality. Then I will introduce the concepte of hyper-parameters
with ridge regression. This chapter will then finish with an introduction to cross-validation as
a way to tune the hyper-parameters of models that features them.

\hypertarget{terminology}{%
\subsection{Terminology}\label{terminology}}

Before continuing discussing about statistical models and model fitting it is worthwhile to discuss
terminology a little bit. Depending on your background, you might call an explanatory variable a
feature or the dependent variable the target. These are the same objects. The matrix of features
is usually called a design matrix, and what statisticians call the intercept is what
machine learning engineers call the bias. Referring to the intercept by bias is unfortunate, as bias
also has a very different meaning; bias is also what we call the error in a model that may cause
\emph{biased} estimates. To finish up, the estimated parameters of the model may be called coefficients
or weights. Here again, I don't like the using \emph{weight} as weight as a very different meaning in
statistics.
So, in the remainder of this chapter, and book, I will use the terminology from the statistical
litterature, using dependent and explanatory variables (\texttt{y} and \texttt{x}), and calling the
estimated parameters coefficients and the intercept\ldots{} well the intercept (the \(\beta\)s of the model).
However, I will talk of \emph{training} a model, instead of \emph{estimating} a model.

\hypertarget{fitting-a-model-to-data}{%
\subsection{Fitting a model to data}\label{fitting-a-model-to-data}}

Suppose you have a variable \texttt{y} that you wish to explain using a set of other variables \texttt{x1}, \texttt{x2},
\texttt{x3}, etc. Let's take a look at the \texttt{Housing} dataset from the \texttt{Ecdat} package:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(Ecdat)}

\FunctionTok{data}\NormalTok{(Housing)}
\end{Highlighting}
\end{Shaded}

You can read a description of the dataset by running:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{?Housing}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
Housing                 package:Ecdat                  R Documentation

Sales Prices of Houses in the City of Windsor

Description:

     a cross-section from 1987

     _number of observations_ : 546

     _observation_ : goods

     _country_ : Canada

Usage:

     data(Housing)

Format:

     A dataframe containing :

     price: sale price of a house

     lotsize: the lot size of a property in square feet

     bedrooms: number of bedrooms

     bathrms: number of full bathrooms

     stories: number of stories excluding basement

     driveway: does the house has a driveway ?

     recroom: does the house has a recreational room ?

     fullbase: does the house has a full finished basement ?

     gashw: does the house uses gas for hot water heating ?

     airco: does the house has central air conditioning ?

     garagepl: number of garage places

     prefarea: is the house located in the preferred neighbourhood of the city ?

Source:

     Anglin, P.M.  and R.  Gencay (1996) “Semiparametric estimation of
     a hedonic price function”, _Journal of Applied Econometrics_,
     *11(6)*, 633-648.

References:

     Verbeek, Marno (2004) _A Guide to Modern Econometrics_, John Wiley
     and Sons, chapter 3.

     Journal of Applied Econometrics data archive : <URL:
     http://qed.econ.queensu.ca/jae/>.

See Also:

     ‘Index.Source’, ‘Index.Economics’, ‘Index.Econometrics’,
     ‘Index.Observations’
\end{verbatim}

or by looking for \texttt{Housing} in the help pane of RStudio. Usually, you would take a look a the data
before doing any modeling:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{glimpse}\NormalTok{(Housing)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 546
## Columns: 12
## $ price    <dbl> 42000, 38500, 49500, 60500, 61000, 66000, 66000, 69000, 83800~
## $ lotsize  <dbl> 5850, 4000, 3060, 6650, 6360, 4160, 3880, 4160, 4800, 5500, 7~
## $ bedrooms <dbl> 3, 2, 3, 3, 2, 3, 3, 3, 3, 3, 3, 2, 3, 3, 2, 2, 3, 4, 1, 2, 3~
## $ bathrms  <dbl> 1, 1, 1, 1, 1, 1, 2, 1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1~
## $ stories  <dbl> 2, 1, 1, 2, 1, 1, 2, 3, 1, 4, 1, 1, 2, 1, 1, 1, 2, 3, 1, 1, 2~
## $ driveway <fct> yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, no, ye~
## $ recroom  <fct> no, no, no, yes, no, yes, no, no, yes, yes, no, no, no, no, n~
## $ fullbase <fct> yes, no, no, no, no, yes, yes, no, yes, no, yes, no, no, no, ~
## $ gashw    <fct> no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, n~
## $ airco    <fct> no, no, no, no, no, yes, no, no, no, yes, yes, no, no, no, no~
## $ garagepl <dbl> 1, 0, 0, 0, 0, 0, 2, 0, 0, 1, 3, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1~
## $ prefarea <fct> no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, n~
\end{verbatim}

Housing prices depend on a set of variables such as the number of bedrooms, the area it is located
and so on. If you believe that housing prices depend linearly on a set of explanatory variables,
you will want to estimate a linear model. To estimate a \emph{linear model}, you will need to use the
built-in \texttt{lm()} function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model1 }\OtherTok{\textless{}{-}} \FunctionTok{lm}\NormalTok{(price }\SpecialCharTok{\textasciitilde{}}\NormalTok{ lotsize }\SpecialCharTok{+}\NormalTok{ bedrooms, }\AttributeTok{data =}\NormalTok{ Housing)}
\end{Highlighting}
\end{Shaded}

\texttt{lm()} takes a formula as an argument, which defines the model you want to estimate. In this case,
I ran the following regression:

\[
\text{price} = \beta_0 + \beta_1 * \text{lotsize} + \beta_2 * \text{bedrooms} + \varepsilon
\]

where \(\beta_0, \beta_1\) and \(\beta_2\) are three parameters to estimate. To take a look at the
results, you can use the \texttt{summary()} method (not to be confused with \texttt{dplyr::summarise()}):

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{summary}\NormalTok{(model1)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Call:
## lm(formula = price ~ lotsize + bedrooms, data = Housing)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -65665 -12498  -2075   8970  97205 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 5.613e+03  4.103e+03   1.368    0.172    
## lotsize     6.053e+00  4.243e-01  14.265  < 2e-16 ***
## bedrooms    1.057e+04  1.248e+03   8.470 2.31e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 21230 on 543 degrees of freedom
## Multiple R-squared:  0.3703, Adjusted R-squared:  0.3679 
## F-statistic: 159.6 on 2 and 543 DF,  p-value: < 2.2e-16
\end{verbatim}

if you wish to remove the intercept (\(\beta_0\) in the above equation) from your model, you can
do so with \texttt{-1}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model2 }\OtherTok{\textless{}{-}} \FunctionTok{lm}\NormalTok{(price }\SpecialCharTok{\textasciitilde{}} \SpecialCharTok{{-}}\DecValTok{1} \SpecialCharTok{+}\NormalTok{ lotsize }\SpecialCharTok{+}\NormalTok{ bedrooms, }\AttributeTok{data =}\NormalTok{ Housing)}

\FunctionTok{summary}\NormalTok{(model2)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Call:
## lm(formula = price ~ -1 + lotsize + bedrooms, data = Housing)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -67229 -12342  -1333   9627  95509 
## 
## Coefficients:
##           Estimate Std. Error t value Pr(>|t|)    
## lotsize      6.283      0.390   16.11   <2e-16 ***
## bedrooms 11968.362    713.194   16.78   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 21250 on 544 degrees of freedom
## Multiple R-squared:  0.916,  Adjusted R-squared:  0.9157 
## F-statistic:  2965 on 2 and 544 DF,  p-value: < 2.2e-16
\end{verbatim}

or if you want to use all the columns inside \texttt{Housing}, replacing the column names by \texttt{.}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model3 }\OtherTok{\textless{}{-}} \FunctionTok{lm}\NormalTok{(price }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ Housing)}

\FunctionTok{summary}\NormalTok{(model3)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Call:
## lm(formula = price ~ ., data = Housing)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -41389  -9307   -591   7353  74875 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept) -4038.3504  3409.4713  -1.184 0.236762    
## lotsize         3.5463     0.3503  10.124  < 2e-16 ***
## bedrooms     1832.0035  1047.0002   1.750 0.080733 .  
## bathrms     14335.5585  1489.9209   9.622  < 2e-16 ***
## stories      6556.9457   925.2899   7.086 4.37e-12 ***
## drivewayyes  6687.7789  2045.2458   3.270 0.001145 ** 
## recroomyes   4511.2838  1899.9577   2.374 0.017929 *  
## fullbaseyes  5452.3855  1588.0239   3.433 0.000642 ***
## gashwyes    12831.4063  3217.5971   3.988 7.60e-05 ***
## aircoyes    12632.8904  1555.0211   8.124 3.15e-15 ***
## garagepl     4244.8290   840.5442   5.050 6.07e-07 ***
## prefareayes  9369.5132  1669.0907   5.614 3.19e-08 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 15420 on 534 degrees of freedom
## Multiple R-squared:  0.6731, Adjusted R-squared:  0.6664 
## F-statistic: 99.97 on 11 and 534 DF,  p-value: < 2.2e-16
\end{verbatim}

You can access different elements of \texttt{model3} with \texttt{\$}, because the result of \texttt{lm()} is a list
(you can check this claim with \texttt{typeof(model3)}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{print}\NormalTok{(model3}\SpecialCharTok{$}\NormalTok{coefficients)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  (Intercept)      lotsize     bedrooms      bathrms      stories  drivewayyes 
## -4038.350425     3.546303  1832.003466 14335.558468  6556.945711  6687.778890 
##   recroomyes  fullbaseyes     gashwyes     aircoyes     garagepl  prefareayes 
##  4511.283826  5452.385539 12831.406266 12632.890405  4244.829004  9369.513239
\end{verbatim}

but I prefer to use the \texttt{\{broom\}} package, and more specifically the \texttt{tidy()} function, which
converts \texttt{model3} into a neat \texttt{data.frame}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{results3 }\OtherTok{\textless{}{-}}\NormalTok{ broom}\SpecialCharTok{::}\FunctionTok{tidy}\NormalTok{(model3)}

\FunctionTok{glimpse}\NormalTok{(results3)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 12
## Columns: 5
## $ term      <chr> "(Intercept)", "lotsize", "bedrooms", "bathrms", "stories", ~
## $ estimate  <dbl> -4038.350425, 3.546303, 1832.003466, 14335.558468, 6556.9457~
## $ std.error <dbl> 3409.4713, 0.3503, 1047.0002, 1489.9209, 925.2899, 2045.2458~
## $ statistic <dbl> -1.184451, 10.123618, 1.749764, 9.621691, 7.086369, 3.269914~
## $ p.value   <dbl> 2.367616e-01, 3.732442e-22, 8.073341e-02, 2.570369e-20, 4.37~
\end{verbatim}

I explicitely write \texttt{broom::tidy()} because \texttt{tidy()} is a popular function name. For instance,
it is also a function from the \texttt{\{yardstick\}} package, which does not do the same thing at all. Since
I will also be using \texttt{\{yardstick\}} I prefer to explicitely write \texttt{broom::tidy()} to avoid conflicts.

Using \texttt{broom::tidy()} is useful, because you can then work on the results easily, for example if
you wish to only keep results that are significant at the 5\% level:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{results3 }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(p.value }\SpecialCharTok{\textless{}} \FloatTok{0.05}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 10 x 5
##    term        estimate std.error statistic  p.value
##    <chr>          <dbl>     <dbl>     <dbl>    <dbl>
##  1 lotsize         3.55     0.350     10.1  3.73e-22
##  2 bathrms     14336.    1490.         9.62 2.57e-20
##  3 stories      6557.     925.         7.09 4.37e-12
##  4 drivewayyes  6688.    2045.         3.27 1.15e- 3
##  5 recroomyes   4511.    1900.         2.37 1.79e- 2
##  6 fullbaseyes  5452.    1588.         3.43 6.42e- 4
##  7 gashwyes    12831.    3218.         3.99 7.60e- 5
##  8 aircoyes    12633.    1555.         8.12 3.15e-15
##  9 garagepl     4245.     841.         5.05 6.07e- 7
## 10 prefareayes  9370.    1669.         5.61 3.19e- 8
\end{verbatim}

You can even add new columns, such as the confidence intervals:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{results3 }\OtherTok{\textless{}{-}}\NormalTok{ broom}\SpecialCharTok{::}\FunctionTok{tidy}\NormalTok{(model3, }\AttributeTok{conf.int =} \ConstantTok{TRUE}\NormalTok{, }\AttributeTok{conf.level =} \FloatTok{0.95}\NormalTok{)}

\FunctionTok{print}\NormalTok{(results3)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 7
##    term        estimate std.error statistic  p.value  conf.low conf.high
##    <chr>          <dbl>     <dbl>     <dbl>    <dbl>     <dbl>     <dbl>
##  1 (Intercept) -4038.    3409.        -1.18 2.37e- 1 -10736.     2659.  
##  2 lotsize         3.55     0.350     10.1  3.73e-22      2.86      4.23
##  3 bedrooms     1832.    1047.         1.75 8.07e- 2   -225.     3889.  
##  4 bathrms     14336.    1490.         9.62 2.57e-20  11409.    17262.  
##  5 stories      6557.     925.         7.09 4.37e-12   4739.     8375.  
##  6 drivewayyes  6688.    2045.         3.27 1.15e- 3   2670.    10705.  
##  7 recroomyes   4511.    1900.         2.37 1.79e- 2    779.     8244.  
##  8 fullbaseyes  5452.    1588.         3.43 6.42e- 4   2333.     8572.  
##  9 gashwyes    12831.    3218.         3.99 7.60e- 5   6511.    19152.  
## 10 aircoyes    12633.    1555.         8.12 3.15e-15   9578.    15688.  
## 11 garagepl     4245.     841.         5.05 6.07e- 7   2594.     5896.  
## 12 prefareayes  9370.    1669.         5.61 3.19e- 8   6091.    12648.
\end{verbatim}

Going back to model estimation, you can of course use \texttt{lm()} in a pipe workflow:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{Housing }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{driveway, }\SpecialCharTok{{-}}\NormalTok{stories) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{lm}\NormalTok{(price }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ .) }\SpecialCharTok{\%\textgreater{}\%}
\NormalTok{  broom}\SpecialCharTok{::}\FunctionTok{tidy}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 10 x 5
##    term        estimate std.error statistic  p.value
##    <chr>          <dbl>     <dbl>     <dbl>    <dbl>
##  1 (Intercept)  3025.    3263.        0.927 3.54e- 1
##  2 lotsize         3.67     0.363    10.1   4.52e-22
##  3 bedrooms     4140.    1036.        3.99  7.38e- 5
##  4 bathrms     16443.    1546.       10.6   4.29e-24
##  5 recroomyes   5660.    2010.        2.82  5.05e- 3
##  6 fullbaseyes  2241.    1618.        1.38  1.67e- 1
##  7 gashwyes    13568.    3411.        3.98  7.93e- 5
##  8 aircoyes    15578.    1597.        9.75  8.53e-21
##  9 garagepl     4232.     883.        4.79  2.12e- 6
## 10 prefareayes 10729.    1753.        6.12  1.81e- 9
\end{verbatim}

The first \texttt{.} in the \texttt{lm()} function is used to indicate that we wish to use all the data from \texttt{Housing}
(minus \texttt{driveway} and \texttt{stories} which I removed using \texttt{select()} and the \texttt{-} sign), and the second \texttt{.} is
used to \emph{place} the result from the two \texttt{dplyr} instructions that preceded is to be placed there.
The picture below should help you understand:

\includegraphics[width=8in]{pics/pipe_to_second_position}

You have to specify this, because by default, when using \texttt{\%\textgreater{}\%} the left hand side argument gets
passed as the first argument of the function on the right hand side.

Since version 4.2, R now also natively includes a placeholder, \texttt{\_}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{Housing }\SpecialCharTok{|\textgreater{}}
  \FunctionTok{select}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{driveway, }\SpecialCharTok{{-}}\NormalTok{stories) }\SpecialCharTok{|\textgreater{}}
  \FunctionTok{lm}\NormalTok{(price }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ \_) }\SpecialCharTok{|\textgreater{}}
\NormalTok{  broom}\SpecialCharTok{::}\FunctionTok{tidy}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 10 x 5
##    term        estimate std.error statistic  p.value
##    <chr>          <dbl>     <dbl>     <dbl>    <dbl>
##  1 (Intercept)  3025.    3263.        0.927 3.54e- 1
##  2 lotsize         3.67     0.363    10.1   4.52e-22
##  3 bedrooms     4140.    1036.        3.99  7.38e- 5
##  4 bathrms     16443.    1546.       10.6   4.29e-24
##  5 recroomyes   5660.    2010.        2.82  5.05e- 3
##  6 fullbaseyes  2241.    1618.        1.38  1.67e- 1
##  7 gashwyes    13568.    3411.        3.98  7.93e- 5
##  8 aircoyes    15578.    1597.        9.75  8.53e-21
##  9 garagepl     4232.     883.        4.79  2.12e- 6
## 10 prefareayes 10729.    1753.        6.12  1.81e- 9
\end{verbatim}

For the example above, I've also switched from \texttt{\%\textgreater{}\%} to \texttt{\textbar{}\textgreater{}}, or else I can't use the \texttt{\_} placeholder.
The advantage of the \texttt{\_} placeholder is that it disambiguates \texttt{.}. So here, the \texttt{.} is a placeholder for
all the variables in the dataset, and \texttt{\_} is a placeholder for the dataset.

\hypertarget{diagnostics}{%
\subsection{Diagnostics}\label{diagnostics}}

Diagnostics are useful metrics to assess model fit. You can read some of these diagnostics, such as
the \(R^2\) at the bottom of the summary (when running \texttt{summary(my\_model)}), but if you want to do
more than simply reading these diagnostics from RStudio, you can put those in a \texttt{data.frame} too,
using \texttt{broom::glance()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{glance}\NormalTok{(model3)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 12
##   r.squared adj.r.~1  sigma stati~2   p.value    df logLik    AIC    BIC devia~3
##       <dbl>    <dbl>  <dbl>   <dbl>     <dbl> <dbl>  <dbl>  <dbl>  <dbl>   <dbl>
## 1     0.673    0.666 15423.    100. 6.18e-122    11 -6034. 12094. 12150. 1.27e11
## # ... with 2 more variables: df.residual <int>, nobs <int>, and abbreviated
## #   variable names 1: adj.r.squared, 2: statistic, 3: deviance
\end{verbatim}

You can also plot the usual diagnostics plots using \texttt{ggfortify::autoplot()} which uses the
\texttt{\{ggplot2\}} package under the hood:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(ggfortify)}

\FunctionTok{autoplot}\NormalTok{(model3, }\AttributeTok{which =} \DecValTok{1}\SpecialCharTok{:}\DecValTok{6}\NormalTok{) }\SpecialCharTok{+} \FunctionTok{theme\_minimal}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-495-1.pdf}

\texttt{which=1:6} is an additional option that shows you all the diagnostics plot. If you omit this
option, you will only get 4 of them.

You can also get the residuals of the regression in two ways; either you grab them directly from
the model fit:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{resi3 }\OtherTok{\textless{}{-}} \FunctionTok{residuals}\NormalTok{(model3)}
\end{Highlighting}
\end{Shaded}

or you can augment the original data with a residuals column, using \texttt{broom::augment()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{housing\_aug }\OtherTok{\textless{}{-}} \FunctionTok{augment}\NormalTok{(model3)}
\end{Highlighting}
\end{Shaded}

Let's take a look at \texttt{housing\_aug}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{glimpse}\NormalTok{(housing\_aug)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 546
## Columns: 18
## $ price      <dbl> 42000, 38500, 49500, 60500, 61000, 66000, 66000, 69000, 838~
## $ lotsize    <dbl> 5850, 4000, 3060, 6650, 6360, 4160, 3880, 4160, 4800, 5500,~
## $ bedrooms   <dbl> 3, 2, 3, 3, 2, 3, 3, 3, 3, 3, 3, 2, 3, 3, 2, 2, 3, 4, 1, 2,~
## $ bathrms    <dbl> 1, 1, 1, 1, 1, 1, 2, 1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 2,~
## $ stories    <dbl> 2, 1, 1, 2, 1, 1, 2, 3, 1, 4, 1, 1, 2, 1, 1, 1, 2, 3, 1, 1,~
## $ driveway   <fct> yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, no, ~
## $ recroom    <fct> no, no, no, yes, no, yes, no, no, yes, yes, no, no, no, no,~
## $ fullbase   <fct> yes, no, no, no, no, yes, yes, no, yes, no, yes, no, no, no~
## $ gashw      <fct> no, no, no, no, no, no, no, no, no, no, no, no, no, no, no,~
## $ airco      <fct> no, no, no, no, no, yes, no, no, no, yes, yes, no, no, no, ~
## $ garagepl   <dbl> 1, 0, 0, 0, 0, 0, 2, 0, 0, 1, 3, 0, 0, 0, 0, 0, 1, 0, 0, 1,~
## $ prefarea   <fct> no, no, no, no, no, no, no, no, no, no, no, no, no, no, no,~
## $ .fitted    <dbl> 66037.98, 41391.15, 39889.63, 63689.09, 49760.43, 66387.12,~
## $ .resid     <dbl> -24037.9757, -2891.1515, 9610.3699, -3189.0873, 11239.5735,~
## $ .hat       <dbl> 0.013477335, 0.008316321, 0.009893730, 0.021510891, 0.01033~
## $ .sigma     <dbl> 15402.01, 15437.14, 15431.98, 15437.02, 15429.89, 15437.64,~
## $ .cooksd    <dbl> 2.803214e-03, 2.476265e-05, 3.265481e-04, 8.004787e-05, 4.6~
## $ .std.resid <dbl> -1.56917096, -0.18823924, 0.62621736, -0.20903274, 0.732539~
\end{verbatim}

A few columns have been added to the original data, among them \texttt{.resid} which contains the
residuals. Let's plot them:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(housing\_aug) }\SpecialCharTok{+}
  \FunctionTok{geom\_density}\NormalTok{(}\FunctionTok{aes}\NormalTok{(.resid))}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-500-1.pdf}

Fitted values are also added to the original data, under the variable \texttt{.fitted}. It would also have
been possible to get the fitted values with:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fit3 }\OtherTok{\textless{}{-}} \FunctionTok{fitted}\NormalTok{(model3)}
\end{Highlighting}
\end{Shaded}

but I prefer using \texttt{augment()}, because the columns get merged to the original data, which then
makes it easier to find specific individuals, for example, you might want to know for which housing
units the model underestimates the price:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{total\_pos }\OtherTok{\textless{}{-}}\NormalTok{ housing\_aug }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(.resid }\SpecialCharTok{\textgreater{}} \DecValTok{0}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{total =} \FunctionTok{n}\NormalTok{()) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{pull}\NormalTok{(total)}
\end{Highlighting}
\end{Shaded}

we find 261 individuals where the residuals are positive. It is also easier to
extract outliers:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{housing\_aug }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{prank =} \FunctionTok{cume\_dist}\NormalTok{(.cooksd)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(prank }\SpecialCharTok{\textgreater{}} \FloatTok{0.99}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{glimpse}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 6
## Columns: 19
## $ price      <dbl> 163000, 125000, 132000, 175000, 190000, 174500
## $ lotsize    <dbl> 7420, 4320, 3500, 9960, 7420, 7500
## $ bedrooms   <dbl> 4, 3, 4, 3, 4, 4
## $ bathrms    <dbl> 1, 1, 2, 2, 2, 2
## $ stories    <dbl> 2, 2, 2, 2, 3, 2
## $ driveway   <fct> yes, yes, yes, yes, yes, yes
## $ recroom    <fct> yes, no, no, no, no, no
## $ fullbase   <fct> yes, yes, no, yes, no, yes
## $ gashw      <fct> no, yes, yes, no, no, no
## $ airco      <fct> yes, no, no, no, yes, yes
## $ garagepl   <dbl> 2, 2, 2, 2, 2, 3
## $ prefarea   <fct> no, no, no, yes, yes, yes
## $ .fitted    <dbl> 94826.68, 77688.37, 85495.58, 108563.18, 115125.03, 118549.~
## $ .resid     <dbl> 68173.32, 47311.63, 46504.42, 66436.82, 74874.97, 55951.00
## $ .hat       <dbl> 0.02671105, 0.05303793, 0.05282929, 0.02819317, 0.02008141,~
## $ .sigma     <dbl> 15144.70, 15293.34, 15298.27, 15159.14, 15085.99, 15240.66
## $ .cooksd    <dbl> 0.04590995, 0.04637969, 0.04461464, 0.04616068, 0.04107317,~
## $ .std.resid <dbl> 4.480428, 3.152300, 3.098176, 4.369631, 4.904193, 3.679815
## $ prank      <dbl> 0.9963370, 1.0000000, 0.9945055, 0.9981685, 0.9926740, 0.99~
\end{verbatim}

\texttt{prank} is a variable I created with \texttt{cume\_dist()} which is a \texttt{dplyr} function that returns the
proportion of all values less than or equal to the current rank. For example:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{example }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\DecValTok{5}\NormalTok{, }\FloatTok{4.6}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{1}\NormalTok{, }\FloatTok{0.8}\NormalTok{, }\DecValTok{0}\NormalTok{, }\SpecialCharTok{{-}}\DecValTok{1}\NormalTok{)}
\FunctionTok{cume\_dist}\NormalTok{(example)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1.0000000 0.8571429 0.7142857 0.5714286 0.4285714 0.2857143 0.1428571
\end{verbatim}

by filtering \texttt{prank\ \textgreater{}\ 0.99} we get the top 1\% of outliers according to Cook's distance.

\hypertarget{interpreting-models}{%
\subsection{Interpreting models}\label{interpreting-models}}

Model interpretation is essential in the social sciences, but it is also getting very important
in machine learning. As usual, the terminology is different; in machine learning, we speak about
explainability. There is a very important aspect that one has to understand when it comes to
interpretability/explainability: \emph{classical, parametric} models, and \emph{black-box} models. This
is very well explained in \citet{breiman2001}, an absolute must read (link to paper, in PDF format:
\href{https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726}{click here}). The gist of the paper
is that there are two cultures of statistical modeling; one culture relies on modeling the data
generating process, for instance, by considering that a variable y (independent variable, or target)
is a linear combination of input variables x (dependent variables, or features) plus some noise. The
other culture uses complex algorithms (random forests, neural networks)
to model the relationship between y and x. The author argues that most statisticians have relied
for too long on modeling data generating processes and do not use all the potential offered by
these complex algorithms. I think that a lot of things have changed since then, and that nowadays
any practitioner that uses data is open to use any type of model or algorithm, as long as it does
the job. However, the paper is very interesting, and the discussion on trade-off between
simplicity of the model and interpretability/explainability is still relevant today.

In this section, I will explain how one can go about interpreting or explaining models from these
two cultures.

Also, it is important to note here that the discussion that will follow will be heavily influenced
by my econometrics background. I will focus on marginal effects as way to interpret parametric
models (models from the first culture described above), but depending on the field, practitioners
might use something else (for instance by computing odds ratios in a logistic regression).

I will start by interpretability of \emph{classical} statistical models.

\hypertarget{marginal-effects}{%
\subsubsection{Marginal effects}\label{marginal-effects}}

If one wants to know the effect of variable \texttt{x} on the dependent variable \texttt{y},
so-called marginal effects have to be computed. This is easily done in R with the \texttt{\{marginaleffects\}} package.
Formally, marginal effects are the partial derivative of the regression equation with respect to the variable
we want to look at.

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(marginaleffects)}

\NormalTok{effects\_model3 }\OtherTok{\textless{}{-}} \FunctionTok{marginaleffects}\NormalTok{(model3)}

\FunctionTok{summary}\NormalTok{(effects\_model3)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##        Term Contrast    Effect Std. Error z value   Pr(>|z|)    2.5 %    97.5 %
## 1   lotsize    dY/dX     3.546     0.3503  10.124 < 2.22e-16     2.86     4.233
## 2  bedrooms    dY/dX  1832.003  1047.0056   1.750 0.08016056  -220.09  3884.097
## 3   bathrms    dY/dX 14335.558  1489.9557   9.621 < 2.22e-16 11415.30 17255.818
## 4   stories    dY/dX  6556.946   925.2943   7.086 1.3771e-12  4743.40  8370.489
## 5  driveway yes - no  6687.779  2045.2459   3.270 0.00107580  2679.17 10696.387
## 6   recroom yes - no  4511.284  1899.9577   2.374 0.01757689   787.44  8235.132
## 7  fullbase yes - no  5452.386  1588.0239   3.433 0.00059597  2339.92  8564.855
## 8     gashw yes - no 12831.406  3217.5970   3.988 6.6665e-05  6525.03 19137.781
## 9     airco yes - no 12632.890  1555.0211   8.124 4.5131e-16  9585.11 15680.676
## 10 garagepl    dY/dX  4244.829   840.5965   5.050 4.4231e-07  2597.29  5892.368
## 11 prefarea yes - no  9369.513  1669.0906   5.614 1.9822e-08  6098.16 12640.871
## 
## Model type:  lm 
## Prediction type:  response
\end{verbatim}

Let's go through this: \texttt{summary(effects\_model3)} shows the average marginal effects for each of the dependent
variables that were used in \texttt{model3}. The way to interpret them is as follows:
\emph{everything else held constant (often you'll read the Latin ceteris paribus for this), a unit increase in
\texttt{lotize} increases the \texttt{price} by 3.546 units, on average.}
The \emph{everything held constant} part is crucial; with marginal effects, you're looking at just the effect of
one variable at a time. For discrete variables, like \texttt{driveway}, this is simpler: imagine you have two
equal houses, exactly the same house, one has a driveway and the other doesn't. The one with the driveway
is 6687 units more expensive, \emph{on average}.

Now it turns out that in the case of a linear model, the average marginal effects are exactly equal to the
coefficients. Just compare \texttt{summary(model3)} to \texttt{effects\_model3} to see
(and remember, I told you that marginal effects were the partial derivative of the regression equation with
respect to the variable of interest. So the derivative of \(\alpha*X_1 + ....\) with respect to \(X_1\) will
be \(\alpha\)). But in the case of a more complex, non-linear model, this is not so obvious. This is
where \texttt{\{marginaleffects\}} will make your life much easier.

It is also possible to plot the results:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{plot}\NormalTok{(effects\_model3)}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-506-1.pdf}

\texttt{effects\_model3} is a data frame containing the effects for each house in the data set. For example,
let's take a look at the first house:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{effects\_model3 }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(rowid }\SpecialCharTok{==} \DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##    rowid     type     term contrast         dydx    std.error statistic
## 1      1 response  lotsize    dY/dX     3.546303    0.3502195 10.125944
## 2      1 response bedrooms    dY/dX  1832.003466 1046.1608842  1.751168
## 3      1 response  bathrms    dY/dX 14335.558468 1490.4827945  9.618064
## 4      1 response  stories    dY/dX  6556.945711  925.4764870  7.084940
## 5      1 response driveway yes - no  6687.778890 2045.2460319  3.269914
## 6      1 response  recroom yes - no  4511.283826 1899.9577182  2.374413
## 7      1 response fullbase yes - no  5452.385539 1588.0237538  3.433441
## 8      1 response    gashw yes - no 12831.406266 3217.5971931  3.987885
## 9      1 response    airco yes - no 12632.890405 1555.0207045  8.123937
## 10     1 response garagepl    dY/dX  4244.829004  840.8930857  5.048001
## 11     1 response prefarea yes - no  9369.513239 1669.0904968  5.613544
##         p.value     conf.low    conf.high predicted predicted_hi predicted_lo
## 1  4.238689e-24     2.859885     4.232721  66037.98     66043.14     66037.98
## 2  7.991698e-02  -218.434189  3882.441121  66037.98     66038.89     66037.98
## 3  6.708200e-22 11414.265872 17256.851065  66037.98     66042.28     66037.98
## 4  1.391042e-12  4743.045128  8370.846295  66037.98     66039.94     66037.98
## 5  1.075801e-03  2679.170328 10696.387452  66037.98     66037.98     59350.20
## 6  1.757689e-02   787.435126  8235.132526  66037.98     70549.26     66037.98
## 7  5.959723e-04  2339.916175  8564.854903  66037.98     66037.98     60585.59
## 8  6.666508e-05  6525.031651 19137.780882  66037.98     78869.38     66037.98
## 9  4.512997e-16  9585.105829 15680.674981  66037.98     78670.87     66037.98
## 10 4.464572e-07  2596.708842  5892.949167  66037.98     66039.25     66037.98
## 11 1.982240e-08  6098.155978 12640.870499  66037.98     75407.49     66037.98
##    price lotsize bedrooms bathrms stories driveway recroom fullbase gashw airco
## 1  42000    5850        3       1       2      yes      no      yes    no    no
## 2  42000    5850        3       1       2      yes      no      yes    no    no
## 3  42000    5850        3       1       2      yes      no      yes    no    no
## 4  42000    5850        3       1       2      yes      no      yes    no    no
## 5  42000    5850        3       1       2      yes      no      yes    no    no
## 6  42000    5850        3       1       2      yes      no      yes    no    no
## 7  42000    5850        3       1       2      yes      no      yes    no    no
## 8  42000    5850        3       1       2      yes      no      yes    no    no
## 9  42000    5850        3       1       2      yes      no      yes    no    no
## 10 42000    5850        3       1       2      yes      no      yes    no    no
## 11 42000    5850        3       1       2      yes      no      yes    no    no
##    garagepl prefarea    eps
## 1         1       no 1.4550
## 2         1       no 0.0005
## 3         1       no 0.0003
## 4         1       no 0.0003
## 5         1       no     NA
## 6         1       no     NA
## 7         1       no     NA
## 8         1       no     NA
## 9         1       no     NA
## 10        1       no 0.0003
## 11        1       no     NA
\end{verbatim}

\texttt{rowid} is column that identifies the houses in the original data set, so \texttt{rowid\ ==\ 1} filters out
the first house. This shows you the marginal effects (column \texttt{dydx} computed for this house; but
remember, since we're dealing with a linear model, the values of the marginal effects are constant.
If you don't see the point of this discussion, don't fret, the next example should make things
clearer.

Let's estimate a logit model and compute the marginal effects. You might know logit models as
\emph{logistic regression}. Logit models can be estimated using the \texttt{glm()} function, which stands for
generalized linear models.

As an example, we are going to use the \texttt{Participation} data, also from the \texttt{\{Ecdat\}} package:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(Participation)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{?Particpation}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
Participation              package:Ecdat               R Documentation

Labor Force Participation

Description:

     a cross-section

     _number of observations_ : 872

     _observation_ : individuals

     _country_ : Switzerland

Usage:

     data(Participation)

Format:

     A dataframe containing :

     lfp labour force participation ?

     lnnlinc the log of nonlabour income

     age age in years divided by 10

     educ years of formal education

     nyc the number of young children (younger than 7)

     noc number of older children

     foreign foreigner ?

Source:

     Gerfin, Michael (1996) “Parametric and semiparametric estimation
     of the binary response”, _Journal of Applied Econometrics_,
     *11(3)*, 321-340.

References:

     Davidson, R.  and James G.  MacKinnon (2004) _Econometric Theory
     and Methods_, New York, Oxford University Press, <URL:
     http://www.econ.queensu.ca/ETM/>, chapter 11.

     Journal of Applied Econometrics data archive : <URL:
     http://qed.econ.queensu.ca/jae/>.

See Also:

     ‘Index.Source’, ‘Index.Economics’, ‘Index.Econometrics’,
     ‘Index.Observations’
\end{verbatim}

The variable of interest is \texttt{lfp}: whether the individual participates in the labour force or not.
To know which variables are relevant in the decision to participate in the labour force, one could
train a logit model, using \texttt{glm()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{logit\_participation }\OtherTok{\textless{}{-}} \FunctionTok{glm}\NormalTok{(lfp }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ Participation, }\AttributeTok{family =} \StringTok{"binomial"}\NormalTok{)}

\NormalTok{broom}\SpecialCharTok{::}\FunctionTok{tidy}\NormalTok{(logit\_participation)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 7 x 5
##   term        estimate std.error statistic  p.value
##   <chr>          <dbl>     <dbl>     <dbl>    <dbl>
## 1 (Intercept)  10.4       2.17       4.79  1.69e- 6
## 2 lnnlinc      -0.815     0.206     -3.97  7.31e- 5
## 3 age          -0.510     0.0905    -5.64  1.72e- 8
## 4 educ          0.0317    0.0290     1.09  2.75e- 1
## 5 nyc          -1.33      0.180     -7.39  1.51e-13
## 6 noc          -0.0220    0.0738    -0.298 7.66e- 1
## 7 foreignyes    1.31      0.200      6.56  5.38e-11
\end{verbatim}

From the results above, one can only interpret the sign of the coefficients. To know how much a
variable influences the labour force participation, one has to use \texttt{marginaleffects()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{effects\_logit\_participation }\OtherTok{\textless{}{-}} \FunctionTok{marginaleffects}\NormalTok{(logit\_participation)}

\FunctionTok{summary}\NormalTok{(effects\_logit\_participation)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      Term Contrast    Effect Std. Error z value   Pr(>|z|)     2.5 %   97.5 %
## 1 lnnlinc    dY/dX -0.169940    0.04151 -4.0939 4.2416e-05 -0.251300 -0.08858
## 2     age    dY/dX -0.106407    0.01759 -6.0492 1.4560e-09 -0.140884 -0.07193
## 3    educ    dY/dX  0.006616    0.00604  1.0954    0.27335 -0.005222  0.01845
## 4     nyc    dY/dX -0.277463    0.03325 -8.3436 < 2.22e-16 -0.342642 -0.21229
## 5     noc    dY/dX -0.004584    0.01538 -0.2981    0.76563 -0.034725  0.02556
## 6 foreign yes - no  0.283377    0.03984  7.1129 1.1361e-12  0.205292  0.36146
## 
## Model type:  glm 
## Prediction type:  response
\end{verbatim}

As you can see, the average marginal effects here are not equal to the estimated coefficients of the
model. Let's take a look at the first row of the data:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{Participation[}\DecValTok{1}\NormalTok{, ]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   lfp lnnlinc age educ nyc noc foreign
## 1  no 10.7875   3    8   1   1      no
\end{verbatim}

and let's now look at \texttt{rowid\ ==\ 1} in the marginal effects data frame:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{effects\_logit\_participation }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(rowid }\SpecialCharTok{==} \DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   rowid     type    term contrast         dydx   std.error  statistic
## 1     1 response lnnlinc    dY/dX -0.156661756 0.038522800 -4.0667282
## 2     1 response     age    dY/dX -0.098097148 0.020123709 -4.8747052
## 3     1 response    educ    dY/dX  0.006099266 0.005367036  1.1364310
## 4     1 response     nyc    dY/dX -0.255784406 0.029367783 -8.7096942
## 5     1 response     noc    dY/dX -0.004226368 0.014167283 -0.2983189
## 6     1 response foreign yes - no  0.305630005 0.045174828  6.7654935
##        p.value     conf.low   conf.high predicted predicted_hi predicted_lo lfp
## 1 4.767780e-05 -0.232165056 -0.08115846 0.2596523    0.2595710    0.2596523  no
## 2 1.089711e-06 -0.137538892 -0.05865540 0.2596523    0.2596111    0.2596523  no
## 3 2.557762e-01 -0.004419931  0.01661846 0.2596523    0.2596645    0.2596523  no
## 4 3.046958e-18 -0.313344203 -0.19822461 0.2596523    0.2595755    0.2596523  no
## 5 7.654598e-01 -0.031993732  0.02354100 0.2596523    0.2596497    0.2596523  no
## 6 1.328556e-11  0.217088969  0.39417104 0.2596523    0.5652823    0.2596523  no
##   lnnlinc age educ nyc noc foreign          eps
## 1 10.7875   3    8   1   1      no 0.0005188749
## 2 10.7875   3    8   1   1      no 0.0004200000
## 3 10.7875   3    8   1   1      no 0.0020000000
## 4 10.7875   3    8   1   1      no 0.0003000000
## 5 10.7875   3    8   1   1      no 0.0006000000
## 6 10.7875   3    8   1   1      no           NA
\end{verbatim}

Let's focus on the first row, where \texttt{term} is \texttt{lnnlinc}. What we see here is the effect of an infinitesimal
increase in the variable \texttt{lnnlinc} on the participation, for an individual who has the following
characteristics: \texttt{lnnlinc\ =\ 10.7875}, \texttt{age\ =\ 3}, \texttt{educ\ =\ 8}, \texttt{nyc\ =\ 1}, \texttt{noc\ =\ 1} and \texttt{foreign\ =\ no}, which
are the characteristics of this first individual in our data.

So let's look at the value of \texttt{dydx} for every individual:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{dydx\_lnnlinc }\OtherTok{\textless{}{-}}\NormalTok{ effects\_logit\_participation }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(term }\SpecialCharTok{==} \StringTok{"lnnlinc"}\NormalTok{)}

\FunctionTok{head}\NormalTok{(dydx\_lnnlinc)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   rowid     type    term contrast        dydx  std.error statistic      p.value
## 1     1 response lnnlinc    dY/dX -0.15666176 0.03852280 -4.066728 4.767780e-05
## 2     2 response lnnlinc    dY/dX -0.20013939 0.05124543 -3.905507 9.402813e-05
## 3     3 response lnnlinc    dY/dX -0.18493932 0.04319729 -4.281271 1.858287e-05
## 4     4 response lnnlinc    dY/dX -0.05376281 0.01586468 -3.388837 7.018964e-04
## 5     5 response lnnlinc    dY/dX -0.18709356 0.04502973 -4.154890 3.254439e-05
## 6     6 response lnnlinc    dY/dX -0.19586185 0.04782143 -4.095692 4.209096e-05
##      conf.low   conf.high  predicted predicted_hi predicted_lo lfp  lnnlinc age
## 1 -0.23216506 -0.08115846 0.25965227   0.25957098   0.25965227  no 10.78750 3.0
## 2 -0.30057859 -0.09970018 0.43340025   0.43329640   0.43340025 yes 10.52425 4.5
## 3 -0.26960445 -0.10027418 0.34808777   0.34799181   0.34808777  no 10.96858 4.6
## 4 -0.08485701 -0.02266862 0.07101902   0.07099113   0.07101902  no 11.10500 3.1
## 5 -0.27535020 -0.09883692 0.35704926   0.35695218   0.35704926  no 11.10847 4.4
## 6 -0.28959014 -0.10213356 0.40160949   0.40150786   0.40160949 yes 11.02825 4.2
##   educ nyc noc foreign          eps
## 1    8   1   1      no 0.0005188749
## 2    8   0   1      no 0.0005188749
## 3    9   0   0      no 0.0005188749
## 4   11   2   0      no 0.0005188749
## 5   12   0   2      no 0.0005188749
## 6   12   0   1      no 0.0005188749
\end{verbatim}

\texttt{dydx\_lnnlinc} is a data frame with all individual marginal effect for the variable \texttt{lnnlinc}.
What if we compute the mean of this column?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{dydx\_lnnlinc }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\FunctionTok{mean}\NormalTok{(dydx))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   mean(dydx)
## 1 -0.1699405
\end{verbatim}

Let's compare this to the average marginal effects:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{summary}\NormalTok{(effects\_logit\_participation)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      Term Contrast    Effect Std. Error z value   Pr(>|z|)     2.5 %   97.5 %
## 1 lnnlinc    dY/dX -0.169940    0.04151 -4.0939 4.2416e-05 -0.251300 -0.08858
## 2     age    dY/dX -0.106407    0.01759 -6.0492 1.4560e-09 -0.140884 -0.07193
## 3    educ    dY/dX  0.006616    0.00604  1.0954    0.27335 -0.005222  0.01845
## 4     nyc    dY/dX -0.277463    0.03325 -8.3436 < 2.22e-16 -0.342642 -0.21229
## 5     noc    dY/dX -0.004584    0.01538 -0.2981    0.76563 -0.034725  0.02556
## 6 foreign yes - no  0.283377    0.03984  7.1129 1.1361e-12  0.205292  0.36146
## 
## Model type:  glm 
## Prediction type:  response
\end{verbatim}

Yep, it's the same! This is why we speak of \emph{average marginal effects}. Now that we know why
these are called average marginal effects, let's go back to interpreting them. This time,
let's plot them, because why not:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{plot}\NormalTok{(effects\_logit\_participation)}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-517-1.pdf}

So an infinitesimal increase, in say, non-labour income (\texttt{lnnlinc}) of 0.001 is associated with a
decrease of the probability of labour force participation by 0.001*17 percentage points.

This is just scratching the surface of interpreting these kinds of models. There are many more
types of effects that you can compute and look at. I highly recommend you read the documentation
of \texttt{\{marginaleffects\}} which you can find
\href{https://vincentarelbundock.github.io/marginaleffects/index.html}{here}. The author
of the package, Vincent Arel-Bundock writes a lot of very helpful documentation for his packages,
so if model interpretation is important for your job, definitely take a look.

\hypertarget{explainability-of-black-box-models}{%
\subsubsection{\texorpdfstring{Explainability of \emph{black-box} models}{Explainability of black-box models}}\label{explainability-of-black-box-models}}

Just read Christoph Molnar's
\href{https://christophm.github.io/interpretable-ml-book/}{Interpretable Machine Learning}.

Seriously, I cannot add anything meaningful to it. His book is brilliant.

\hypertarget{comparing-models}{%
\subsection{Comparing models}\label{comparing-models}}

Consider this section more as an illustration of what is possible with the knowledge you acquired
at this point. Imagine that the task at hand is to compare two models. We would like to select
the one which has the best fit to the data.
Let's first estimate another model on the same data; prices are only positive, so a linear regression
might not be the best model, because the model could predict negative prices. Let's look at the
distribution of prices:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ggplot}\NormalTok{(Housing) }\SpecialCharTok{+}
  \FunctionTok{geom\_density}\NormalTok{(}\FunctionTok{aes}\NormalTok{(price))}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-518-1.pdf}

it looks like modeling the log of \texttt{price} might provide a better fit:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model\_log }\OtherTok{\textless{}{-}} \FunctionTok{lm}\NormalTok{(}\FunctionTok{log}\NormalTok{(price) }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ Housing)}

\NormalTok{result\_log }\OtherTok{\textless{}{-}}\NormalTok{ broom}\SpecialCharTok{::}\FunctionTok{tidy}\NormalTok{(model\_log)}

\FunctionTok{print}\NormalTok{(result\_log)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 12 x 5
##    term          estimate  std.error statistic  p.value
##    <chr>            <dbl>      <dbl>     <dbl>    <dbl>
##  1 (Intercept) 10.0       0.0472        212.   0       
##  2 lotsize      0.0000506 0.00000485     10.4  2.91e-23
##  3 bedrooms     0.0340    0.0145          2.34 1.94e- 2
##  4 bathrms      0.168     0.0206          8.13 3.10e-15
##  5 stories      0.0923    0.0128          7.20 2.10e-12
##  6 drivewayyes  0.131     0.0283          4.61 5.04e- 6
##  7 recroomyes   0.0735    0.0263          2.79 5.42e- 3
##  8 fullbaseyes  0.0994    0.0220          4.52 7.72e- 6
##  9 gashwyes     0.178     0.0446          4.00 7.22e- 5
## 10 aircoyes     0.178     0.0215          8.26 1.14e-15
## 11 garagepl     0.0508    0.0116          4.36 1.58e- 5
## 12 prefareayes  0.127     0.0231          5.50 6.02e- 8
\end{verbatim}

Let's take a look at the diagnostics:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{glance}\NormalTok{(model\_log)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 12
##   r.squared adj.r.squ~1 sigma stati~2   p.value    df logLik   AIC   BIC devia~3
##       <dbl>       <dbl> <dbl>   <dbl>     <dbl> <dbl>  <dbl> <dbl> <dbl>   <dbl>
## 1     0.677       0.670 0.214    102. 3.67e-123    11   73.9 -122. -65.8    24.4
## # ... with 2 more variables: df.residual <int>, nobs <int>, and abbreviated
## #   variable names 1: adj.r.squared, 2: statistic, 3: deviance
\end{verbatim}

Let's compare these to the ones from the previous model:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{diag\_lm }\OtherTok{\textless{}{-}} \FunctionTok{glance}\NormalTok{(model3)}

\NormalTok{diag\_lm }\OtherTok{\textless{}{-}}\NormalTok{ diag\_lm }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{model =} \StringTok{"lin{-}lin model"}\NormalTok{)}

\NormalTok{diag\_log }\OtherTok{\textless{}{-}} \FunctionTok{glance}\NormalTok{(model\_log)}

\NormalTok{diag\_log  }\OtherTok{\textless{}{-}}\NormalTok{ diag\_log }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{model =} \StringTok{"log{-}lin model"}\NormalTok{)}

\NormalTok{diagnostics\_models }\OtherTok{\textless{}{-}} \FunctionTok{full\_join}\NormalTok{(diag\_lm, diag\_log) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{select}\NormalTok{(model, }\FunctionTok{everything}\NormalTok{()) }\CommentTok{\# put the \textasciigrave{}model\textasciigrave{} column first}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Joining, by = c("r.squared", "adj.r.squared", "sigma", "statistic", "p.value", "df", "logLik", "AIC", "BIC", "deviance",
## "df.residual", "nobs", "model")
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{print}\NormalTok{(diagnostics\_models)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 2 x 13
##   model   r.squ~1 adj.r~2   sigma stati~3   p.value    df  logLik    AIC     BIC
##   <chr>     <dbl>   <dbl>   <dbl>   <dbl>     <dbl> <dbl>   <dbl>  <dbl>   <dbl>
## 1 lin-li~   0.673   0.666 1.54e+4    100. 6.18e-122    11 -6034.  12094. 12150. 
## 2 log-li~   0.677   0.670 2.14e-1    102. 3.67e-123    11    73.9  -122.   -65.8
## # ... with 3 more variables: deviance <dbl>, df.residual <int>, nobs <int>, and
## #   abbreviated variable names 1: r.squared, 2: adj.r.squared, 3: statistic
\end{verbatim}

I saved the diagnostics in two different \texttt{data.frame} objects using the \texttt{glance()} function and added a
\texttt{model} column to indicate which model the diagnostics come from. Then I merged both datasets using
\texttt{full\_join()}, a \texttt{\{dplyr\}} function. Using this approach, we can easily build a data frame with the
diagnostics of several models and compare them. The model using the logarithm of prices has lower
AIC and BIC (and this higher likelihood), so if you're worried about selecting the model with the better
fit to the data, you'd go for this model.

\hypertarget{using-a-model-for-prediction}{%
\subsection{Using a model for prediction}\label{using-a-model-for-prediction}}

Once you estimated a model, you might want to use it for prediction. This is easily done using the
\texttt{predict()} function that works with most models. Prediction is also useful as a way to test the
accuracy of your model: split your data into a training set (used for training) and a testing
set (used for the pseudo-prediction) and see if your model overfits the data. We are going to see
how to do that in a later section; for now, let's just get acquainted with \texttt{predict()} and other
functions. I insist, keep in mind that this section is only to get acquainted with these functions.
We are going to explore prediction, overfitting and tuning of models in a later section.

Let's go back to the models we trained in the previous section, \texttt{model3} and \texttt{model\_log}. Let's also
take a subsample of data, which we will be using for prediction:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{set.seed}\NormalTok{(}\DecValTok{1234}\NormalTok{)}

\NormalTok{pred\_set }\OtherTok{\textless{}{-}}\NormalTok{ Housing }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{sample\_n}\NormalTok{(}\DecValTok{20}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

In order to always get the same \texttt{pred\_set}, I set the random seed first. Let's take a look at the
data:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{print}\NormalTok{(pred\_set)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##      price lotsize bedrooms bathrms stories driveway recroom fullbase gashw
## 284  45000    6750        2       1       1      yes      no       no    no
## 101  57000    4500        3       2       2       no      no      yes    no
## 400  85000    7231        3       1       2      yes     yes      yes    no
## 98   59900    8250        3       1       1      yes      no      yes    no
## 103 125000    4320        3       1       2      yes      no      yes   yes
## 326  99000    8880        3       2       2      yes      no      yes    no
## 79   55000    3180        2       2       1      yes      no      yes    no
## 270  59000    4632        4       1       2      yes      no       no    no
## 382 112500    6550        3       1       2      yes      no      yes    no
## 184  63900    3510        3       1       2      yes      no       no    no
## 4    60500    6650        3       1       2      yes     yes       no    no
## 212  42000    2700        2       1       1       no      no       no    no
## 195  33000    3180        2       1       1      yes      no       no    no
## 511  70000    4646        3       1       2      yes     yes      yes    no
## 479  88000    5450        4       2       1      yes      no      yes    no
## 510  64000    4040        3       1       2      yes      no       no    no
## 424  62900    2880        3       1       2      yes      no       no    no
## 379  84000    7160        3       1       1      yes      no      yes    no
## 108  58500    3680        3       2       2      yes      no       no    no
## 131  35000    4840        2       1       2      yes      no       no    no
##     airco garagepl prefarea
## 284    no        0       no
## 101   yes        0       no
## 400   yes        0      yes
## 98     no        3       no
## 103    no        2       no
## 326   yes        1       no
## 79     no        2       no
## 270   yes        0       no
## 382   yes        0      yes
## 184    no        0       no
## 4      no        0       no
## 212    no        0       no
## 195    no        0       no
## 511    no        2       no
## 479   yes        0      yes
## 510    no        1       no
## 424    no        0      yes
## 379    no        2      yes
## 108    no        0       no
## 131    no        0       no
\end{verbatim}

If we wish to use it for prediction, this is easily done with \texttt{predict()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{predict}\NormalTok{(model3, pred\_set)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##       284       101       400        98       103       326        79       270 
##  51143.48  77286.31  93204.28  76481.82  77688.37 103751.72  66760.79  66486.26 
##       382       184         4       212       195       511       479       510 
##  86277.96  48042.41  63689.09  30093.18  38483.18  70524.34  91987.65  54166.78 
##       424       379       108       131 
##  55177.75  77741.03  62980.84  50926.99
\end{verbatim}

This returns a vector of predicted prices. This can then be used to compute the Root Mean Squared Error
for instance. Let's do it within a \texttt{tidyverse} pipeline:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{rmse }\OtherTok{\textless{}{-}}\NormalTok{ pred\_set }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{predictions =} \FunctionTok{predict}\NormalTok{(model3, .)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\FunctionTok{sqrt}\NormalTok{(}\FunctionTok{sum}\NormalTok{(predictions }\SpecialCharTok{{-}}\NormalTok{ price)}\SpecialCharTok{**}\DecValTok{2}\SpecialCharTok{/}\FunctionTok{n}\NormalTok{()))}
\end{Highlighting}
\end{Shaded}

The root mean square error of \texttt{model3} is 3646.08173472198.

I also used the \texttt{n()} function which returns the number of observations in a group (or all the
observations, if the data is not grouped). Let's compare \texttt{model3} 's RMSE with the one from
\texttt{model\_log}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{rmse2 }\OtherTok{\textless{}{-}}\NormalTok{ pred\_set }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{predictions =} \FunctionTok{exp}\NormalTok{(}\FunctionTok{predict}\NormalTok{(model\_log, .))) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\FunctionTok{sqrt}\NormalTok{(}\FunctionTok{sum}\NormalTok{(predictions }\SpecialCharTok{{-}}\NormalTok{ price)}\SpecialCharTok{**}\DecValTok{2}\SpecialCharTok{/}\FunctionTok{n}\NormalTok{()))}
\end{Highlighting}
\end{Shaded}

Don't forget to exponentiate the predictions, remember you're dealing with a log-linear model! \texttt{model\_log}'s
RMSE is 12125.1329518719 which is lower than \texttt{model3}'s. However, keep in mind that the model was trained
on the whole data, and then the prediction quality was assessed using a subsample of the data the
model was trained on\ldots{} so actually we can't really say if \texttt{model\_log}'s predictions are very useful.
Of course, this is the same for \texttt{model3}.
In a later section we are going to learn how to do cross validation to avoid this issue.

Just as a side note, notice that I had to copy and paste basically the same lines twice to compute
the predictions for both models. That's not much, but if I wanted to compare 10 models, copy and
paste mistakes could have sneaked in. Instead, it would have been nice to have a function that
computes the RMSE and then use it on my models. We are going to learn how to write our own functions
and use it just like if it was another built-in R function.

\hypertarget{beyond-linear-regression}{%
\subsection{Beyond linear regression}\label{beyond-linear-regression}}

R has a lot of other built-in functions for regression, such as \texttt{glm()} (for Generalized Linear
Models) and \texttt{nls()} for (for Nonlinear Least Squares). There are also functions and additional
packages for time series, panel data, machine learning, bayesian and nonparametric methods.
Presenting everything here would take too much space, and would be pretty useless as you can find
whatever you need using an internet search engine. What you have learned until now is quite general
and should work on many type of models. To help you out, here is a list of methods and the
recommended packages that you can use:

\begin{longtable}[]{@{}lll@{}}
\toprule()
Model & Package & Quick example \\
\midrule()
\endhead
Robust Linear Regression & \texttt{MASS} & \texttt{rlm(y\ \textasciitilde{}\ x,\ data\ =\ mydata)} \\
Nonlinear Least Squares & \texttt{stats}\footnote{This package gets installed with R, no need to add it} & \texttt{nls(y\ \textasciitilde{}\ x1\ /\ (1\ +\ x2),\ data\ =\ mydata)}\footnote{The formula in the example is shown for illustration purposes.} \\
Logit & \texttt{stats} & \texttt{glm(y\ \textasciitilde{}\ x,\ data\ =\ mydata,\ family\ =\ "binomial")} \\
Probit & \texttt{stats} & \texttt{glm(y\ \textasciitilde{}\ x,\ data\ =\ mydata,\ family\ =\ binomial(link\ =\ "probit"))} \\
K-Means & \texttt{stats} & \texttt{kmeans(data,\ n)}\footnote{\texttt{data} must only contain numeric values, and \texttt{n} is the number of clusters.} \\
PCA & \texttt{stats} & \texttt{prcomp(data,\ scale\ =\ TRUE,\ center\ =\ TRUE)}\footnote{\texttt{data} must only contain numeric values, or a formula can be provided.} \\
Multinomial Logit & \texttt{mlogit} & Requires several steps of data pre-processing and formula definition, refer to the \href{https://cran.r-project.org/web/packages/mlogit/vignettes/mlogit.pdf}{Vignette} for more details. \\
Cox PH & \texttt{survival} & \texttt{coxph(Surv(y\_time,\ y\_status)\ \textasciitilde{}\ x,\ data\ =\ mydata)}\footnote{\texttt{Surv(y\_time,\ y\_status)} creates a \emph{survival} object, where \texttt{y\_time} is the time to event \texttt{y\_status}. It is possible to create more complex survival objects depending on exactly which data you are dealing with.} \\
Time series & Several, depending on your needs. & Time series in R is a vast subject that would require a very thick book to cover. You can get started with the following series of blog articles, \href{http://www.business-science.io/timeseries-analysis/2017/07/02/tidy-timeseries-analysis.html}{Tidy time-series, part 1}, \href{http://www.business-science.io/timeseries-analysis/2017/07/23/tidy-timeseries-analysis-pt-2.html}{Tidy time-series, part 2}, \href{http://www.business-science.io/timeseries-analysis/2017/07/30/tidy-timeseries-analysis-pt-3.html}{Tidy time-series, part 3} and \href{http://www.business-science.io/timeseries-analysis/2017/08/30/tidy-timeseries-analysis-pt-4.html}{Tidy time-series, part 4} \\
Panel data & \texttt{plm} & \texttt{plm(y\ \textasciitilde{}\ x,\ data\ =\ mydata,\ model\ =\ "within\textbar{}random")} \\
Machine learning & Several, depending on your needs. & R is a very popular programming language for machine learning. \href{https://www.tmwr.org/}{This book} is a must read if you need to do machine learning with R. \\
Nonparametric regression & \texttt{np} & Several functions and options available, refer to the \href{https://cran.r-project.org/web/packages/np/vignettes/np.pdf}{Vignette} for more details. \\
\bottomrule()
\end{longtable}

This table is far from being complete. Should you be a Bayesian, you'd want to look at packages
such as \texttt{\{rstan\}}, which uses \texttt{STAN}, an external piece of software that must be installed on your
system. It is also possible to train models using Bayesian inference without the need of external
tools, with the \texttt{\{bayesm\}} package which estimates the usual micro-econometric models. There really
are a lot of packages available for Bayesian inference, and you can find them all in the \href{https://cran.r-project.org/web/views/Bayesian.html}{related
CRAN Task View}.

\hypertarget{hyper-parameters}{%
\subsection{Hyper-parameters}\label{hyper-parameters}}

Hyper-parameters are parameters of the model that cannot be directly learned from the data.
A linear regression does not have any hyper-parameters, but a random forest for instance has several.
You might have heard of ridge regression, lasso and elasticnet. These are
extensions of linear models that avoid over-fitting by penalizing \emph{large} models. These
extensions of the linear regression have hyper-parameters that the practitioner has to tune. There
are several ways one can tune these parameters, for example, by doing a grid-search, or a random
search over the grid or using more elaborate methods. To introduce hyper-parameters, let's get
to know ridge regression, also called Tikhonov regularization.

\hypertarget{ridge-regression}{%
\subsubsection{Ridge regression}\label{ridge-regression}}

Ridge regression is used when the data you are working with has a lot of explanatory variables,
or when there is a risk that a simple linear regression might overfit to the training data, because,
for example, your explanatory variables are collinear.
If you are training a linear model and then you notice that it generalizes very badly to new,
unseen data, it is very likely that the linear model you trained overfit the data.
In this case, ridge regression might prove useful. The way ridge regression works might seem
counter-intuititive; it boils down to fitting a \emph{worse} model to the training data, but in return,
this worse model will generalize better to new data.

The closed form solution of the ordinary least squares estimator is defined as:

\[
\widehat{\beta} = (X'X)^{-1}X'Y
\]

where \(X\) is the design matrix (the matrix made up of the explanatory variables) and \(Y\) is the
dependent variable. For ridge regression, this closed form solution changes a little bit:

\[
\widehat{\beta} = (X'X + \lambda I_p)^{-1}X'Y
\]

where \(\lambda \in \mathbb{R}\) is an hyper-parameter and \(I_p\) is the identity matrix of dimension \(p\)
(\(p\) is the number of explanatory variables).
This formula above is the closed form solution to the following optimisation program:

\[
\sum_{i=1}^n \left(y_i - \sum_{j=1}^px_{ij}\beta_j\right)^2 
\]

such that:

\[
\sum_{j=1}^p(\beta_j)^2 < c
\]

for any strictly positive \(c\).

The \texttt{glmnet()} function from the \texttt{\{glmnet\}} package can be used for ridge regression, by setting
the \texttt{alpha} argument to 0 (setting it to 1 would do LASSO, and setting it to a number between
0 and 1 would do elasticnet). But in order to compare linear regression and ridge regression,
let me first divide the data into a training set and a testing set:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{index }\OtherTok{\textless{}{-}} \DecValTok{1}\SpecialCharTok{:}\FunctionTok{nrow}\NormalTok{(Housing)}

\FunctionTok{set.seed}\NormalTok{(}\DecValTok{12345}\NormalTok{)}
\NormalTok{train\_index }\OtherTok{\textless{}{-}} \FunctionTok{sample}\NormalTok{(index, }\FunctionTok{round}\NormalTok{(}\FloatTok{0.90}\SpecialCharTok{*}\FunctionTok{nrow}\NormalTok{(Housing)), }\AttributeTok{replace =} \ConstantTok{FALSE}\NormalTok{)}

\NormalTok{test\_index }\OtherTok{\textless{}{-}} \FunctionTok{setdiff}\NormalTok{(index, train\_index)}

\NormalTok{train\_x }\OtherTok{\textless{}{-}}\NormalTok{ Housing[train\_index, ] }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{select}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{price)}

\NormalTok{train\_y }\OtherTok{\textless{}{-}}\NormalTok{ Housing[train\_index, ] }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{pull}\NormalTok{(price)}

\NormalTok{test\_x }\OtherTok{\textless{}{-}}\NormalTok{ Housing[test\_index, ] }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{select}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{price)}

\NormalTok{test\_y }\OtherTok{\textless{}{-}}\NormalTok{ Housing[test\_index, ] }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{pull}\NormalTok{(price)}
\end{Highlighting}
\end{Shaded}

I do the train/test split this way, because \texttt{glmnet()} requires a design matrix as input, and not
a formula. Design matrices can be created using the \texttt{model.matrix()} function:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(}\StringTok{"glmnet"}\NormalTok{)}

\NormalTok{train\_matrix }\OtherTok{\textless{}{-}} \FunctionTok{model.matrix}\NormalTok{(train\_y }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ train\_x)}

\NormalTok{test\_matrix }\OtherTok{\textless{}{-}} \FunctionTok{model.matrix}\NormalTok{(test\_y }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ test\_x)}
\end{Highlighting}
\end{Shaded}

Let's now run a linear regression, by setting the penalty to 0:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model\_lm\_ridge }\OtherTok{\textless{}{-}} \FunctionTok{glmnet}\NormalTok{(}\AttributeTok{y =}\NormalTok{ train\_y, }\AttributeTok{x =}\NormalTok{ train\_matrix, }\AttributeTok{alpha =} \DecValTok{0}\NormalTok{, }\AttributeTok{lambda =} \DecValTok{0}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

The model above provides the same result as a linear regression, because I set \texttt{lambda} to 0. Let's
compare the coefficients between the two:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{coef}\NormalTok{(model\_lm\_ridge)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 13 x 1 sparse Matrix of class "dgCMatrix"
##                       s0
## (Intercept) -2667.542863
## (Intercept)     .       
## lotsize         3.397596
## bedrooms     2081.087654
## bathrms     13294.192823
## stories      6400.454580
## drivewayyes  6530.644895
## recroomyes   5389.856794
## fullbaseyes  4899.099463
## gashwyes    12575.611265
## aircoyes    13078.144146
## garagepl     4155.249461
## prefareayes 10260.781753
\end{verbatim}

and now the coefficients of the linear regression (because I provide a design matrix, I have to use
\texttt{lm.fit()} instead of \texttt{lm()} which requires a formula, not a matrix.)

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{coef}\NormalTok{(}\FunctionTok{lm.fit}\NormalTok{(}\AttributeTok{x =}\NormalTok{ train\_matrix, }\AttributeTok{y =}\NormalTok{ train\_y))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  (Intercept)      lotsize     bedrooms      bathrms      stories  drivewayyes 
## -2667.052098     3.397629  2081.344118 13293.707725  6400.416730  6529.972544 
##   recroomyes  fullbaseyes     gashwyes     aircoyes     garagepl  prefareayes 
##  5388.871137  4899.024787 12575.970220 13077.988867  4155.269629 10261.056772
\end{verbatim}

as you can see, the coefficients are the same. Let's compute the RMSE for the unpenalized linear
regression:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{preds\_lm }\OtherTok{\textless{}{-}} \FunctionTok{predict}\NormalTok{(model\_lm\_ridge, test\_matrix)}

\NormalTok{rmse\_lm }\OtherTok{\textless{}{-}} \FunctionTok{sqrt}\NormalTok{(}\FunctionTok{mean}\NormalTok{(preds\_lm }\SpecialCharTok{{-}}\NormalTok{ test\_y)}\SpecialCharTok{\^{}}\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

The RMSE for the linear unpenalized regression is equal to 1731.5553157.

Let's now run a ridge regression, with \texttt{lambda} equal to 100, and see if the RMSE is smaller:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{model\_ridge }\OtherTok{\textless{}{-}} \FunctionTok{glmnet}\NormalTok{(}\AttributeTok{y =}\NormalTok{ train\_y, }\AttributeTok{x =}\NormalTok{ train\_matrix, }\AttributeTok{alpha =} \DecValTok{0}\NormalTok{, }\AttributeTok{lambda =} \DecValTok{100}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

and let's compute the RMSE again:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{preds }\OtherTok{\textless{}{-}} \FunctionTok{predict}\NormalTok{(model\_ridge, test\_matrix)}

\NormalTok{rmse }\OtherTok{\textless{}{-}} \FunctionTok{sqrt}\NormalTok{(}\FunctionTok{mean}\NormalTok{(preds }\SpecialCharTok{{-}}\NormalTok{ test\_y)}\SpecialCharTok{\^{}}\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

The RMSE for the linear penalized regression is equal to 1726.7632312, which is smaller than before.
But which value of \texttt{lambda} gives smallest RMSE? To find out, one must run model over a grid of
\texttt{lambda} values and pick the model with lowest RMSE. This procedure is available in the \texttt{cv.glmnet()}
function, which picks the best value for \texttt{lambda}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{best\_model }\OtherTok{\textless{}{-}} \FunctionTok{cv.glmnet}\NormalTok{(train\_matrix, train\_y)}
\CommentTok{\# lambda that minimises the MSE}
\NormalTok{best\_model}\SpecialCharTok{$}\NormalTok{lambda.min}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 61.42681
\end{verbatim}

According to \texttt{cv.glmnet()} the best value for \texttt{lambda} is 61.4268056. In the
next section, we will implement cross validation ourselves, in order to find the hyper-parameters
of a random forest.

\hypertarget{training-validating-and-testing-models}{%
\subsection{Training, validating, and testing models}\label{training-validating-and-testing-models}}

Cross-validation is an important procedure which is used to compare models but also to tune the
hyper-parameters of a model. In this section, we are going to use several packages from the
\href{https://github.com/tidymodels}{\texttt{\{tidymodels\}}} collection of packages, namely
\href{https://tidymodels.github.io/recipes/}{\texttt{\{recipes\}}},
\href{https://tidymodels.github.io/rsample/}{\texttt{\{rsample\}}} and
\href{https://tidymodels.github.io/parsnip/}{\texttt{\{parsnip\}}} to train a random forest the tidy way. I will
also use \href{http://mlrmbo.mlr-org.com/}{\texttt{\{mlrMBO\}}} to tune the hyper-parameters of the random forest.

\hypertarget{set-up}{%
\subsubsection{Set up}\label{set-up}}

Let's load the needed packages:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(}\StringTok{"tidyverse"}\NormalTok{)}
\FunctionTok{library}\NormalTok{(}\StringTok{"recipes"}\NormalTok{)}
\FunctionTok{library}\NormalTok{(}\StringTok{"rsample"}\NormalTok{)}
\FunctionTok{library}\NormalTok{(}\StringTok{"parsnip"}\NormalTok{)}
\FunctionTok{library}\NormalTok{(}\StringTok{"yardstick"}\NormalTok{)}
\FunctionTok{library}\NormalTok{(}\StringTok{"brotools"}\NormalTok{)}
\FunctionTok{library}\NormalTok{(}\StringTok{"mlbench"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Load the data which is included in the \texttt{\{mlrbench\}} package:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(}\StringTok{"BostonHousing2"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

I will train a random forest to predict the housing prices, which is the \texttt{cmedv} column:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{head}\NormalTok{(BostonHousing2)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##         town tract      lon     lat medv cmedv    crim zn indus chas   nox
## 1     Nahant  2011 -70.9550 42.2550 24.0  24.0 0.00632 18  2.31    0 0.538
## 2 Swampscott  2021 -70.9500 42.2875 21.6  21.6 0.02731  0  7.07    0 0.469
## 3 Swampscott  2022 -70.9360 42.2830 34.7  34.7 0.02729  0  7.07    0 0.469
## 4 Marblehead  2031 -70.9280 42.2930 33.4  33.4 0.03237  0  2.18    0 0.458
## 5 Marblehead  2032 -70.9220 42.2980 36.2  36.2 0.06905  0  2.18    0 0.458
## 6 Marblehead  2033 -70.9165 42.3040 28.7  28.7 0.02985  0  2.18    0 0.458
##      rm  age    dis rad tax ptratio      b lstat
## 1 6.575 65.2 4.0900   1 296    15.3 396.90  4.98
## 2 6.421 78.9 4.9671   2 242    17.8 396.90  9.14
## 3 7.185 61.1 4.9671   2 242    17.8 392.83  4.03
## 4 6.998 45.8 6.0622   3 222    18.7 394.63  2.94
## 5 7.147 54.2 6.0622   3 222    18.7 396.90  5.33
## 6 6.430 58.7 6.0622   3 222    18.7 394.12  5.21
\end{verbatim}

Only keep relevant columns:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{boston }\OtherTok{\textless{}{-}}\NormalTok{ BostonHousing2 }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{select}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{medv, }\SpecialCharTok{{-}}\NormalTok{tract, }\SpecialCharTok{{-}}\NormalTok{lon, }\SpecialCharTok{{-}}\NormalTok{lat) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{rename}\NormalTok{(}\AttributeTok{price =}\NormalTok{ cmedv)}
\end{Highlighting}
\end{Shaded}

I remove \texttt{tract}, \texttt{lat} and \texttt{lon} because the information contained in the column \texttt{town} is enough.

To train and evaluate the model's performance, I split the data in two.
One data set, called the training set, will be further split into two down below. I won't
touch the second data set, the test set, until the very end, to finally assess the model's
performance.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{train\_test\_split }\OtherTok{\textless{}{-}} \FunctionTok{initial\_split}\NormalTok{(boston, }\AttributeTok{prop =} \FloatTok{0.9}\NormalTok{)}

\NormalTok{housing\_train }\OtherTok{\textless{}{-}} \FunctionTok{training}\NormalTok{(train\_test\_split)}

\NormalTok{housing\_test }\OtherTok{\textless{}{-}} \FunctionTok{testing}\NormalTok{(train\_test\_split)}
\end{Highlighting}
\end{Shaded}

\texttt{initial\_split()}, \texttt{training()} and \texttt{testing()} are functions from the \texttt{\{rsample\}} package.

I will train a random forest on the training data, but the question, is \emph{which} random forest?
Because random forests have several hyper-parameters, and as explained in the intro these
hyper-parameters cannot be directly learned from the data, which one should we choose? We could
train 6 random forests for instance and compare their performance, but why only 6? Why not 16?

In order to find the right hyper-parameters, the practitioner can
use values from the literature that seemed to have worked well (like is done in Macro-econometrics)
or you can further split the train set into two, create a grid of hyperparameter, train the model
on one part of the data for all values of the grid, and compare the predictions of the models on the
second part of the data. You then stick with the model that performed the best, for example, the
model with lowest RMSE. The thing is, you can't estimate the true value of the RMSE with only
one value. It's like if you wanted to estimate the height of the population by drawing one single
observation from the population. You need a bit more observations. To approach the true value of the
RMSE for a give set of hyperparameters, instead of doing one split, let's do 30. Then we
compute the average RMSE, which implies training 30 models for each combination of the values of the
hyperparameters.

First, let's split the training data again, using the \texttt{mc\_cv()} function from \texttt{\{rsample\}} package.
This function implements Monte Carlo cross-validation:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{validation\_data }\OtherTok{\textless{}{-}} \FunctionTok{mc\_cv}\NormalTok{(housing\_train, }\AttributeTok{prop =} \FloatTok{0.9}\NormalTok{, }\AttributeTok{times =} \DecValTok{30}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

What does \texttt{validation\_data} look like?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{validation\_data}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # Monte Carlo cross-validation (0.9/0.1) with 30 resamples  
## # A tibble: 30 x 2
##    splits           id        
##    <list>           <chr>     
##  1 <split [409/46]> Resample01
##  2 <split [409/46]> Resample02
##  3 <split [409/46]> Resample03
##  4 <split [409/46]> Resample04
##  5 <split [409/46]> Resample05
##  6 <split [409/46]> Resample06
##  7 <split [409/46]> Resample07
##  8 <split [409/46]> Resample08
##  9 <split [409/46]> Resample09
## 10 <split [409/46]> Resample10
## # ... with 20 more rows
\end{verbatim}

Let's look further down:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{validation\_data}\SpecialCharTok{$}\NormalTok{splits[[}\DecValTok{1}\NormalTok{]]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## <Analysis/Assess/Total>
## <409/46/455>
\end{verbatim}

The first value is the number of rows of the first set, the second value of the second, and the third
was the original amount of values in the training data, before splitting again.

How should we call these two new data sets? The author of \texttt{\{rsample\}}, Max Kuhn, talks about
the \emph{analysis} and the \emph{assessment} sets, and I'm going to use this terminology as well.

Now, in order to continue I need to pre-process the data. I will do this in three steps.
The first and the second steps are used to center and scale the numeric variables and the third step
converts character and factor variables to dummy variables. This is needed because I will train a
random forest, which cannot handle factor variables directly. Let's define a recipe to do that,
and start by pre-processing the testing set. I write a wrapper function around the recipe,
because I will need to apply this recipe to various data sets:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simple\_recipe }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataset)\{}
    \FunctionTok{recipe}\NormalTok{(price }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ dataset) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{step\_center}\NormalTok{(}\FunctionTok{all\_numeric}\NormalTok{()) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{step\_scale}\NormalTok{(}\FunctionTok{all\_numeric}\NormalTok{()) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{step\_dummy}\NormalTok{(}\FunctionTok{all\_nominal}\NormalTok{())}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

We have not learned yet about writing functions, and will do so in the next chapter. However, for
now, you only need to know that you can write your own functions, and that these functions can
take any arguments you need. In the case of the above function, which we called \texttt{simple\_recipe()},
we only need one argument, which is a dataset, and which we called \texttt{dataset}.

Once the recipe is defined, I can use the \texttt{prep()} function, which estimates the parameters from
the data which are needed to process the data. For example, for centering, \texttt{prep()} estimates
the mean which will then be subtracted from the variables. With \texttt{bake()} the estimates are then
applied on the data:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{testing\_rec }\OtherTok{\textless{}{-}} \FunctionTok{prep}\NormalTok{(}\FunctionTok{simple\_recipe}\NormalTok{(housing\_test), }\AttributeTok{testing =}\NormalTok{ housing\_test)}

\NormalTok{test\_data }\OtherTok{\textless{}{-}} \FunctionTok{bake}\NormalTok{(testing\_rec, }\AttributeTok{new\_data =}\NormalTok{ housing\_test)}
\end{Highlighting}
\end{Shaded}

It is important to split the data before using \texttt{prep()} and \texttt{bake()}, because if not, you will
use observations from the test set in the \texttt{prep()} step, and thus introduce knowledge from the test
set into the training data. This is called data leakage, and must be avoided. This is why it is
necessary to first split the training data into an analysis and an assessment set, and then also
pre-process these sets separately. However, the \texttt{validation\_data} object cannot now be used with
\texttt{recipe()}, because it is not a dataframe. No worries, I simply need to write a function that extracts
the analysis and assessment sets from the \texttt{validation\_data} object, applies the pre-processing, trains
the model, and returns the RMSE. This will be a big function, at the center of the analysis.

But before that, let's run a simple linear regression, as a benchmark. For the linear regression, I will
not use any CV, so let's pre-process the training set:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{trainlm\_rec }\OtherTok{\textless{}{-}} \FunctionTok{prep}\NormalTok{(}\FunctionTok{simple\_recipe}\NormalTok{(housing\_train), }\AttributeTok{testing =}\NormalTok{ housing\_train)}

\NormalTok{trainlm\_data }\OtherTok{\textless{}{-}} \FunctionTok{bake}\NormalTok{(trainlm\_rec, }\AttributeTok{new\_data =}\NormalTok{ housing\_train)}

\NormalTok{linreg\_model }\OtherTok{\textless{}{-}} \FunctionTok{lm}\NormalTok{(price }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ trainlm\_data)}

\NormalTok{broom}\SpecialCharTok{::}\FunctionTok{augment}\NormalTok{(linreg\_model, }\AttributeTok{newdata =}\NormalTok{ test\_data) }\SpecialCharTok{\%\textgreater{}\%} 
\NormalTok{    yardstick}\SpecialCharTok{::}\FunctionTok{rmse}\NormalTok{(price, .fitted)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning in predict.lm(x, newdata = newdata, na.action = na.pass, ...):
## prediction from a rank-deficient fit may be misleading
\end{verbatim}

\begin{verbatim}
## # A tibble: 1 x 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 rmse    standard       0.439
\end{verbatim}

\texttt{broom::augment()} adds the predictions to the \texttt{test\_data} in a new column, \texttt{.fitted}. I won't
use this trick with the random forest, because there is no \texttt{augment()} method for random forests
from the \texttt{\{ranger\}} package which I'll use. I'll add the predictions to the data myself.

Ok, now let's go back to the random forest and write the big function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_rf }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(mtry, trees, split, id)\{}
    
\NormalTok{    analysis\_set }\OtherTok{\textless{}{-}} \FunctionTok{analysis}\NormalTok{(split)}
    
\NormalTok{    analysis\_prep }\OtherTok{\textless{}{-}} \FunctionTok{prep}\NormalTok{(}\FunctionTok{simple\_recipe}\NormalTok{(analysis\_set), }\AttributeTok{training =}\NormalTok{ analysis\_set)}
    
\NormalTok{    analysis\_processed }\OtherTok{\textless{}{-}} \FunctionTok{bake}\NormalTok{(analysis\_prep, }\AttributeTok{new\_data =}\NormalTok{ analysis\_set)}
    
\NormalTok{    model }\OtherTok{\textless{}{-}} \FunctionTok{rand\_forest}\NormalTok{(}\AttributeTok{mode =} \StringTok{"regression"}\NormalTok{, }\AttributeTok{mtry =}\NormalTok{ mtry, }\AttributeTok{trees =}\NormalTok{ trees) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{set\_engine}\NormalTok{(}\StringTok{"ranger"}\NormalTok{, }\AttributeTok{importance =} \StringTok{\textquotesingle{}impurity\textquotesingle{}}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{fit}\NormalTok{(price }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ analysis\_processed)}

\NormalTok{    assessment\_set }\OtherTok{\textless{}{-}} \FunctionTok{assessment}\NormalTok{(split)}
    
\NormalTok{    assessment\_prep }\OtherTok{\textless{}{-}} \FunctionTok{prep}\NormalTok{(}\FunctionTok{simple\_recipe}\NormalTok{(assessment\_set), }\AttributeTok{testing =}\NormalTok{ assessment\_set)}
    
\NormalTok{    assessment\_processed }\OtherTok{\textless{}{-}} \FunctionTok{bake}\NormalTok{(assessment\_prep, }\AttributeTok{new\_data =}\NormalTok{ assessment\_set)}

\NormalTok{    tibble}\SpecialCharTok{::}\FunctionTok{tibble}\NormalTok{(}\StringTok{"id"} \OtherTok{=}\NormalTok{ id,}
        \StringTok{"truth"} \OtherTok{=}\NormalTok{ assessment\_processed}\SpecialCharTok{$}\NormalTok{price,}
        \StringTok{"prediction"} \OtherTok{=} \FunctionTok{unlist}\NormalTok{(}\FunctionTok{predict}\NormalTok{(model, }\AttributeTok{new\_data =}\NormalTok{ assessment\_processed)))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

The \texttt{rand\_forest()} function is available in the \texttt{\{parsnip\}} package. This package provides an
unified interface to a lot of other machine learning packages. This means that instead of having to
learn the syntax of \texttt{range()} and \texttt{randomForest()} and, and\ldots{} you can simply use the \texttt{rand\_forest()}
function and change the \texttt{engine} argument to the one you want (\texttt{ranger}, \texttt{randomForest}, etc).

Let's try this function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{results\_example }\OtherTok{\textless{}{-}} \FunctionTok{map2\_df}\NormalTok{(}\AttributeTok{.x =}\NormalTok{ validation\_data}\SpecialCharTok{$}\NormalTok{splits,}
                           \AttributeTok{.y =}\NormalTok{ validation\_data}\SpecialCharTok{$}\NormalTok{id,}
                           \SpecialCharTok{\textasciitilde{}}\FunctionTok{my\_rf}\NormalTok{(}\AttributeTok{mtry =} \DecValTok{3}\NormalTok{, }\AttributeTok{trees =} \DecValTok{200}\NormalTok{, }\AttributeTok{split =}\NormalTok{ .x, }\AttributeTok{id =}\NormalTok{ .y))}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{head}\NormalTok{(results\_example)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 3
##   id           truth prediction
##   <chr>        <dbl>      <dbl>
## 1 Resample01 -0.328     -0.0274
## 2 Resample01  1.06       0.686 
## 3 Resample01  1.04       0.726 
## 4 Resample01 -0.418     -0.0190
## 5 Resample01  0.909      0.642 
## 6 Resample01  0.0926    -0.134
\end{verbatim}

I can now compute the RMSE when \texttt{mtry} = 3 and \texttt{trees} = 200:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{results\_example }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(id) }\SpecialCharTok{\%\textgreater{}\%}
\NormalTok{    yardstick}\SpecialCharTok{::}\FunctionTok{rmse}\NormalTok{(truth, prediction) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_rmse =} \FunctionTok{mean}\NormalTok{(.estimate)) }\SpecialCharTok{\%\textgreater{}\%}
\NormalTok{    pull}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.6305034
\end{verbatim}

The random forest has already lower RMSE than the linear regression. The goal now is to lower this
RMSE by tuning the \texttt{mtry} and \texttt{trees} hyperparameters. For this, I will use Bayesian Optimization
methods implemented in the \texttt{\{mlrMBO\}} package.

\hypertarget{bayesian-hyperparameter-optimization}{%
\subsubsection{Bayesian hyperparameter optimization}\label{bayesian-hyperparameter-optimization}}

I will re-use the code from above, and define a function that does everything from pre-processing
to returning the metric I want to minimize by tuning the hyperparameters, the RMSE:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{tuning }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(param, validation\_data)\{}

\NormalTok{    mtry }\OtherTok{\textless{}{-}}\NormalTok{ param[}\DecValTok{1}\NormalTok{]}
\NormalTok{    trees }\OtherTok{\textless{}{-}}\NormalTok{ param[}\DecValTok{2}\NormalTok{]}

\NormalTok{    results }\OtherTok{\textless{}{-}}\NormalTok{ purrr}\SpecialCharTok{::}\FunctionTok{map2\_df}\NormalTok{(}\AttributeTok{.x =}\NormalTok{ validation\_data}\SpecialCharTok{$}\NormalTok{splits,}
                       \AttributeTok{.y =}\NormalTok{ validation\_data}\SpecialCharTok{$}\NormalTok{id,}
                       \SpecialCharTok{\textasciitilde{}}\FunctionTok{my\_rf}\NormalTok{(}\AttributeTok{mtry =}\NormalTok{ mtry, }\AttributeTok{trees =}\NormalTok{ trees, }\AttributeTok{split =}\NormalTok{ .x, }\AttributeTok{id =}\NormalTok{ .y))}

\NormalTok{    results }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{group\_by}\NormalTok{(id) }\SpecialCharTok{\%\textgreater{}\%}
\NormalTok{        yardstick}\SpecialCharTok{::}\FunctionTok{rmse}\NormalTok{(truth, prediction) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_rmse =} \FunctionTok{mean}\NormalTok{(.estimate)) }\SpecialCharTok{\%\textgreater{}\%}
\NormalTok{        pull}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

This is exactly the code from before, but it now returns the RMSE. Let's try the function
with the values from before:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{tuning}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{, }\DecValTok{200}\NormalTok{), validation\_data)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.6319843
\end{verbatim}

I now follow the code that can be found in the \href{https://arxiv.org/abs/1703.03373}{arxiv} paper to
run the optimization. A simpler model, called the surrogate model, is used to look for promising
points and to evaluate the value of the function at these points. This seems somewhat similar
(in spirit) to the \emph{Indirect Inference} method as described in
\href{https://www.jstor.org/stable/2285076}{Gourieroux, Monfort, Renault}.

If you don't really get what follows, no worries, it is not really important as such. The idea
is simply to look for hyper-parameters in an efficient way, and bayesian optimisation provides
this efficient way. However, you could use another method, for example a grid search. This would not
change anything to the general approach. So I will not spend too much time explaining what is
going on below, as you can read the details in the paper cited above as well as the package's
documentation. The focus here is not on this particular method, but rather showing you how you can
use various packages to solve a data science problem.

Let's first load the package and create the function to optimize:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(}\StringTok{"mlrMBO"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fn }\OtherTok{\textless{}{-}} \FunctionTok{makeSingleObjectiveFunction}\NormalTok{(}\AttributeTok{name =} \StringTok{"tuning"}\NormalTok{,}
                                 \AttributeTok{fn =}\NormalTok{ tuning,}
                                 \AttributeTok{par.set =} \FunctionTok{makeParamSet}\NormalTok{(}\FunctionTok{makeIntegerParam}\NormalTok{(}\StringTok{"x1"}\NormalTok{, }\AttributeTok{lower =} \DecValTok{3}\NormalTok{, }\AttributeTok{upper =} \DecValTok{8}\NormalTok{),}
                                                        \FunctionTok{makeIntegerParam}\NormalTok{(}\StringTok{"x2"}\NormalTok{, }\AttributeTok{lower =} \DecValTok{100}\NormalTok{, }\AttributeTok{upper =} \DecValTok{500}\NormalTok{)))}
\end{Highlighting}
\end{Shaded}

This function is based on the function I defined before. The parameters to optimize are also
defined as are their bounds. I will look for \texttt{mtry} between the values of 3 and 8, and \texttt{trees}
between 50 and 500.

We still need to define some other objects before continuing:

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{\# Create initial random Latin Hypercube Design of 10 points}
\FunctionTok{library}\NormalTok{(lhs)}\CommentTok{\# for randomLHS}
\NormalTok{des }\OtherTok{\textless{}{-}} \FunctionTok{generateDesign}\NormalTok{(}\AttributeTok{n =}\NormalTok{ 5L }\SpecialCharTok{*}\NormalTok{ 2L, }\FunctionTok{getParamSet}\NormalTok{(fn), }\AttributeTok{fun =}\NormalTok{ randomLHS)}
\end{Highlighting}
\end{Shaded}

Then we choose the surrogate model, a random forest too:

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{\# Specify kriging model with standard error estimation}
\NormalTok{surrogate }\OtherTok{\textless{}{-}} \FunctionTok{makeLearner}\NormalTok{(}\StringTok{"regr.ranger"}\NormalTok{, }\AttributeTok{predict.type =} \StringTok{"se"}\NormalTok{, }\AttributeTok{keep.inbag =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Here I define some options:

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{\# Set general controls}
\NormalTok{ctrl }\OtherTok{\textless{}{-}} \FunctionTok{makeMBOControl}\NormalTok{()}
\NormalTok{ctrl }\OtherTok{\textless{}{-}} \FunctionTok{setMBOControlTermination}\NormalTok{(ctrl, }\AttributeTok{iters =}\NormalTok{ 10L)}
\NormalTok{ctrl }\OtherTok{\textless{}{-}} \FunctionTok{setMBOControlInfill}\NormalTok{(ctrl, }\AttributeTok{crit =} \FunctionTok{makeMBOInfillCritEI}\NormalTok{())}
\end{Highlighting}
\end{Shaded}

And this is the optimization part:

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{\# Start optimization}
\NormalTok{result }\OtherTok{\textless{}{-}} \FunctionTok{mbo}\NormalTok{(fn, des, surrogate, ctrl, }\AttributeTok{more.args =} \FunctionTok{list}\NormalTok{(}\StringTok{"validation\_data"} \OtherTok{=}\NormalTok{ validation\_data))}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{result}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Recommended parameters:
## x1=8; x2=314
## Objective: y = 0.484
## 
## Optimization path
## 10 + 10 entries in total, displaying last 10 (or less):
##    x1  x2         y dob eol error.message exec.time            ei error.model
## 11  8 283 0.4855415   1  NA          <NA>     7.353 -3.276847e-04        <NA>
## 12  8 284 0.4852047   2  NA          <NA>     7.321 -3.283713e-04        <NA>
## 13  8 314 0.4839817   3  NA          <NA>     7.703 -3.828517e-04        <NA>
## 14  8 312 0.4841398   4  NA          <NA>     7.633 -2.829713e-04        <NA>
## 15  8 318 0.4841066   5  NA          <NA>     7.692 -2.668354e-04        <NA>
## 16  8 314 0.4845221   6  NA          <NA>     7.574 -1.382333e-04        <NA>
## 17  8 321 0.4843018   7  NA          <NA>     7.693 -3.828924e-05        <NA>
## 18  8 318 0.4868457   8  NA          <NA>     7.696 -8.692828e-07        <NA>
## 19  8 310 0.4862687   9  NA          <NA>     7.594 -1.061185e-07        <NA>
## 20  8 313 0.4878694  10  NA          <NA>     7.628 -5.153015e-07        <NA>
##    train.time prop.type propose.time           se      mean
## 11      0.011 infill_ei        0.450 0.0143886864 0.5075765
## 12      0.011 infill_ei        0.427 0.0090265872 0.4971003
## 13      0.012 infill_ei        0.443 0.0062693960 0.4916927
## 14      0.012 infill_ei        0.435 0.0037308971 0.4878950
## 15      0.012 infill_ei        0.737 0.0024446891 0.4860699
## 16      0.013 infill_ei        0.442 0.0012713838 0.4850705
## 17      0.012 infill_ei        0.444 0.0006371109 0.4847248
## 18      0.013 infill_ei        0.467 0.0002106381 0.4844576
## 19      0.014 infill_ei        0.435 0.0002182254 0.4846214
## 20      0.013 infill_ei        0.748 0.0002971160 0.4847383
\end{verbatim}

So the recommended parameters are 8 for \texttt{mtry} and 314 for \texttt{trees}. The
user can access these recommended parameters with \texttt{result\$x\$x1} and \texttt{result\$x\$x2}.
The value of the RMSE is lower than before, and equals 0.4839817. It can be accessed with
\texttt{result\$y}.
Let's now train the random forest on the training data with this values. First, I pre-process the
training data

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{training\_rec }\OtherTok{\textless{}{-}} \FunctionTok{prep}\NormalTok{(}\FunctionTok{simple\_recipe}\NormalTok{(housing\_train), }\AttributeTok{testing =}\NormalTok{ housing\_train)}

\NormalTok{train\_data }\OtherTok{\textless{}{-}} \FunctionTok{bake}\NormalTok{(training\_rec, }\AttributeTok{new\_data =}\NormalTok{ housing\_train)}
\end{Highlighting}
\end{Shaded}

Let's now train our final model and predict the prices:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{final\_model }\OtherTok{\textless{}{-}} \FunctionTok{rand\_forest}\NormalTok{(}\AttributeTok{mode =} \StringTok{"regression"}\NormalTok{, }\AttributeTok{mtry =}\NormalTok{ result}\SpecialCharTok{$}\NormalTok{x}\SpecialCharTok{$}\NormalTok{x1, }\AttributeTok{trees =}\NormalTok{ result}\SpecialCharTok{$}\NormalTok{x}\SpecialCharTok{$}\NormalTok{x2) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{set\_engine}\NormalTok{(}\StringTok{"ranger"}\NormalTok{, }\AttributeTok{importance =} \StringTok{\textquotesingle{}impurity\textquotesingle{}}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{fit}\NormalTok{(price }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ train\_data)}

\NormalTok{price\_predict }\OtherTok{\textless{}{-}} \FunctionTok{predict}\NormalTok{(final\_model, }\AttributeTok{new\_data =} \FunctionTok{select}\NormalTok{(test\_data, }\SpecialCharTok{{-}}\NormalTok{price))}
\end{Highlighting}
\end{Shaded}

Let's transform the data back and compare the predicted prices to the true ones visually:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{cbind}\NormalTok{(price\_predict }\SpecialCharTok{*} \FunctionTok{sd}\NormalTok{(housing\_train}\SpecialCharTok{$}\NormalTok{price) }\SpecialCharTok{+} \FunctionTok{mean}\NormalTok{(housing\_train}\SpecialCharTok{$}\NormalTok{price), }
\NormalTok{      housing\_test}\SpecialCharTok{$}\NormalTok{price)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##       .pred housing_test$price
## 1  16.76938               13.5
## 2  27.59510               30.8
## 3  23.14952               24.7
## 4  21.92390               21.2
## 5  21.35030               20.0
## 6  23.15809               22.9
## 7  23.00947               23.9
## 8  25.74268               26.6
## 9  24.13122               22.6
## 10 34.97671               43.8
## 11 19.30543               18.8
## 12 18.09146               15.7
## 13 18.82922               19.2
## 14 18.63397               13.3
## 15 19.14438               14.0
## 16 17.05549               15.6
## 17 23.79491               27.0
## 18 20.30125               17.4
## 19 22.99200               23.6
## 20 32.77092               33.3
## 21 31.66258               34.6
## 22 28.79583               34.9
## 23 39.02755               50.0
## 24 23.53336               21.7
## 25 24.66551               24.3
## 26 24.91737               24.0
## 27 25.11847               25.1
## 28 24.42518               23.7
## 29 24.59139               23.7
## 30 24.91760               26.2
## 31 38.73875               43.5
## 32 29.71848               35.1
## 33 36.89490               46.0
## 34 24.04041               26.4
## 35 20.91349               20.3
## 36 21.18602               23.1
## 37 22.57069               22.2
## 38 25.21751               23.9
## 39 28.55841               50.0
## 40 14.38216                7.2
## 41 12.76573                8.5
## 42 11.78237                9.5
## 43 13.29279               13.4
## 44 14.95076               16.4
## 45 15.79182               19.1
## 46 18.26510               19.6
## 47 14.84985               13.3
## 48 16.01508               16.7
## 49 24.09930               25.0
## 50 20.75357               21.8
## 51 19.49487               19.7
\end{verbatim}

Let's now compute the RMSE:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{tibble}\SpecialCharTok{::}\FunctionTok{tibble}\NormalTok{(}\StringTok{"truth"} \OtherTok{=}\NormalTok{ test\_data}\SpecialCharTok{$}\NormalTok{price,}
        \StringTok{"prediction"} \OtherTok{=} \FunctionTok{unlist}\NormalTok{(price\_predict)) }\SpecialCharTok{\%\textgreater{}\%} 
\NormalTok{    yardstick}\SpecialCharTok{::}\FunctionTok{rmse}\NormalTok{(truth, prediction)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 rmse    standard       0.425
\end{verbatim}

As I mentioned above, all the part about looking for hyper-parameters could be changed to something
else. The general approach though remains what I have described, and can be applied for any models
that have hyper-parameters.

\hypertarget{defining-your-own-functions}{%
\section{Defining your own functions}\label{defining-your-own-functions}}

In this section we are going to learn some advanced concepts that are going to make you into a
full-fledged R programmer. Before this chapter you only used whatever R came with, as well as the
functions contained in packages. We did define some functions ourselves in Chapter 6 already, but
without going into many details. In this chapter, we will learn about building functions ourselves,
and do so in greater detail than what we did before.

\hypertarget{control-flow}{%
\subsection{Control flow}\label{control-flow}}

Knowing about control flow is essential to build your own functions. Without control flow statements,
such as if-else statements or loops (or, in the case of pure functional programming languages, recursion),
programming languages would be very limited.

\hypertarget{if-else}{%
\subsubsection{If-else}\label{if-else}}

Imagine you want a variable to be equal to a certain value if a condition is met. This is a typical
problem that requires the \texttt{if\ ...\ else\ ...} construct. For instance:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \DecValTok{4}
\NormalTok{b }\OtherTok{\textless{}{-}} \DecValTok{5}
\end{Highlighting}
\end{Shaded}

Suppose that if \texttt{a\ \textgreater{}\ b} then \texttt{f} should be equal to 20, else \texttt{f} should be equal to 10. Using \texttt{if\ ...\ else\ ...} you can achieve this like so:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{if}\NormalTok{ (a }\SpecialCharTok{\textgreater{}}\NormalTok{ b) \{}
\NormalTok{  f }\OtherTok{\textless{}{-}} \DecValTok{20}
\NormalTok{    \} }\ControlFlowTok{else}\NormalTok{ \{}
\NormalTok{  f }\OtherTok{\textless{}{-}} \DecValTok{10}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Obviously, here \texttt{f\ =\ 10}. Another way to achieve this is by using the \texttt{ifelse()} function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{f }\OtherTok{\textless{}{-}} \FunctionTok{ifelse}\NormalTok{(a }\SpecialCharTok{\textgreater{}}\NormalTok{ b, }\DecValTok{20}\NormalTok{, }\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\texttt{if...else...} and \texttt{ifelse()} might seem interchangeable, but they're not. \texttt{ifelse()} is vectorized, while
\texttt{if...else..} is not. Let's try the following:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ifelse}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{4}\NormalTok{) }\SpecialCharTok{\textgreater{}} \FunctionTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{, }\DecValTok{1}\NormalTok{, }\DecValTok{0}\NormalTok{), }\StringTok{"yes"}\NormalTok{, }\StringTok{"no"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "no"  "yes" "yes"
\end{verbatim}

The result is a vector. Now, let's see what happens if we use \texttt{if...else...} instead of \texttt{ifelse()}:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{if}\NormalTok{ (}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{4}\NormalTok{) }\SpecialCharTok{\textgreater{}} \FunctionTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{, }\DecValTok{1}\NormalTok{, }\DecValTok{0}\NormalTok{)) }\FunctionTok{print}\NormalTok{(}\StringTok{"yes"}\NormalTok{) }\ControlFlowTok{else} \FunctionTok{print}\NormalTok{(}\StringTok{"no"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\SpecialCharTok{\textgreater{}}\NormalTok{ Error }\ControlFlowTok{in} \ControlFlowTok{if}\NormalTok{ (}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{4}\NormalTok{) }\SpecialCharTok{\textgreater{}} \FunctionTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{, }\DecValTok{1}\NormalTok{, }\DecValTok{0}\NormalTok{)) }\FunctionTok{print}\NormalTok{(}\StringTok{"yes"}\NormalTok{) }\ControlFlowTok{else} \FunctionTok{print}\NormalTok{(}\StringTok{"no"}\NormalTok{) }\SpecialCharTok{:} 
\NormalTok{  the condition has length }\SpecialCharTok{\textgreater{}} \DecValTok{1}
\end{Highlighting}
\end{Shaded}

This results in an error (in previous R version, only the first element of the vector would get used).
We have already discussed this in Chapter 2, remember? If you want to make sure that such an expression
evaluates to \texttt{TRUE}, then you need to use \texttt{all()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ifelse}\NormalTok{(}\FunctionTok{all}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{4}\NormalTok{) }\SpecialCharTok{\textgreater{}} \FunctionTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{, }\DecValTok{1}\NormalTok{, }\DecValTok{0}\NormalTok{)), }\StringTok{"all elements are greater"}\NormalTok{, }\StringTok{"not all elements are greater"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "not all elements are greater"
\end{verbatim}

You may also remember the \texttt{any()} function:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{ifelse}\NormalTok{(}\FunctionTok{any}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{4}\NormalTok{) }\SpecialCharTok{\textgreater{}} \FunctionTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{, }\DecValTok{1}\NormalTok{, }\DecValTok{0}\NormalTok{)), }\StringTok{"at least one element is greater"}\NormalTok{, }\StringTok{"no element greater"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "at least one element is greater"
\end{verbatim}

These are the basics. But sometimes, you might need to test for more complex conditions, which can
lead to using nested \texttt{if...else...} constructs. These, however, can get messy:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{if}\NormalTok{ (}\DecValTok{10} \SpecialCharTok{\%\%} \DecValTok{3} \SpecialCharTok{==} \DecValTok{0}\NormalTok{) \{}
  \FunctionTok{print}\NormalTok{(}\StringTok{"10 is divisible by 3"}\NormalTok{)}
\NormalTok{  \} }\ControlFlowTok{else} \ControlFlowTok{if}\NormalTok{ (}\DecValTok{10} \SpecialCharTok{\%\%} \DecValTok{2} \SpecialCharTok{==} \DecValTok{0}\NormalTok{) \{}
    \FunctionTok{print}\NormalTok{(}\StringTok{"10 is divisible by 2"}\NormalTok{)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "10 is divisible by 2"
\end{verbatim}

10 being obviously divisible by 2 and not 3, it is the second sentence that will be printed. The
\texttt{\%\%} operator is the modulus operator, which gives the rest of the division of 10 by 2. In such
cases, it is easier to use \texttt{dplyr::case\_when()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{case\_when}\NormalTok{(}\DecValTok{10} \SpecialCharTok{\%\%} \DecValTok{3} \SpecialCharTok{==} \DecValTok{0} \SpecialCharTok{\textasciitilde{}} \StringTok{"10 is divisible by 3"}\NormalTok{,}
          \DecValTok{10} \SpecialCharTok{\%\%} \DecValTok{2} \SpecialCharTok{==} \DecValTok{0} \SpecialCharTok{\textasciitilde{}} \StringTok{"10 is divisible by 2"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "10 is divisible by 2"
\end{verbatim}

We have already encountered this function in Chapter 4, inside a \texttt{dplyr::mutate()} call to create a new column.

Let's now discuss loops.

\hypertarget{for-loops}{%
\subsubsection{For loops}\label{for-loops}}

For loops make it possible to repeat a set of instructions \texttt{i} times. For example, try the following:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{for}\NormalTok{ (i }\ControlFlowTok{in} \DecValTok{1}\SpecialCharTok{:}\DecValTok{10}\NormalTok{)\{}
  \FunctionTok{print}\NormalTok{(}\StringTok{"hello"}\NormalTok{)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
\end{verbatim}

It is also possible to do computations using for loops. Let's compute the sum of the first
100 integers:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{result }\OtherTok{\textless{}{-}} \DecValTok{0}
\ControlFlowTok{for}\NormalTok{ (i }\ControlFlowTok{in} \DecValTok{1}\SpecialCharTok{:}\DecValTok{100}\NormalTok{)\{}
\NormalTok{  result }\OtherTok{\textless{}{-}}\NormalTok{ result }\SpecialCharTok{+}\NormalTok{ i}
\NormalTok{\}}

\FunctionTok{print}\NormalTok{(result)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 5050
\end{verbatim}

\texttt{result} is equal to 5050, the expected result. What happened in that loop? First, we defined a
variable called \texttt{result} and set it to 0. Then, when the loops starts, \texttt{i} equals 1, so we add
\texttt{result} to \texttt{1}, which is 1. Then, \texttt{i} equals 2, and again, we add \texttt{result} to \texttt{i}. But this time,
\texttt{result} equals 1 and \texttt{i} equals 2, so now \texttt{result} equals 3, and we repeat this until \texttt{i}
equals 100. If you know a programming language like C, this probably looks familiar. However, R is
not C, and you should, if possible, avoid writing code that looks like this. You should always
ask yourself the following questions:

\begin{itemize}
\tightlist
\item
  Is there an inbuilt function to achieve what I need? In this case we have \texttt{sum()}, so we could use \texttt{sum(seq(1,\ 100))}.
\item
  Is there a way to use matrix algebra? This can sometimes make things easier, but it depends how comfortable
  you are with matrix algebra. This would be the solution with matrix algebra: \texttt{rep(1,\ 100)\ \%*\%\ seq(1,\ 100)}.
\item
  Is there a way to use building blocks that are already available? For instance, suppose that \texttt{sum()}
  would not be a function available in R. Another way to solve this issue would be to use the following
  building blocks: \texttt{+}, which computes the sum of two numbers and \texttt{Reduce()}, which \emph{reduces} a list
  of elements using an operator. Sounds complicated? Let's see how \texttt{Reduce()} works. First, let me show you how
  I combine these two functions to achieve the same result as when using \texttt{sum()}:
\end{itemize}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{Reduce}\NormalTok{(}\StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{, }\FunctionTok{seq}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{100}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 5050
\end{verbatim}

We will see how \texttt{Reduce()} works in greater detail in the next chapter, but what happened was something like this:

\begin{verbatim}
Reduce(`+`, seq(1, 100)) = 
 1 + Reduce(`+`, seq(2, 100)) = 
 1 + 2 + Reduce(`+`, seq(3, 100)) = 
 1 + 2 + 3 + Reduce(`+`, seq(4, 100)) = 
 ....
\end{verbatim}

If you ask yourself these questions, it turns out that you only rarely actually need to write loops, but loops are
still important, because sometimes there simply isn't an alternative. Also, there are other situations where loops
are also important, so I refer you to the following \href{http://adv-r.had.co.nz/Functionals.html\#functionals-not}{section}
of Hadley Wickham's \emph{Advanced R} for an in-depth discussion on situations where loops make more
sense than using functions such as \texttt{Reduce()}.

\hypertarget{while-loops}{%
\subsubsection{While loops}\label{while-loops}}

While loops are very similar to for loops. The instructions inside a while loop are repeated while a
certain condition holds true. Let's consider the sum of the first 100 integers again:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{result }\OtherTok{\textless{}{-}} \DecValTok{0}
\NormalTok{i }\OtherTok{\textless{}{-}} \DecValTok{1}
\ControlFlowTok{while}\NormalTok{ (i}\SpecialCharTok{\textless{}=}\DecValTok{100}\NormalTok{)\{}
\NormalTok{  result }\OtherTok{=}\NormalTok{ result }\SpecialCharTok{+}\NormalTok{ i}
\NormalTok{  i }\OtherTok{=}\NormalTok{ i }\SpecialCharTok{+} \DecValTok{1}
\NormalTok{\}}

\FunctionTok{print}\NormalTok{(result)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 5050
\end{verbatim}

Here, we first set \texttt{result} and \texttt{i} to 0. Then, while \texttt{i} is less than, or equal to 100, we add \texttt{i}
to \texttt{result}. Notice that there is one more line than in the for loop version of this code: we need
to increment the value of \texttt{i} at each iteration, if not, \texttt{i} would stay equal to 1, and the
condition would always be fulfilled, and the loop would run forever (not really, only until your
computer runs out of memory, or until the heat death of the universe, whichever comes first).

Now that we know how to write loops, and know about \texttt{if...else...} constructs, we have (almost) all
the ingredients to write our own functions.

\hypertarget{writing-your-own-functions}{%
\subsection{Writing your own functions}\label{writing-your-own-functions}}

As you have seen by now, R includes a very large amount of in-built functions, but also many
more functions are available in packages. However, there will be a lot of situations where you will
need to write your own. In this section we are going to learn how to write our own functions.

\hypertarget{declaring-functions-in-r}{%
\subsubsection{Declaring functions in R}\label{declaring-functions-in-r}}

Suppose you want to create the following function: \(f(x) = \dfrac{1}{\sqrt{x}}\).
Writing this in R is quite simple:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x)\{}
  \DecValTok{1}\SpecialCharTok{/}\FunctionTok{sqrt}\NormalTok{(x)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

The argument of the function, \texttt{x}, gets passed to the \texttt{function()} function and the \emph{body} of
the function (more on that in the next Chapter) contains the function definition. Of course,
you could define functions that use more than one input:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x, y)\{}
  \DecValTok{1}\SpecialCharTok{/}\FunctionTok{sqrt}\NormalTok{(x }\SpecialCharTok{+}\NormalTok{ y)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

or inputs with names longer than one character:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(argument1, argument2)\{}
  \DecValTok{1}\SpecialCharTok{/}\FunctionTok{sqrt}\NormalTok{(argument1 }\SpecialCharTok{+}\NormalTok{ argument2)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Functions written by the user get called just the same way as functions included in R:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_function}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.3015113
\end{verbatim}

It is also possible to provide default values to the function's arguments, which are values that are used
if the user omits them:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(argument1, }\AttributeTok{argument2 =} \DecValTok{10}\NormalTok{)\{}
\DecValTok{1}\SpecialCharTok{/}\FunctionTok{sqrt}\NormalTok{(argument1 }\SpecialCharTok{+}\NormalTok{ argument2)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_function}\NormalTok{(}\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.3015113
\end{verbatim}

This is especially useful for functions with many arguments. Consider also the following example,
where the function has a default method:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(argument1, argument2, }\AttributeTok{method =} \StringTok{"foo"}\NormalTok{)\{}
  
\NormalTok{  x }\OtherTok{\textless{}{-}}\NormalTok{ argument1 }\SpecialCharTok{+}\NormalTok{ argument2}
  
  \ControlFlowTok{if}\NormalTok{(method }\SpecialCharTok{==} \StringTok{"foo"}\NormalTok{)\{}
    \DecValTok{1}\SpecialCharTok{/}\FunctionTok{sqrt}\NormalTok{(x)}
\NormalTok{  \} }\ControlFlowTok{else} \ControlFlowTok{if}\NormalTok{ (method }\SpecialCharTok{==} \StringTok{"bar"}\NormalTok{)\{}
    \StringTok{"this is a string"}
\NormalTok{    \}}
\NormalTok{\}}

\FunctionTok{my\_function}\NormalTok{(}\DecValTok{10}\NormalTok{, }\DecValTok{11}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.2182179
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_function}\NormalTok{(}\DecValTok{10}\NormalTok{, }\DecValTok{11}\NormalTok{, }\StringTok{"bar"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "this is a string"
\end{verbatim}

As you see, depending on the ``method'' chosen, the returned result is either a numeric, or a string.
What happens if the user provides a ``method'' that is neither ``foo'' nor ``bar''?

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_function}\NormalTok{(}\DecValTok{10}\NormalTok{, }\DecValTok{11}\NormalTok{, }\StringTok{"spam"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

As you can see nothing happens. It is possible to add safeguards to your function to avoid such
situations:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(argument1, argument2, }\AttributeTok{method =} \StringTok{"foo"}\NormalTok{)\{}
  
  \ControlFlowTok{if}\NormalTok{(}\SpecialCharTok{!}\NormalTok{(method }\SpecialCharTok{\%in\%} \FunctionTok{c}\NormalTok{(}\StringTok{"foo"}\NormalTok{, }\StringTok{"bar"}\NormalTok{)))\{}
    \FunctionTok{return}\NormalTok{(}\StringTok{"Method must be either \textquotesingle{}foo\textquotesingle{} or \textquotesingle{}bar\textquotesingle{}"}\NormalTok{)}
\NormalTok{  \}}
  
\NormalTok{  x }\OtherTok{\textless{}{-}}\NormalTok{ argument1 }\SpecialCharTok{+}\NormalTok{ argument2}
  
  \ControlFlowTok{if}\NormalTok{(method }\SpecialCharTok{==} \StringTok{"foo"}\NormalTok{)\{}
    \DecValTok{1}\SpecialCharTok{/}\FunctionTok{sqrt}\NormalTok{(x)}
\NormalTok{  \} }\ControlFlowTok{else} \ControlFlowTok{if}\NormalTok{ (method }\SpecialCharTok{==} \StringTok{"bar"}\NormalTok{)\{}
    \StringTok{"this is a string"}
\NormalTok{    \}}
\NormalTok{\}}

\FunctionTok{my\_function}\NormalTok{(}\DecValTok{10}\NormalTok{, }\DecValTok{11}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 0.2182179
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_function}\NormalTok{(}\DecValTok{10}\NormalTok{, }\DecValTok{11}\NormalTok{, }\StringTok{"bar"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "this is a string"
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_function}\NormalTok{(}\DecValTok{10}\NormalTok{, }\DecValTok{11}\NormalTok{, }\StringTok{"foobar"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "Method must be either 'foo' or 'bar'"
\end{verbatim}

Notice that I have used \texttt{return()} inside my first \texttt{if} statement. This is to immediately stop
evaluation of the function and return a value. If I had omitted it, evaluation would have
continued, as it is always the last expression that gets evaluated. Remove \texttt{return()} and run the
function again, and see what happens. Later, we are going to learn how to add better safeguards to
your functions and to avoid runtime errors.

While in general, it is a good idea to add comments to your functions to explain what they do, I
would avoid adding comments to functions that do things that are very obvious, such as with this
one. Function names should be of the form: \texttt{function\_name()}. Always give your function very
explicit names! In mathematics it is standard to give functions just one letter as a name, but I
would advise against doing that in your code. Functions that you write are not special in any way;
this means that R will treat them the same way, and they will work in conjunction with any other
function just as if it was built-in into R.

They have one limitation though (which is shared with R's native function): just like in math,
they can only return one value. However, sometimes, you may need to return more than one value.
To be able to do this, you must put your values in a list, and return the list of values. For example:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{average\_and\_sd }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x)\{}
\FunctionTok{c}\NormalTok{(}\FunctionTok{mean}\NormalTok{(x), }\FunctionTok{sd}\NormalTok{(x))}
\NormalTok{\}}

\FunctionTok{average\_and\_sd}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{9}\NormalTok{, }\DecValTok{10}\NormalTok{, }\DecValTok{12}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 7.166667 4.262237
\end{verbatim}

You're still returning a single object, but it's a vector. You can also return a named list:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{average\_and\_sd }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x)\{}
\FunctionTok{list}\NormalTok{(}\StringTok{"mean\_x"} \OtherTok{=}  \FunctionTok{mean}\NormalTok{(x), }\StringTok{"sd\_x"} \OtherTok{=} \FunctionTok{sd}\NormalTok{(x))}
\NormalTok{\}}

\FunctionTok{average\_and\_sd}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{9}\NormalTok{, }\DecValTok{10}\NormalTok{, }\DecValTok{12}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## $mean_x
## [1] 7.166667
## 
## $sd_x
## [1] 4.262237
\end{verbatim}

As described before, you can use \texttt{return()} at the end of your functions:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{average\_and\_sd }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x)\{}
\NormalTok{  result }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\FunctionTok{mean}\NormalTok{(x), }\FunctionTok{sd}\NormalTok{(x))}
\FunctionTok{return}\NormalTok{(result)}
\NormalTok{\}}

\FunctionTok{average\_and\_sd}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{9}\NormalTok{, }\DecValTok{10}\NormalTok{, }\DecValTok{12}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 7.166667 4.262237
\end{verbatim}

But this is only needed if you need to return a value early:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{average\_and\_sd }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x)\{}
\ControlFlowTok{if}\NormalTok{(}\FunctionTok{any}\NormalTok{(}\FunctionTok{is.na}\NormalTok{(x)))\{}
    \FunctionTok{return}\NormalTok{(}\ConstantTok{NA}\NormalTok{)}
\NormalTok{  \} }\ControlFlowTok{else}\NormalTok{ \{}
    \FunctionTok{c}\NormalTok{(}\FunctionTok{mean}\NormalTok{(x), }\FunctionTok{sd}\NormalTok{(x))}
\NormalTok{    \}}
\NormalTok{\}}

\FunctionTok{average\_and\_sd}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{9}\NormalTok{, }\DecValTok{10}\NormalTok{, }\DecValTok{12}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 7.166667 4.262237
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{average\_and\_sd}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{3}\NormalTok{, }\ConstantTok{NA}\NormalTok{, }\DecValTok{9}\NormalTok{, }\DecValTok{10}\NormalTok{, }\DecValTok{12}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] NA
\end{verbatim}

If you need to use a function from a package inside your function use \texttt{::}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_sum }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(a\_vector)\{}
\NormalTok{  purrr}\SpecialCharTok{::}\FunctionTok{reduce}\NormalTok{(a\_vector, }\StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

However, if you need to use more than one function, this can become tedious. A quick and dirty
way of doing that, is to use \texttt{library(package\_name)}, inside the function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_sum }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(a\_vector)\{}
  \FunctionTok{library}\NormalTok{(purrr)}
  \FunctionTok{reduce}\NormalTok{(a\_vector, }\StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Loading the library inside the function has the advantage that you will be sure that the package
upon which your function depends will be loaded. If the package is already loaded, it will not be
loaded again, thus not impact performance, but if you forgot to load it at the beginning of your
script, then, no worries, your function will load it the first time you use it! However, the very
best way would be to write your own package and declare the packages upon which your functions
depend as dependencies. This is something we are going to explore in Chapter 11.

You can put a lot of instructions inside a function, such as loops. Let's create the function that
returns Fionacci numbers.

\hypertarget{fibonacci-numbers}{%
\subsubsection{Fibonacci numbers}\label{fibonacci-numbers}}

The Fibonacci sequence is the following:

\[1, 1, 2, 3, 5, 8, 13, 21, 34, 55, ...\]

Each subsequent number is composed of the sum of the two preceding ones. In R, it is possible to define a function that returns the \(n^{th}\) fibonacci number:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_fibo }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(n)\{}
\NormalTok{ a }\OtherTok{\textless{}{-}} \DecValTok{0}
\NormalTok{ b }\OtherTok{\textless{}{-}} \DecValTok{1}
 \ControlFlowTok{for}\NormalTok{ (i }\ControlFlowTok{in} \DecValTok{1}\SpecialCharTok{:}\NormalTok{n)\{}
\NormalTok{  temp }\OtherTok{\textless{}{-}}\NormalTok{ b}
\NormalTok{  b }\OtherTok{\textless{}{-}}\NormalTok{ a}
\NormalTok{  a }\OtherTok{\textless{}{-}}\NormalTok{ a }\SpecialCharTok{+}\NormalTok{ temp}
\NormalTok{ \}}
\NormalTok{ a}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Inside the loop, we defined a variable called \texttt{temp}. Defining temporary variables is usually very
useful. Let's try to understand what happens inside this loop:

\begin{itemize}
\tightlist
\item
  First, we assign the value 0 to variable \texttt{a} and value 1 to variable \texttt{b}.
\item
  We start a loop, that goes from 1 to \texttt{n}.
\item
  We assign the value inside of \texttt{b} to a temporary variable, called \texttt{temp}.
\item
  \texttt{b} becomes \texttt{a}.
\item
  We assign the sum of \texttt{a} and \texttt{temp} to \texttt{a}.
\item
  When the loop is finished, we return \texttt{a}.
\end{itemize}

What happens if we want the 3rd fibonacci number? At \texttt{n\ =\ 1} we have first \texttt{a\ =\ 0} and \texttt{b\ =\ 1},
then \texttt{temp\ =\ 1}, \texttt{b\ =\ 0} and \texttt{a\ =\ 0\ +\ 1}. Then \texttt{n\ =\ 2}. Now \texttt{b\ =\ 0} and \texttt{temp\ =\ 0}. The previous
result, \texttt{a\ =\ 0\ +\ 1} is now assigned to \texttt{b}, so \texttt{b\ =\ 1}. Then, \texttt{a\ =\ 1\ +\ 0}. Finally, \texttt{n\ =\ 3}. \texttt{temp\ =\ 1} (because \texttt{b\ =\ 1}), the previous result \texttt{a\ =\ 1} is assigned to \texttt{b} and finally, \texttt{a\ =\ 1\ +\ 1}. So
the third fibonacci number equals 2. Reading this might be a bit confusing; I strongly advise you
to run the algorithm on a sheet of paper, step by step.

The above algorithm is called an iterative algorithm, because it uses a loop to compute the result.
Let's look at another way to think about the problem, with a so-called recursive function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fibo\_recur }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(n)\{}
 \ControlFlowTok{if}\NormalTok{ (n }\SpecialCharTok{==} \DecValTok{0} \SpecialCharTok{||}\NormalTok{ n }\SpecialCharTok{==} \DecValTok{1}\NormalTok{)\{}
   \FunctionTok{return}\NormalTok{(n)}
\NormalTok{   \} }\ControlFlowTok{else}\NormalTok{ \{}
   \FunctionTok{fibo\_recur}\NormalTok{(n}\DecValTok{{-}1}\NormalTok{) }\SpecialCharTok{+} \FunctionTok{fibo\_recur}\NormalTok{(n}\DecValTok{{-}2}\NormalTok{)}
\NormalTok{   \}}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

This algorithm should be easier to understand: if \texttt{n\ =\ 0} or \texttt{n\ =\ 1} the function should return \texttt{n}
(0 or 1). If \texttt{n} is strictly bigger than \texttt{1}, \texttt{fibo\_recur()} should return the sum of
\texttt{fibo\_recur(n-1)} and \texttt{fibo\_recur(n-2)}. This version of the function is very much the same as the
mathematical definition of the fibonacci sequence. So why not use only recursive algorithms
then? Try to run the following:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{system.time}\NormalTok{(}\FunctionTok{my\_fibo}\NormalTok{(}\DecValTok{30}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##    user  system elapsed 
##   0.007   0.000   0.007
\end{verbatim}

The result should be printed very fast (the \texttt{system.time()} function returns the time that it took
to execute \texttt{my\_fibo(30)}). Let's try with the recursive version:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{system.time}\NormalTok{(}\FunctionTok{fibo\_recur}\NormalTok{(}\DecValTok{30}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##    user  system elapsed 
##   1.463   0.072   1.544
\end{verbatim}

It takes much longer to execute! Recursive algorithms are very CPU demanding, so if speed is
critical, it's best to avoid recursive algorithms. Also, in \texttt{fibo\_recur()} try to remove this line:
\texttt{if\ (n\ ==\ 0\ \textbar{}\textbar{}\ n\ ==\ 1)} and try to run \texttt{fibo\_recur(5)} and see what happens. You should
get an error: this is because for recursive algorithms you need a stopping condition, or else,
it would run forever. This is not the case for iterative algorithms, because the stopping
condition is the last step of the loop.

So as you can see, for recursive relationships, for or while loops are the way to go in R, whether
you're writing these loops inside functions or not.

\hypertarget{exercises-4}{%
\subsection{Exercises}\label{exercises-4}}

\hypertarget{exercise-1-4}{%
\subsubsection*{Exercise 1}\label{exercise-1-4}}
\addcontentsline{toc}{subsubsection}{Exercise 1}

In this exercise, you will write a function to compute the sum of the n first integers. Combine the
algorithm we saw in section about while loops and what you learned about functions
in this section.

\hypertarget{exercise-2-2}{%
\subsubsection*{Exercise 2}\label{exercise-2-2}}
\addcontentsline{toc}{subsubsection}{Exercise 2}

Write a function called \texttt{my\_fact()} that computes the factorial of a number \texttt{n}. Do it using a
loop, using a recursive function, and using a functional:

\hypertarget{exercise-3-2}{%
\subsubsection*{Exercise 3}\label{exercise-3-2}}
\addcontentsline{toc}{subsubsection}{Exercise 3}

Write a function to find the roots of quadratic functions. Your function should take 3 arguments,
\texttt{a}, \texttt{b} and \texttt{c} and return the two roots. Only consider the case where there are two real roots
(delta \textgreater{} 0).

\hypertarget{functions-that-take-functions-as-arguments-writing-your-own-higher-order-functions}{%
\subsection{Functions that take functions as arguments: writing your own higher-order functions}\label{functions-that-take-functions-as-arguments-writing-your-own-higher-order-functions}}

Functions that take functions as arguments are very powerful and useful tools. You already know a
couple, \texttt{purrr::map()} and \texttt{purrr::reduce()}, discussed briefly in Chapter 4.
But you can also write your own! A very simple example would be the following:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_func }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x, func)\{}
  \FunctionTok{func}\NormalTok{(x)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\texttt{my\_func()} is a very simple function that takes \texttt{x} and \texttt{func()} as arguments and that simply
executes \texttt{func(x)}. This might not seem very useful (after all, you could simply use \texttt{func(x)!}) but
this is just for illustration purposes, in practice, your functions would be more useful than that!
Let's try to use \texttt{my\_func()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_func}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{1}\NormalTok{, }\DecValTok{0}\NormalTok{, }\DecValTok{8}\NormalTok{), mean)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 3.6
\end{verbatim}

As expected, this returns the mean of the given vector. But now suppose the following:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_func}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{1}\NormalTok{, }\ConstantTok{NA}\NormalTok{, }\DecValTok{8}\NormalTok{), mean)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] NA
\end{verbatim}

Because one element of the list is \texttt{NA}, the whole mean is \texttt{NA}. \texttt{mean()} has a \texttt{na.rm} argument
that you can set to \texttt{TRUE} to ignore the \texttt{NA}s in the vector. However, here, there is no way to
provide this argument to the function \texttt{mean()}! Let's see what happens when we try to:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_func}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{1}\NormalTok{, }\ConstantTok{NA}\NormalTok{, }\DecValTok{8}\NormalTok{), mean, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
Error in my_func(c(1, 8, 1, NA, 8), mean, na.rm = TRUE) :
  unused argument (na.rm = TRUE)
\end{verbatim}

So what you could do is pass the value \texttt{TRUE} to the \texttt{na.rm} argument of \texttt{mean()} from your own
function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_func }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x, func, remove\_na)\{}
  \FunctionTok{func}\NormalTok{(x, }\AttributeTok{na.rm =}\NormalTok{ remove\_na)}
\NormalTok{\}}

\FunctionTok{my\_func}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{1}\NormalTok{, }\ConstantTok{NA}\NormalTok{, }\DecValTok{8}\NormalTok{), mean, }\AttributeTok{remove\_na =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 4.5
\end{verbatim}

This is one solution, but \texttt{mean()} also has another argument called \texttt{trim}. What if some other
user needs this argument? Should you also add it to your function? Surely there's a way to avoid
this problem? Yes, there is, and it by using the \emph{dots}. The \texttt{...} simply mean ``any other
argument as needed'', and it's very easy to use:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_func }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x, func, ...)\{}
  \FunctionTok{func}\NormalTok{(x, ...)}
\NormalTok{\}}

\FunctionTok{my\_func}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{1}\NormalTok{, }\ConstantTok{NA}\NormalTok{, }\DecValTok{8}\NormalTok{), mean, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 4.5
\end{verbatim}

or, now, if you need the \texttt{trim} argument:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_func}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{1}\NormalTok{, }\ConstantTok{NA}\NormalTok{, }\DecValTok{8}\NormalTok{), mean, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{, }\AttributeTok{trim =} \FloatTok{0.1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 4.5
\end{verbatim}

The \texttt{...} are very useful when writing higher-order functions such as \texttt{my\_func()}, because it allows
you to pass arguments \emph{down} to the underlying functions.

\hypertarget{functions-that-return-functions}{%
\subsection{Functions that return functions}\label{functions-that-return-functions}}

The example from before, \texttt{my\_func()} took three arguments, some \texttt{x}, a function \texttt{func}, and \texttt{...} (dots). \texttt{my\_func()}
was a kind of wrapper that evaluated \texttt{func} on its arguments \texttt{x} and \texttt{...}. But sometimes this is not quite what you
need or want. It is sometimes useful to write a function that returns a modified function. This type of function
is called a function factory, as it \emph{builds} functions. For instance, suppose that we want to time how long functions
take to run. An idea would be to proceed like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{tic }\OtherTok{\textless{}{-}} \FunctionTok{Sys.time}\NormalTok{()}
\FunctionTok{very\_slow\_function}\NormalTok{(x)}
\NormalTok{toc }\OtherTok{\textless{}{-}} \FunctionTok{Sys.time}\NormalTok{()}

\NormalTok{running\_time }\OtherTok{\textless{}{-}}\NormalTok{ toc }\SpecialCharTok{{-}}\NormalTok{ tic}
\end{Highlighting}
\end{Shaded}

but if you want to time several functions, this gets very tedious. It would be much easier if functions would
time \emph{themselves}. We could achieve this by writing a wrapper, like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{timed\_very\_slow\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(...)\{}

\NormalTok{  tic }\OtherTok{\textless{}{-}} \FunctionTok{Sys.time}\NormalTok{()}
\NormalTok{  result }\OtherTok{\textless{}{-}} \FunctionTok{very\_slow\_function}\NormalTok{(x)}
\NormalTok{  toc }\OtherTok{\textless{}{-}} \FunctionTok{Sys.time}\NormalTok{()}

\NormalTok{  running\_time }\OtherTok{\textless{}{-}}\NormalTok{ toc }\SpecialCharTok{{-}}\NormalTok{ tic}

  \FunctionTok{list}\NormalTok{(}\StringTok{"result"} \OtherTok{=}\NormalTok{ result,}
       \StringTok{"running\_time"} \OtherTok{=}\NormalTok{ running\_time)}

\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

The problem here is that we have to change each function we need to time. But thanks to the concept of function
factories, we can write a function that does this for us:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{time\_f }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(.f, ...)\{}

  \ControlFlowTok{function}\NormalTok{(...)\{}

\NormalTok{    tic }\OtherTok{\textless{}{-}} \FunctionTok{Sys.time}\NormalTok{()}
\NormalTok{    result }\OtherTok{\textless{}{-}} \FunctionTok{.f}\NormalTok{(...)}
\NormalTok{    toc }\OtherTok{\textless{}{-}} \FunctionTok{Sys.time}\NormalTok{()}

\NormalTok{    running\_time }\OtherTok{\textless{}{-}}\NormalTok{ toc }\SpecialCharTok{{-}}\NormalTok{ tic}

    \FunctionTok{list}\NormalTok{(}\StringTok{"result"} \OtherTok{=}\NormalTok{ result,}
         \StringTok{"running\_time"} \OtherTok{=}\NormalTok{ running\_time)}

\NormalTok{  \}}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\texttt{time\_f()} is a function that returns a function, a function factory. Calling it on a function returns, as expected,
a function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{t\_mean }\OtherTok{\textless{}{-}} \FunctionTok{time\_f}\NormalTok{(mean)}

\NormalTok{t\_mean}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## function(...){
## 
##     tic <- Sys.time()
##     result <- .f(...)
##     toc <- Sys.time()
## 
##     running_time <- toc - tic
## 
##     list("result" = result,
##          "running_time" = running_time)
## 
##   }
## <environment: 0x55a89a9f6158>
\end{verbatim}

This function can now be used like any other function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{output }\OtherTok{\textless{}{-}} \FunctionTok{t\_mean}\NormalTok{(}\FunctionTok{seq}\NormalTok{(}\SpecialCharTok{{-}}\DecValTok{500000}\NormalTok{, }\DecValTok{500000}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\texttt{output} is a list of two elements, the first being simply the result of \texttt{mean(seq(-500000,\ 500000))}, and the other
being the running time.

This approach is super flexible. For instance, imagine that there is an \texttt{NA} in the vector. This would result in
the mean of this vector being \texttt{NA}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{t\_mean}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\ConstantTok{NA}\NormalTok{, }\FunctionTok{seq}\NormalTok{(}\SpecialCharTok{{-}}\DecValTok{500000}\NormalTok{, }\DecValTok{500000}\NormalTok{)))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## $result
## [1] NA
## 
## $running_time
## Time difference of 0.006770611 secs
\end{verbatim}

But because we use the \texttt{...} in the definition of \texttt{time\_f()}, we can now simply pass \texttt{mean()}'s option down to it:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{t\_mean}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\ConstantTok{NA}\NormalTok{, }\FunctionTok{seq}\NormalTok{(}\SpecialCharTok{{-}}\DecValTok{500000}\NormalTok{, }\DecValTok{500000}\NormalTok{)), }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## $result
## [1] 0
## 
## $running_time
## Time difference of 0.01325297 secs
\end{verbatim}

\hypertarget{functions-that-take-columns-of-data-as-arguments}{%
\subsection{Functions that take columns of data as arguments}\label{functions-that-take-columns-of-data-as-arguments}}

\hypertarget{the-enquo---approach}{%
\subsubsection{\texorpdfstring{The \texttt{enquo()\ -\ !!()} approach}{The enquo() - !!() approach}}\label{the-enquo---approach}}

In many situations, you will want to write functions that look similar to this:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{my\_function}\NormalTok{(my\_data, one\_column\_inside\_data)}
\end{Highlighting}
\end{Shaded}

Such a function would be useful in situation where you have to apply a certain number of operations
to columns for different data frames. For example if you need to create tables of descriptive
statistics or graphs periodically, it might be very interesting to put these operations inside a
function and then call the function whenever you need it, on the fresh batch of data.

However, if you try to write something like that, something that might seem unexpected, at first,
will happen:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(mtcars)}

\NormalTok{simple\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataset, col\_name)\{}
\NormalTok{  dataset }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(col\_name) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_speed =} \FunctionTok{mean}\NormalTok{(speed))}
\NormalTok{\}}


\FunctionTok{simple\_function}\NormalTok{(cars, }\StringTok{"dist"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
Error: unknown variable to group by : col_name
\end{verbatim}

The variable \texttt{col\_name} is passed to \texttt{simple\_function()} as a string, but \texttt{group\_by()} requires a
variable name. So why not try to convert \texttt{col\_name} to a name?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simple\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataset, col\_name)\{}
\NormalTok{  col\_name }\OtherTok{\textless{}{-}} \FunctionTok{as.name}\NormalTok{(col\_name)}
\NormalTok{  dataset }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(col\_name) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_speed =} \FunctionTok{mean}\NormalTok{(speed))}
\NormalTok{\}}


\FunctionTok{simple\_function}\NormalTok{(cars, }\StringTok{"dist"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
Error: unknown variable to group by : col_name
\end{verbatim}

This is because R is literally looking for the variable \texttt{"dist"} somewhere in the global
environment, and not as a column of the data. R does not understand that you are refering to the
column \texttt{"dist"} that is inside the dataset. So how can we make R understands what you mean?

To be able to do that, we need to use a framework that was introduced in the \texttt{\{tidyverse\}},
called \emph{tidy evaluation}. This framework can be used by installing the \texttt{\{rlang\}} package.
\texttt{\{rlang\}} is quite a technical package, so I will spare you the details. But you should at
the very least take a look at the following documents
\href{http://dplyr.tidyverse.org/articles/programming.html}{here} and
\href{https://rlang.r-lib.org/reference/topic-data-mask.html}{here}. The
discussion can get complicated, but you don't need to know everything about \texttt{\{rlang\}}.
As you will see, knowing some of the capabilities \texttt{\{rlang\}} provides can be incredibly useful.
Take a look at the code below:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simple\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataset, col\_name)\{}
\NormalTok{  col\_name }\OtherTok{\textless{}{-}} \FunctionTok{enquo}\NormalTok{(col\_name)}
\NormalTok{  dataset }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(}\SpecialCharTok{!!}\NormalTok{col\_name) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_mpg =} \FunctionTok{mean}\NormalTok{(mpg))}
\NormalTok{\}}


\FunctionTok{simple\_function}\NormalTok{(mtcars, cyl)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 3 x 2
##     cyl mean_mpg
##   <dbl>    <dbl>
## 1     4     26.7
## 2     6     19.7
## 3     8     15.1
\end{verbatim}

As you can see, the previous idea we had, which was using \texttt{as.name()} was not very far away from
the solution. The solution, with \texttt{\{rlang\}}, consists in using \texttt{enquo()}, which (for our purposes),
does something similar to \texttt{as.name()}. Now that \texttt{col\_name} is (R programmers call it) quoted, or
\emph{defused}, we need to tell \texttt{group\_by()} to evaluate the input as is. This is done with \texttt{!!()},
called the \href{https://rlang.r-lib.org/reference/injection-operator.html}{injection operator}, which
is another \texttt{\{rlang\}} function. I say it again; don't worry if you don't understand everything. Just
remember to use \texttt{enquo()} on your column names and then \texttt{!!()} inside the \texttt{\{dplyr\}} function you
want to use.

Let's see some other examples:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simple\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataset, col\_name, value)\{}
\NormalTok{  col\_name }\OtherTok{\textless{}{-}} \FunctionTok{enquo}\NormalTok{(col\_name)}
\NormalTok{  dataset }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{filter}\NormalTok{((}\SpecialCharTok{!!}\NormalTok{col\_name) }\SpecialCharTok{==}\NormalTok{ value) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean\_cyl =} \FunctionTok{mean}\NormalTok{(cyl))}
\NormalTok{\}}


\FunctionTok{simple\_function}\NormalTok{(mtcars, am, }\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   mean_cyl
## 1 5.076923
\end{verbatim}

Notice that I've written:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{filter}\NormalTok{((}\SpecialCharTok{!!}\NormalTok{col\_name) }\SpecialCharTok{==}\NormalTok{ value)}
\end{Highlighting}
\end{Shaded}

and not:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{filter}\NormalTok{(}\SpecialCharTok{!!}\NormalTok{col\_name }\SpecialCharTok{==}\NormalTok{ value)}
\end{Highlighting}
\end{Shaded}

I have enclosed \texttt{!!col\_name} inside parentheses. This is because operators such as \texttt{==} have
precedence over \texttt{!!}, so you have to be explicit. Also, notice that I didn't have to quote \texttt{1}.
This is because it's \emph{standard} variable, not a column inside the dataset. Let's make this function
a bit more general. I hard-coded the variable cyl inside the body of the function, but maybe you'd
like the mean of another variable?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simple\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataset, filter\_col, mean\_col, value)\{}
\NormalTok{  filter\_col }\OtherTok{\textless{}{-}} \FunctionTok{enquo}\NormalTok{(filter\_col)}
\NormalTok{  mean\_col }\OtherTok{\textless{}{-}} \FunctionTok{enquo}\NormalTok{(mean\_col)}
\NormalTok{  dataset }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{filter}\NormalTok{((}\SpecialCharTok{!!}\NormalTok{filter\_col) }\SpecialCharTok{==}\NormalTok{ value) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\FunctionTok{mean}\NormalTok{((}\SpecialCharTok{!!}\NormalTok{mean\_col)))}
\NormalTok{\}}


\FunctionTok{simple\_function}\NormalTok{(mtcars, am, cyl, }\DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   mean(cyl)
## 1  5.076923
\end{verbatim}

Notice that I had to quote \texttt{mean\_col} too.

Using the \texttt{...} that we discovered in the previous section, we can pass more than one column:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simple\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataset, ...)\{}
\NormalTok{  col\_vars }\OtherTok{\textless{}{-}} \FunctionTok{quos}\NormalTok{(...)}
\NormalTok{  dataset }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise\_at}\NormalTok{(}\FunctionTok{vars}\NormalTok{(}\SpecialCharTok{!!!}\NormalTok{col\_vars), }\FunctionTok{funs}\NormalTok{(mean, sd))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Because these \emph{dots} contain more than one variable, you have to use \texttt{quos()} instead of \texttt{enquo()}.
This will put the arguments provided via the dots in a list. Then, because we have a list of
columns, we have to use \texttt{summarise\_at()}, which you should know if you did the exercices of
Chapter 4. So if you didn't do them, go back to them and finish them first. Doing the exercise will
also teach you what \texttt{vars()} and \texttt{funs()} are. The last thing you have to pay attention to is to
use \texttt{!!!()} if you used \texttt{quos()}. So 3 \texttt{!} instead of only 2. This allows you to then do things
like this:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{simple\_function}\NormalTok{(mtcars, am, cyl, mpg)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   am_mean cyl_mean mpg_mean     am_sd   cyl_sd   mpg_sd
## 1 0.40625   6.1875 20.09062 0.4989909 1.785922 6.026948
\end{verbatim}

Using \texttt{...} with \texttt{!!!()} allows you to write very flexible functions.

If you need to be even more general, you can also provide the summary functions as arguments of
your function, but you have to rewrite your function a little bit:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simple\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataset, cols, funcs)\{}
\NormalTok{  dataset }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise\_at}\NormalTok{(}\FunctionTok{vars}\NormalTok{(}\SpecialCharTok{!!!}\NormalTok{cols), }\FunctionTok{funs}\NormalTok{(}\SpecialCharTok{!!!}\NormalTok{funcs))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

You might be wondering where the \texttt{quos()} went? Well because now we are passing two lists, a list of
columns that we have to quote, and a list of functions, that we also have to quote, we need to use \texttt{quos()}
when calling the function:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{simple\_function}\NormalTok{(mtcars, }\FunctionTok{quos}\NormalTok{(am, cyl, mpg), }\FunctionTok{quos}\NormalTok{(mean, sd, sum))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   am_mean cyl_mean mpg_mean     am_sd   cyl_sd   mpg_sd am_sum cyl_sum mpg_sum
## 1 0.40625   6.1875 20.09062 0.4989909 1.785922 6.026948     13     198   642.9
\end{verbatim}

This works, but I don't think you'll need to have that much flexibility; either the columns
are variables, or the functions, but rarely both at the same time.

To conclude this function, I should also talk about \texttt{as\_label()} which allows you to change the
name of a variable, for instance if you want to call the resulting column \texttt{mean\_mpg} when you
compute the mean of the \texttt{mpg} column:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simple\_function }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataset, filter\_col, mean\_col, value)\{}

\NormalTok{  filter\_col }\OtherTok{\textless{}{-}} \FunctionTok{enquo}\NormalTok{(filter\_col)}
\NormalTok{  mean\_col }\OtherTok{\textless{}{-}} \FunctionTok{enquo}\NormalTok{(mean\_col)}
\NormalTok{  mean\_name }\OtherTok{\textless{}{-}} \FunctionTok{paste0}\NormalTok{(}\StringTok{"mean\_"}\NormalTok{, }\FunctionTok{as\_label}\NormalTok{(mean\_col))}
  
\NormalTok{  dataset }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{filter}\NormalTok{((}\SpecialCharTok{!!}\NormalTok{filter\_col) }\SpecialCharTok{==}\NormalTok{ value) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\SpecialCharTok{!!}\NormalTok{(mean\_name) }\SpecialCharTok{:}\ErrorTok{=} \FunctionTok{mean}\NormalTok{((}\SpecialCharTok{!!}\NormalTok{mean\_col)))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Pay attention to the \texttt{:=} operator in the last line. This is needed when using \texttt{as\_label()}.

\hypertarget{curly-curly-a-simplified-approach-to-enquo-and}{%
\subsubsection{\texorpdfstring{Curly Curly, a simplified approach to \texttt{enquo()} and \texttt{!!()}}{Curly Curly, a simplified approach to enquo() and !!()}}\label{curly-curly-a-simplified-approach-to-enquo-and}}

The previous section might have been a bit difficult to grasp, but there is a simplified way of doing it,
which consists in using \texttt{\{\{\}\}}, introduced in \texttt{\{rlang\}} version 0.4.0.
The suggested pronunciation of \texttt{\{\{\}\}} is \emph{curly-curly}, but there is no
\href{https://twitter.com/JonTheGeek/status/1144815369766547456}{consensus yet}.

Let's suppose that I need to write a function that takes a data frame, as well as a column from
this data frame as arguments, just like before:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{how\_many\_na }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataframe, column\_name)\{}
\NormalTok{  dataframe }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{filter}\NormalTok{(}\FunctionTok{is.na}\NormalTok{(column\_name)) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{count}\NormalTok{()}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Let's try this function out on the \texttt{starwars} data:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(starwars)}

\FunctionTok{head}\NormalTok{(starwars)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 6 x 14
##   name         height  mass hair_~1 skin_~2 eye_c~3 birth~4 sex   gender homew~5
##   <chr>         <int> <dbl> <chr>   <chr>   <chr>     <dbl> <chr> <chr>  <chr>  
## 1 Luke Skywal~    172    77 blond   fair    blue       19   male  mascu~ Tatooi~
## 2 C-3PO           167    75 <NA>    gold    yellow    112   none  mascu~ Tatooi~
## 3 R2-D2            96    32 <NA>    white,~ red        33   none  mascu~ Naboo  
## 4 Darth Vader     202   136 none    white   yellow     41.9 male  mascu~ Tatooi~
## 5 Leia Organa     150    49 brown   light   brown      19   fema~ femin~ Aldera~
## 6 Owen Lars       178   120 brown,~ light   blue       52   male  mascu~ Tatooi~
## # ... with 4 more variables: species <chr>, films <list>, vehicles <list>,
## #   starships <list>, and abbreviated variable names 1: hair_color,
## #   2: skin_color, 3: eye_color, 4: birth_year, 5: homeworld
\end{verbatim}

As you can see, there are missing values in the \texttt{hair\_color} column. Let's try to count how many
missing values are in this column:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{how\_many\_na}\NormalTok{(starwars, hair\_color)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
Error: object 'hair_color' not found
\end{verbatim}

Just as expected, this does not work. The issue is that the column is inside the dataframe,
but when calling the function with \texttt{hair\_color} as the second argument, R is looking for a
variable called \texttt{hair\_color} that does not exist. What about trying with \texttt{"hair\_color"}?

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{how\_many\_na}\NormalTok{(starwars, }\StringTok{"hair\_color"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 1
##       n
##   <int>
## 1     0
\end{verbatim}

Now we get something, but something wrong!

One way to solve this issue, is to not use the \texttt{filter()} function, and instead rely on base R:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{how\_many\_na\_base }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataframe, column\_name)\{}
\NormalTok{  na\_index }\OtherTok{\textless{}{-}} \FunctionTok{is.na}\NormalTok{(dataframe[, column\_name])}
  \FunctionTok{nrow}\NormalTok{(dataframe[na\_index, column\_name])}
\NormalTok{\}}

\FunctionTok{how\_many\_na\_base}\NormalTok{(starwars, }\StringTok{"hair\_color"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 5
\end{verbatim}

This works, but not using the \texttt{\{tidyverse\}} at all is not always an option. For instance,
the next function, which uses a grouping variable, would be difficult to implement without the
\texttt{\{tidyverse\}}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{summarise\_groups }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataframe, grouping\_var, column\_name)\{}
\NormalTok{  dataframe }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(grouping\_var) }\SpecialCharTok{\%\textgreater{}\%}  
    \FunctionTok{summarise}\NormalTok{(}\FunctionTok{mean}\NormalTok{(column\_name, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Calling this function results in the following error message, as expected:

\begin{verbatim}
Error: Column `grouping_var` is unknown
\end{verbatim}

In the previous section, we solved the issue like so:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{summarise\_groups }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataframe, grouping\_var, column\_name)\{}

\NormalTok{  grouping\_var }\OtherTok{\textless{}{-}} \FunctionTok{enquo}\NormalTok{(grouping\_var)}
\NormalTok{  column\_name }\OtherTok{\textless{}{-}} \FunctionTok{enquo}\NormalTok{(column\_name)}
\NormalTok{  mean\_name }\OtherTok{\textless{}{-}} \FunctionTok{paste0}\NormalTok{(}\StringTok{"mean\_"}\NormalTok{, }\FunctionTok{as\_label}\NormalTok{(column\_name))}

\NormalTok{  dataframe }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(}\SpecialCharTok{!!}\NormalTok{grouping\_var) }\SpecialCharTok{\%\textgreater{}\%}  
    \FunctionTok{summarise}\NormalTok{(}\SpecialCharTok{!!}\NormalTok{(mean\_name) }\SpecialCharTok{:}\ErrorTok{=} \FunctionTok{mean}\NormalTok{(}\SpecialCharTok{!!}\NormalTok{column\_name, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

The core of the function remained very similar to the version from before, but now one has to
use the \texttt{enquo()}-\texttt{!!} syntax.

Now this can be simplified using the new \texttt{\{\{\}\}} syntax:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{summarise\_groups }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataframe, grouping\_var, column\_name)\{}

\NormalTok{  dataframe }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(\{\{grouping\_var\}\}) }\SpecialCharTok{\%\textgreater{}\%}  
    \FunctionTok{summarise}\NormalTok{(\{\{column\_name\}\} }\SpecialCharTok{:}\ErrorTok{=} \FunctionTok{mean}\NormalTok{(\{\{column\_name\}\}, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Much easier and cleaner! You still have to use the \texttt{:=} operator instead of \texttt{=} for the column name
however, and if you want to modify the column names, for instance in this
case return \texttt{"mean\_height"} instead of \texttt{height} you have to keep using the \texttt{enquo()}-\texttt{!!} syntax.

\hypertarget{functions-that-use-loops}{%
\subsection{Functions that use loops}\label{functions-that-use-loops}}

It is entirely possible to put a loop inside a function. For example, consider the following
function that return the square root of a number using Newton's algorithm:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{sqrt\_newton }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(a, }\AttributeTok{init =} \DecValTok{1}\NormalTok{, }\AttributeTok{eps =} \FloatTok{0.01}\NormalTok{)\{}
    \FunctionTok{stopifnot}\NormalTok{(a }\SpecialCharTok{\textgreater{}=} \DecValTok{0}\NormalTok{)}
    \ControlFlowTok{while}\NormalTok{(}\FunctionTok{abs}\NormalTok{(init}\SpecialCharTok{**}\DecValTok{2} \SpecialCharTok{{-}}\NormalTok{ a) }\SpecialCharTok{\textgreater{}}\NormalTok{ eps)\{}
\NormalTok{        init }\OtherTok{\textless{}{-}} \DecValTok{1}\SpecialCharTok{/}\DecValTok{2} \SpecialCharTok{*}\NormalTok{(init }\SpecialCharTok{+}\NormalTok{ a}\SpecialCharTok{/}\NormalTok{init)}
\NormalTok{    \}}
\NormalTok{    init}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

This functions contains a while loop inside its body. Let's see if it works:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sqrt\_newton}\NormalTok{(}\DecValTok{16}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 4.000001
\end{verbatim}

In the definition of the function, I wrote \texttt{init\ =\ 1} and \texttt{eps\ =\ 0.01} which means that this
argument can be omitted and will have the provided value (0.01) as the default. You can then use
this function as any other, for example with \texttt{map()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{16}\NormalTok{, }\DecValTok{7}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{9}\NormalTok{, }\DecValTok{12}\NormalTok{), sqrt\_newton)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 4.000001
## 
## [[2]]
## [1] 2.645767
## 
## [[3]]
## [1] 2.828469
## 
## [[4]]
## [1] 3.000092
## 
## [[5]]
## [1] 3.464616
\end{verbatim}

This is what I meant before with ``your functions are nothing special''. Once the function is
defined, you can use it like any other base R function.

Notice the use of \texttt{stopifnot()} inside the body of the function. This is a way to return an error
in case a condition is not fulfilled. We are going to learn more about this type of functions
in the next chapter.

\hypertarget{anonymous-functions}{%
\subsection{Anonymous functions}\label{anonymous-functions}}

As the name implies, anonymous functions are functions that do not have a name. These are useful inside
functions that have functions as arguments, such as \texttt{purrr::map()} or \texttt{purrr::reduce()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{3}\NormalTok{,}\DecValTok{4}\NormalTok{), }\ControlFlowTok{function}\NormalTok{(x)\{}\DecValTok{1}\SpecialCharTok{/}\FunctionTok{sqrt}\NormalTok{(x)\})}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 1
## 
## [[2]]
## [1] 0.7071068
## 
## [[3]]
## [1] 0.5773503
## 
## [[4]]
## [1] 0.5
\end{verbatim}

These anonymous functions get defined in a very similar way to regular functions, you just skip the
name and that's it. \texttt{\{tidyverse\}} functions also support formulas; these get converted to anonymous functions:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{3}\NormalTok{,}\DecValTok{4}\NormalTok{), }\SpecialCharTok{\textasciitilde{}}\NormalTok{\{}\DecValTok{1}\SpecialCharTok{/}\FunctionTok{sqrt}\NormalTok{(.)\})}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 1
## 
## [[2]]
## [1] 0.7071068
## 
## [[3]]
## [1] 0.5773503
## 
## [[4]]
## [1] 0.5
\end{verbatim}

Using a formula instead of an anonymous function is less verbose; you use \texttt{\textasciitilde{}} instead of \texttt{function(x)}
and a single dot \texttt{.} instead of \texttt{x}. What if you need an anonymous function that requires more than
one argument? This is not a problem:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map2}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{4}\NormalTok{, }\DecValTok{5}\NormalTok{), }\FunctionTok{c}\NormalTok{(}\DecValTok{9}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{7}\NormalTok{, }\DecValTok{6}\NormalTok{, }\DecValTok{5}\NormalTok{), }\ControlFlowTok{function}\NormalTok{(x, y)\{(x}\SpecialCharTok{**}\DecValTok{2}\NormalTok{)}\SpecialCharTok{/}\NormalTok{y\})}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 0.1111111
## 
## [[2]]
## [1] 0.5
## 
## [[3]]
## [1] 1.285714
## 
## [[4]]
## [1] 2.666667
## 
## [[5]]
## [1] 5
\end{verbatim}

or, using a formula:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map2}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{4}\NormalTok{, }\DecValTok{5}\NormalTok{), }\FunctionTok{c}\NormalTok{(}\DecValTok{9}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{7}\NormalTok{, }\DecValTok{6}\NormalTok{, }\DecValTok{5}\NormalTok{), }\SpecialCharTok{\textasciitilde{}}\NormalTok{\{(.x}\SpecialCharTok{**}\DecValTok{2}\NormalTok{)}\SpecialCharTok{/}\NormalTok{.y\})}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 0.1111111
## 
## [[2]]
## [1] 0.5
## 
## [[3]]
## [1] 1.285714
## 
## [[4]]
## [1] 2.666667
## 
## [[5]]
## [1] 5
\end{verbatim}

Because you have now two arguments, a single dot could not work, so instead you use \texttt{.x} and \texttt{.y} to
avoid confusion.

Since version 4.1, R introduced a short-hand for defining anonymous functions:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map}\NormalTok{(}\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{2}\NormalTok{,}\DecValTok{3}\NormalTok{,}\DecValTok{4}\NormalTok{), \textbackslash{}(x)(}\DecValTok{1}\SpecialCharTok{/}\FunctionTok{sqrt}\NormalTok{(x)))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 1
## 
## [[2]]
## [1] 0.7071068
## 
## [[3]]
## [1] 0.5773503
## 
## [[4]]
## [1] 0.5
\end{verbatim}

\texttt{\textbackslash{}(x)} is supposed to look like this notation: \(\lambda(x)\). This is a notation comes from lambda calculus, where functions
are defined like this:

\[
\lambda(x).1/sqrt(x)
\]

which is equivalent to \(f(x) = 1/sqrt(x)\). You can use \texttt{\textbackslash{}(x)} or \texttt{function(x)} interchangeably.

You now know a lot about writing your own functions. In the next chapter, we are going to learn
about functional programming, the programming paradigm I described in the introduction of this
book.

\hypertarget{exercises-5}{%
\subsection{Exercises}\label{exercises-5}}

\hypertarget{exercise-1-5}{%
\subsubsection*{Exercise 1}\label{exercise-1-5}}
\addcontentsline{toc}{subsubsection}{Exercise 1}

\begin{itemize}
\tightlist
\item
  Create the following vector:
\end{itemize}

\[a = (1,6,7,8,8,9,2)\]

Using a for loop and a while loop, compute the sum of its elements. To avoid issues, use \texttt{i}
as the counter inside the for loop, and \texttt{j} as the counter for the while loop.

\begin{itemize}
\tightlist
\item
  How would you achieve that with a functional (a function that takes a function as an argument)?
\end{itemize}

\hypertarget{exercise-2-3}{%
\subsubsection*{Exercise 2}\label{exercise-2-3}}
\addcontentsline{toc}{subsubsection}{Exercise 2}

\begin{itemize}
\tightlist
\item
  Let's use a loop to get the matrix product of a matrix A and B. Follow these steps to create the loop:
\end{itemize}

\begin{enumerate}
\def\labelenumi{\arabic{enumi})}
\tightlist
\item
  Create matrix A:
\end{enumerate}

\[A = \left(
  \begin{array}{ccc}
   9 & 4 & 12 \\
   5 & 0 & 7 \\
   2 & 6 & 8 \\
   9 & 2 & 9
  \end{array} \right)
\]

\begin{enumerate}
\def\labelenumi{\arabic{enumi})}
\setcounter{enumi}{1}
\tightlist
\item
  Create matrix B:
\end{enumerate}

\[B = \left(
\begin{array}{cccc}
 5 & 4 & 2 & 5 \\
 2 & 7 & 2 & 1 \\
 8 & 3 & 2 & 6 \\
\end{array} \right)
\]

\begin{enumerate}
\def\labelenumi{\arabic{enumi})}
\setcounter{enumi}{2}
\item
  Create a matrix C, with dimension 4x4 that will hold the result. Use this command: `C = matrix(rep(0,16), nrow = 4)\}
\item
  Using a for loop, loop over the rows of A first: `for(i in 1:nrow(A))\}
\item
  Inside this loop, loop over the columns of B: `for(j in 1:ncol(B))\}
\item
  Again, inside this loop, loop over the rows of B: `for(k in 1:nrow(B))\}
\item
  Inside this last loop, compute the result and save it inside C: `C{[}i,j{]} = C{[}i,j{]} + A{[}i,k{]} * B{[}k,j{]}\}
\item
  Now write a function that takes two matrices as arguments, and returns their product.
\end{enumerate}

\begin{itemize}
\tightlist
\item
  R has a built-in function to compute the dot product of 2 matrices. Which is it?
\end{itemize}

\hypertarget{exercise-3-3}{%
\subsubsection*{Exercise 3}\label{exercise-3-3}}
\addcontentsline{toc}{subsubsection}{Exercise 3}

\begin{itemize}
\item
  Fizz Buzz: Print integers from 1 to 100. If a number is divisible by 3, print the word \texttt{"Fizz"} if
  it's divisible by 5, print \texttt{"Buzz"}. Use a for loop and if statements.
\item
  Write a function that takes an integer as arguments, and prints \texttt{"Fizz"} or \texttt{"Buzz"} up to that integer.
\end{itemize}

\hypertarget{exercise-4-2}{%
\subsubsection*{Exercise 4}\label{exercise-4-2}}
\addcontentsline{toc}{subsubsection}{Exercise 4}

\begin{itemize}
\item
  Fizz Buzz 2: Same as above, but now add this third condition: if a number is both divisible by 3 and 5, print \texttt{"FizzBuzz"}.
\item
  Write a function that takes an integer as argument, and prints \texttt{Fizz}, \texttt{Buzz} or \texttt{FizzBuzz} up to that integer.
\end{itemize}

\hypertarget{functional-programming}{%
\section{Functional programming}\label{functional-programming}}

Functional programming is a paradigm that I find very suitable for data science. In functional
programming, your code is organised into functions that perform the operations you need. Your scripts
will only be a sequence of calls to these functions, making them easier to understand. R is not a pure
functional programming language, so we need some self-discipline to apply pure functional programming
principles. However, these efforts are worth it, because pure functions are easier to debug, extend
and document. In this chapter, we are going to learn about functional programming principles that you
can adopt and start using to make your code better.

\hypertarget{function-definitions}{%
\subsection{Function definitions}\label{function-definitions}}

You should now be familiar with function definitions in R. Let's suppose you want to write a function
to compute the square root of a number and want to do so using Newton's algorithm:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{sqrt\_newton }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(a, init, }\AttributeTok{eps =} \FloatTok{0.01}\NormalTok{)\{}
    \ControlFlowTok{while}\NormalTok{(}\FunctionTok{abs}\NormalTok{(init}\SpecialCharTok{**}\DecValTok{2} \SpecialCharTok{{-}}\NormalTok{ a) }\SpecialCharTok{\textgreater{}}\NormalTok{ eps)\{}
\NormalTok{        init }\OtherTok{\textless{}{-}} \DecValTok{1}\SpecialCharTok{/}\DecValTok{2} \SpecialCharTok{*}\NormalTok{(init }\SpecialCharTok{+}\NormalTok{ a}\SpecialCharTok{/}\NormalTok{init)}
\NormalTok{    \}}
\NormalTok{    init}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

You can then use this function to get the square root of a number:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sqrt\_newton}\NormalTok{(}\DecValTok{16}\NormalTok{, }\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 4.00122
\end{verbatim}

We are using a \texttt{while} loop inside the body of the function. The \emph{body} of a function are the
instructions that define the function. You can get the body of a function with \texttt{body(some\_func)}.
In \emph{pure} functional programming languages, like Haskell, loops do not exist. How can you
program without loops, you may ask? In functional programming, loops are replaced by recursion,
which we already discussed in the previous chapter. Let's rewrite our little example above
with recursion:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{sqrt\_newton\_recur }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(a, init, }\AttributeTok{eps =} \FloatTok{0.01}\NormalTok{)\{}
    \ControlFlowTok{if}\NormalTok{(}\FunctionTok{abs}\NormalTok{(init}\SpecialCharTok{**}\DecValTok{2} \SpecialCharTok{{-}}\NormalTok{ a) }\SpecialCharTok{\textless{}}\NormalTok{ eps)\{}
\NormalTok{        result }\OtherTok{\textless{}{-}}\NormalTok{ init}
\NormalTok{    \} }\ControlFlowTok{else}\NormalTok{ \{}
\NormalTok{        init }\OtherTok{\textless{}{-}} \DecValTok{1}\SpecialCharTok{/}\DecValTok{2} \SpecialCharTok{*}\NormalTok{ (init }\SpecialCharTok{+}\NormalTok{ a}\SpecialCharTok{/}\NormalTok{init)}
\NormalTok{        result }\OtherTok{\textless{}{-}} \FunctionTok{sqrt\_newton\_recur}\NormalTok{(a, init, eps)}
\NormalTok{    \}}
\NormalTok{    result}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sqrt\_newton\_recur}\NormalTok{(}\DecValTok{16}\NormalTok{, }\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 4.00122
\end{verbatim}

R is not a pure functional programming language though, so we can still use loops (be it \texttt{while} or
\texttt{for} loops) in the bodies of our functions. As discussed in the previous chapter, it is actually
better, performance-wise, to use loops instead of recursion, because R is not tail-call optimized.
I won't got into the details of what tail-call optimization is but just remember that if
performance is important a loop will be faster. However, sometimes, it is easier to write a
function using recursion. I personally tend to avoid loops if performance is not important,
because I find that code that avoids loops is easier to read and debug. However, knowing that
you can use loops is reassuring, and encapsulating loops inside functions gives you the benefits of
both using functions, and loops. In the coming sections I will show you some built-in functions
that make it possible to avoid writing loops and that don't rely on recursion, so performance
won't be penalized.

\hypertarget{properties-of-functions}{%
\subsection{Properties of functions}\label{properties-of-functions}}

Mathematical functions have a nice property: we always get the same output for a given input. This
is called referential transparency and we should aim to write our R functions in such a way.
For example, the following function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{increment }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x)\{}
\NormalTok{    x }\SpecialCharTok{+} \DecValTok{1}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Is a referential transparent function. We always get the same result for any \texttt{x} that we give to
this function.

This:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{increment}\NormalTok{(}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 11
\end{verbatim}

will always produce \texttt{11}.

However, this one:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{increment\_opaque }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x)\{}
\NormalTok{    x }\SpecialCharTok{+}\NormalTok{ spam}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

is not a referential transparent function, because its value depends on the global variable \texttt{spam}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{spam }\OtherTok{\textless{}{-}} \DecValTok{1}

\FunctionTok{increment\_opaque}\NormalTok{(}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 11
\end{verbatim}

will produce \texttt{11} if \texttt{spam\ =\ 1}. But what if \texttt{spam\ =\ 19}?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{spam }\OtherTok{\textless{}{-}} \DecValTok{19}

\FunctionTok{increment\_opaque}\NormalTok{(}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 29
\end{verbatim}

To make \texttt{increment\_opaque()} a referential transparent function, it is enough to make \texttt{spam} an
argument:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{increment\_not\_opaque }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x, spam)\{}
\NormalTok{    x }\SpecialCharTok{+}\NormalTok{ spam}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Now even if there is a global variable called \texttt{spam}, this will not influence our function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{spam }\OtherTok{\textless{}{-}} \DecValTok{19}

\FunctionTok{increment\_not\_opaque}\NormalTok{(}\DecValTok{10}\NormalTok{, }\DecValTok{34}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 44
\end{verbatim}

This is because the variable \texttt{spam} defined in the body of the function is a local variable. It
could have been called anything else, really. Avoiding opaque functions makes our life easier.

Another property that adepts of functional programming value is that functions should have no, or
very limited, side-effects. This means that functions should not change the state of your program.

For example this function (which is not a referential transparent function):

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{count\_iter }\OtherTok{\textless{}{-}} \DecValTok{0}

\NormalTok{sqrt\_newton\_side\_effect }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(a, init, }\AttributeTok{eps =} \FloatTok{0.01}\NormalTok{)\{}
    \ControlFlowTok{while}\NormalTok{(}\FunctionTok{abs}\NormalTok{(init}\SpecialCharTok{**}\DecValTok{2} \SpecialCharTok{{-}}\NormalTok{ a) }\SpecialCharTok{\textgreater{}}\NormalTok{ eps)\{}
\NormalTok{        init }\OtherTok{\textless{}{-}} \DecValTok{1}\SpecialCharTok{/}\DecValTok{2} \SpecialCharTok{*}\NormalTok{(init }\SpecialCharTok{+}\NormalTok{ a}\SpecialCharTok{/}\NormalTok{init)}
\NormalTok{        count\_iter }\OtherTok{\textless{}\textless{}{-}}\NormalTok{ count\_iter }\SpecialCharTok{+} \DecValTok{1} \CommentTok{\# The "\textless{}\textless{}{-}" symbol means that we assign the}
\NormalTok{    \}                                 }\CommentTok{\# RHS value in a variable inside the global environment}
\NormalTok{    init}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

If you look in the environment pane, you will see that \texttt{count\_iter} equals 0. Now call this
function with the following arguments:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sqrt\_newton\_side\_effect}\NormalTok{(}\DecValTok{16000}\NormalTok{, }\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 126.4911
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{print}\NormalTok{(count\_iter)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 9
\end{verbatim}

If you check the value of \texttt{count\_iter} now, you will see that it increased! This is a side effect,
because the function changed something outside of its scope. It changed a value in the global
environment. In general, it is good practice to avoid side-effects. For example, we could make the
above function not have any side effects like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{sqrt\_newton\_count }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(a, init, }\AttributeTok{count\_iter =} \DecValTok{0}\NormalTok{, }\AttributeTok{eps =} \FloatTok{0.01}\NormalTok{)\{}
    \ControlFlowTok{while}\NormalTok{(}\FunctionTok{abs}\NormalTok{(init}\SpecialCharTok{**}\DecValTok{2} \SpecialCharTok{{-}}\NormalTok{ a) }\SpecialCharTok{\textgreater{}}\NormalTok{ eps)\{}
\NormalTok{        init }\OtherTok{\textless{}{-}} \DecValTok{1}\SpecialCharTok{/}\DecValTok{2} \SpecialCharTok{*}\NormalTok{(init }\SpecialCharTok{+}\NormalTok{ a}\SpecialCharTok{/}\NormalTok{init)}
\NormalTok{        count\_iter }\OtherTok{\textless{}{-}}\NormalTok{ count\_iter }\SpecialCharTok{+} \DecValTok{1}
\NormalTok{    \}}
    \FunctionTok{c}\NormalTok{(init, count\_iter)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Now, this function returns a list with two elements, the result, and the number of iterations it
took to get the result:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sqrt\_newton\_count}\NormalTok{(}\DecValTok{16000}\NormalTok{, }\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 126.4911   9.0000
\end{verbatim}

Writing to disk is also considered a side effect, because the function changes something (a file)
outside its scope. But this cannot be avoided since you \emph{want} to write to disk.
Just remember: try to avoid having functions changing variables in the global environment unless
you have a very good reason of doing so.

Very long scripts that don't use functions and use a lot of global variables with loops changing
the values of global variables are a nightmare to debug. If something goes wrong, it might be very
difficult to pinpoint where the problem is. Is there an error in one of the loops?
Is your code running for a particular value of a particular variable in the global environment, but
not for other values? Which values? And of which variables? It can be very difficult to know what
is wrong with such a script.
With functional programming, you can avoid a lot of this pain for free (well not entirely for free,
it still requires some effort, since R is not a pure functional language). Writing functions also
makes it easier to parallelize your code. We are going to learn about that later in this chapter too.

Finally, another property of mathematical functions, is that they do one single thing. Functional
programming purists also program their functions to do one single task. This has benefits, but
can complicate things. The function we wrote previously does two things: it computes the square
root of a number and also returns the number of iterations it took to compute the result. However,
this is not a bad thing; the function is doing two tasks, but these tasks are related to each other
and it makes sense to have them together. My piece of advice: avoid having functions that do
many \emph{unrelated} things. This makes debugging harder.

In conclusion: you should strive for referential transparency, try to avoid side effects unless you
have a good reason to have them and try to keep your functions short and do as little tasks as
possible. This makes testing and debugging easier, as you will see in the next chapter, but also
improves readability and maintainability of your code.

\hypertarget{functional-programming-with-purrr}{%
\subsection{\texorpdfstring{Functional programming with \texttt{\{purrr\}}}{Functional programming with \{purrr\}}}\label{functional-programming-with-purrr}}

I mentioned it several times already, but R is not a pure functional programming language. It is
possible to write R code using the functional programming paradigm, but some effort is required.
The \texttt{\{purrr\}} package extends R's base functional programming capabilities with some very interesting
functions. We have already seen \texttt{map()} and \texttt{reduce()}, which we are going to see in more detail now.
Then, we are going to learn about some other functions included in \texttt{\{purrr\}} that make functional
programming easier in R.

\hypertarget{doing-away-with-loops-the-map-family-of-functions}{%
\subsubsection{\texorpdfstring{Doing away with loops: the \texttt{map*()} family of functions}{Doing away with loops: the map*() family of functions}}\label{doing-away-with-loops-the-map-family-of-functions}}

Instead of using loops, pure functional programming languages use functions that achieve
the same result. These functions are often called \texttt{Map} or \texttt{Reduce} (also called \texttt{Fold}). R comes
with the \texttt{*apply()} family of functions (which are implementations of \texttt{Map}),
as well as \texttt{Reduce()} for functional programming.

Within this family, you can find \texttt{lapply()}, \texttt{sapply()}, \texttt{vapply()}, \texttt{tapply()}, \texttt{mapply()}, \texttt{rapply()},
\texttt{eapply()} and \texttt{apply()} (I might have forgotten one or the other, but that's not important).
Each version of an \texttt{*apply()} function has a different purpose, but it is not very easy to
remember which does what exactly. To add even more confusion, the arguments are sometimes different between
each of these.

In the \texttt{\{purrr\}} package, these functions are replaced by the \texttt{map*()} family of functions. As you will
shortly see, they are very consistent, and thus easier to use.
The first part of these functions' names all start with \texttt{map\_} and the second part tells you what
this function is going to return. For example, if you want \texttt{double}s out, you would use \texttt{map\_dbl()}.
If you are working on data frames and want a data frame back, you would use \texttt{map\_df()}. Let's start
with the basic \texttt{map()} function. The following gif
(source: \href{https://en.wikipedia.org/wiki/Map_(higher-order_function)}{Wikipedia}) illustrates
what \texttt{map()} does fairly well:

\includegraphics{https://upload.wikimedia.org/wikipedia/commons/0/06/Mapping-steps-loillibe-new.gif}

\(X\) is a vector composed of the following scalars: \((0, 5, 8, 3, 2, 1)\). The function we want to
map to each element of \(X\) is \(f(x) = x + 1\). \(X'\) is the result of this operation. Using R, we
would do the following:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(}\StringTok{"purrr"}\NormalTok{)}
\NormalTok{numbers }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\DecValTok{0}\NormalTok{, }\DecValTok{5}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{1}\NormalTok{)}

\NormalTok{plus\_one }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x) (x }\SpecialCharTok{+} \DecValTok{1}\NormalTok{)}

\NormalTok{my\_results }\OtherTok{\textless{}{-}} \FunctionTok{map}\NormalTok{(numbers, plus\_one)}

\NormalTok{my\_results}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 1
## 
## [[2]]
## [1] 6
## 
## [[3]]
## [1] 9
## 
## [[4]]
## [1] 4
## 
## [[5]]
## [1] 3
## 
## [[6]]
## [1] 2
\end{verbatim}

Using a loop, you would write:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{numbers }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\DecValTok{0}\NormalTok{, }\DecValTok{5}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{1}\NormalTok{)}

\NormalTok{plus\_one }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x) (x }\SpecialCharTok{+} \DecValTok{1}\NormalTok{)}

\NormalTok{my\_results }\OtherTok{\textless{}{-}} \FunctionTok{vector}\NormalTok{(}\StringTok{"list"}\NormalTok{, }\DecValTok{6}\NormalTok{)}

\ControlFlowTok{for}\NormalTok{(number }\ControlFlowTok{in} \FunctionTok{seq\_along}\NormalTok{(numbers))\{}
\NormalTok{  my\_results[[number]] }\OtherTok{\textless{}{-}} \FunctionTok{plus\_one}\NormalTok{(number)}
\NormalTok{\}}

\NormalTok{my\_results}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 2
## 
## [[2]]
## [1] 3
## 
## [[3]]
## [1] 4
## 
## [[4]]
## [1] 5
## 
## [[5]]
## [1] 6
## 
## [[6]]
## [1] 7
\end{verbatim}

Now I don't know about you, but I prefer the first option. Using functional programming, you don't
need to create an empty list to hold your results, and the code is more concise. Plus,
it is less error prone. I had to try several times to get the loop right
(and I've using R for almost 10 years now). Why? Well, first of all I used \texttt{\%in\%} instead of \texttt{in}.
Then, I forgot about \texttt{seq\_along()}. After that, I made a typo, \texttt{plos\_one()} instead of \texttt{plus\_one()}
(ok, that one is unrelated to the loop). Let's also see how this works using base R:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{numbers }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\DecValTok{0}\NormalTok{, }\DecValTok{5}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{1}\NormalTok{)}

\NormalTok{plus\_one }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x) (x }\SpecialCharTok{+} \DecValTok{1}\NormalTok{)}

\NormalTok{my\_results }\OtherTok{\textless{}{-}} \FunctionTok{lapply}\NormalTok{(numbers, plus\_one)}

\NormalTok{my\_results}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 1
## 
## [[2]]
## [1] 6
## 
## [[3]]
## [1] 9
## 
## [[4]]
## [1] 4
## 
## [[5]]
## [1] 3
## 
## [[6]]
## [1] 2
\end{verbatim}

So what is the added value of using \texttt{\{purrr\}}, you might ask. Well, imagine that instead of a list,
I need to an atomic vector of \texttt{numeric}s. This is fairly easy with \texttt{\{purrr\}}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(}\StringTok{"purrr"}\NormalTok{)}
\NormalTok{numbers }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\DecValTok{0}\NormalTok{, }\DecValTok{5}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{1}\NormalTok{)}

\NormalTok{plus\_one }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x) (x }\SpecialCharTok{+} \DecValTok{1}\NormalTok{)}

\NormalTok{my\_results }\OtherTok{\textless{}{-}} \FunctionTok{map\_dbl}\NormalTok{(numbers, plus\_one)}

\NormalTok{my\_results}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1 6 9 4 3 2
\end{verbatim}

We're going to discuss these functions below, but know that in base R, outputting something else
involves more effort.

Let's go back to our \texttt{sqrt\_newton()} function. This function has more than one parameter. Often,
we would like to map functions with more than one parameter to a list, while holding constant
some of the functions parameters. This is easily achieved like so:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(}\StringTok{"purrr"}\NormalTok{)}
\NormalTok{numbers }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\DecValTok{7}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{19}\NormalTok{, }\DecValTok{64}\NormalTok{)}

\FunctionTok{map}\NormalTok{(numbers, sqrt\_newton, }\AttributeTok{init =} \DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 2.645767
## 
## [[2]]
## [1] 2.828469
## 
## [[3]]
## [1] 4.358902
## 
## [[4]]
## [1] 8.000002
\end{verbatim}

It is also possible to use a formula:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(}\StringTok{"purrr"}\NormalTok{)}
\NormalTok{numbers }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\DecValTok{7}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{19}\NormalTok{, }\DecValTok{64}\NormalTok{)}

\FunctionTok{map}\NormalTok{(numbers, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{sqrt\_newton}\NormalTok{(., }\AttributeTok{init =} \DecValTok{1}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 2.645767
## 
## [[2]]
## [1] 2.828469
## 
## [[3]]
## [1] 4.358902
## 
## [[4]]
## [1] 8.000002
\end{verbatim}

Another function that is similar to \texttt{map()} is \texttt{rerun()}. You guessed it, this one simply
reruns an expression:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{rerun}\NormalTok{(}\DecValTok{10}\NormalTok{, }\StringTok{"hello"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] "hello"
## 
## [[2]]
## [1] "hello"
## 
## [[3]]
## [1] "hello"
## 
## [[4]]
## [1] "hello"
## 
## [[5]]
## [1] "hello"
## 
## [[6]]
## [1] "hello"
## 
## [[7]]
## [1] "hello"
## 
## [[8]]
## [1] "hello"
## 
## [[9]]
## [1] "hello"
## 
## [[10]]
## [1] "hello"
\end{verbatim}

\texttt{rerun()} simply runs an expression (which can be arbitrarily complex) \texttt{n} times, whereas \texttt{map()}
maps a function to a list of inputs, so to achieve the same with \texttt{map()}, you need to map the \texttt{print()}
function to a vector of characters:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map}\NormalTok{(}\FunctionTok{rep}\NormalTok{(}\StringTok{"hello"}\NormalTok{, }\DecValTok{10}\NormalTok{), print)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
## [1] "hello"
\end{verbatim}

\begin{verbatim}
## [[1]]
## [1] "hello"
## 
## [[2]]
## [1] "hello"
## 
## [[3]]
## [1] "hello"
## 
## [[4]]
## [1] "hello"
## 
## [[5]]
## [1] "hello"
## 
## [[6]]
## [1] "hello"
## 
## [[7]]
## [1] "hello"
## 
## [[8]]
## [1] "hello"
## 
## [[9]]
## [1] "hello"
## 
## [[10]]
## [1] "hello"
\end{verbatim}

\texttt{rep()} is a function that creates a vector by repeating something, in this case the string ``hello'',
as many times as needed, here 10. The output here is a bit different that before though, because first
you will see ``hello'' printed 10 times and then the list where each element is ``hello''.
This is because the \texttt{print()} function has a side effect, which is, well printing to the console.
We see this side effect 10 times, plus then the list created with \texttt{map()}.

\texttt{rerun()} is useful if you want to run simulation. For instance, let's suppose that I perform a simulation
where I throw a die 5 times, and compute the mean of the points obtained, as well as the variance:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mean\_var\_throws }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(n)\{}
\NormalTok{  throws }\OtherTok{\textless{}{-}} \FunctionTok{sample}\NormalTok{(}\DecValTok{1}\SpecialCharTok{:}\DecValTok{6}\NormalTok{, n, }\AttributeTok{replace =} \ConstantTok{TRUE}\NormalTok{)}

\NormalTok{  mean\_throws }\OtherTok{\textless{}{-}} \FunctionTok{mean}\NormalTok{(throws)}
\NormalTok{  var\_throws }\OtherTok{\textless{}{-}} \FunctionTok{var}\NormalTok{(throws)}

\NormalTok{  tibble}\SpecialCharTok{::}\FunctionTok{tribble}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\NormalTok{mean\_throws, }\SpecialCharTok{\textasciitilde{}}\NormalTok{var\_throws,}
\NormalTok{                   mean\_throws, var\_throws)}
\NormalTok{\}}

\FunctionTok{mean\_var\_throws}\NormalTok{(}\DecValTok{5}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 2
##   mean_throws var_throws
##         <dbl>      <dbl>
## 1         2.2        1.7
\end{verbatim}

\texttt{mean\_var\_throws()} returns a \texttt{tibble} object with mean of points and the variance of the points. Now suppose
I want to compute the expected value of the distribution of throwing dice. We know from theory that it should
be equal to \(3.5 (= 1*1/6 + 2*1/6 + 3*1/6 + 4*1/6 + 5*1/6 + 6*1/6)\).

Let's rerun the simulation 50 times:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simulations }\OtherTok{\textless{}{-}} \FunctionTok{rerun}\NormalTok{(}\DecValTok{50}\NormalTok{, }\FunctionTok{mean\_var\_throws}\NormalTok{(}\DecValTok{5}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

Let's see what the \texttt{simulations} object is made of:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{str}\NormalTok{(simulations)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 50
##  $ :Classes 'tbl_df', 'tbl' and 'data.frame':    1 obs. of  2 variables:
##   ..$ mean_throws: num 2
##   ..$ var_throws : num 3
##  $ :Classes 'tbl_df', 'tbl' and 'data.frame':    1 obs. of  2 variables:
##   ..$ mean_throws: num 2.8
##   ..$ var_throws : num 0.2
##  $ :Classes 'tbl_df', 'tbl' and 'data.frame':    1 obs. of  2 variables:
##   ..$ mean_throws: num 2.8
##   ..$ var_throws : num 0.7
##  $ :Classes 'tbl_df', 'tbl' and 'data.frame':    1 obs. of  2 variables:
##   ..$ mean_throws: num 2.8
##   ..$ var_throws : num 1.7
.....
\end{verbatim}

\texttt{simulations} is a list of 50 data frames. We can easily combine them into a single data frame, and compute the
mean of the means, which should return something close to the expected value of 3.5:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{bind\_rows}\NormalTok{(simulations) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{expected\_value =} \FunctionTok{mean}\NormalTok{(mean\_throws))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 1
##   expected_value
##            <dbl>
## 1           3.44
\end{verbatim}

Pretty close! Now of course, one could have simply done something like this:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{mean}\NormalTok{(}\FunctionTok{sample}\NormalTok{(}\DecValTok{1}\SpecialCharTok{:}\DecValTok{6}\NormalTok{, }\DecValTok{1000}\NormalTok{, }\AttributeTok{replace =} \ConstantTok{TRUE}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 3.481
\end{verbatim}

but the point was to illustrate that \texttt{rerun()} can run any arbitrarily complex expression, and that it is good
practice to put the result in a data frame or list, for easier further manipulation.

You now know the standard \texttt{map()} function, and also \texttt{rerun()}, which return lists, but there are a
number of variants of this function. \texttt{map\_dbl()} returns an atomic vector of doubles, as seen
we've seen before. A little reminder below:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map\_dbl}\NormalTok{(numbers, sqrt\_newton, }\AttributeTok{init =} \DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 2.645767 2.828469 4.358902 8.000002
\end{verbatim}

In a similar fashion, \texttt{map\_chr()} returns an atomic vector of strings:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map\_chr}\NormalTok{(numbers, sqrt\_newton, }\AttributeTok{init =} \DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] "2.645767" "2.828469" "4.358902" "8.000002"
\end{verbatim}

\texttt{map\_lgl()} returns an atomic vector of \texttt{TRUE} or \texttt{FALSE}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{divisible }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x, y)\{}
  \FunctionTok{if\_else}\NormalTok{(x }\SpecialCharTok{\%\%}\NormalTok{ y }\SpecialCharTok{==} \DecValTok{0}\NormalTok{, }\ConstantTok{TRUE}\NormalTok{, }\ConstantTok{FALSE}\NormalTok{)}
\NormalTok{\}}

\FunctionTok{map\_lgl}\NormalTok{(}\FunctionTok{seq}\NormalTok{(}\DecValTok{1}\SpecialCharTok{:}\DecValTok{100}\NormalTok{), divisible, }\DecValTok{3}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##   [1] FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE
##  [13] FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE
##  [25] FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE
##  [37] FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE
##  [49] FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE
##  [61] FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE
##  [73] FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE
##  [85] FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE
##  [97] FALSE FALSE  TRUE FALSE
\end{verbatim}

There are also other interesting variants, such as \texttt{map\_if()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \FunctionTok{seq}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{10}\NormalTok{)}

\FunctionTok{map\_if}\NormalTok{(a, (}\ControlFlowTok{function}\NormalTok{(x) }\FunctionTok{divisible}\NormalTok{(x, }\DecValTok{2}\NormalTok{)), sqrt)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 1
## 
## [[2]]
## [1] 1.414214
## 
## [[3]]
## [1] 3
## 
## [[4]]
## [1] 2
## 
## [[5]]
## [1] 5
## 
## [[6]]
## [1] 2.44949
## 
## [[7]]
## [1] 7
## 
## [[8]]
## [1] 2.828427
## 
## [[9]]
## [1] 9
## 
## [[10]]
## [1] 3.162278
\end{verbatim}

I used \texttt{map\_if()} to take the square root of only those numbers in vector \texttt{a} that are divisble by 2,
by using an anonymous function that checks if a number is divisible by 2 (by wrapping \texttt{divisible()}).

\texttt{map\_at()} is similar to \texttt{map\_if()} but maps the function at a position specified by the user:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map\_at}\NormalTok{(numbers, }\FunctionTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{3}\NormalTok{), sqrt)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 2.645751
## 
## [[2]]
## [1] 8
## 
## [[3]]
## [1] 4.358899
## 
## [[4]]
## [1] 64
\end{verbatim}

or if you have a named list:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{recipe }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\StringTok{"spam"} \OtherTok{=} \DecValTok{1}\NormalTok{, }\StringTok{"eggs"} \OtherTok{=} \DecValTok{3}\NormalTok{, }\StringTok{"bacon"} \OtherTok{=} \DecValTok{10}\NormalTok{)}

\FunctionTok{map\_at}\NormalTok{(recipe, }\StringTok{"bacon"}\NormalTok{, }\StringTok{\textasciigrave{}}\AttributeTok{*}\StringTok{\textasciigrave{}}\NormalTok{, }\DecValTok{2}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## $spam
## [1] 1
## 
## $eggs
## [1] 3
## 
## $bacon
## [1] 20
\end{verbatim}

I used \texttt{map\_at()} to double the quantity of bacon in the recipe (by using the \texttt{*} function, and specifying
its second argument, \texttt{2}. Try the following in the command prompt: \texttt{\textasciigrave{}*\textasciigrave{}(3,\ 4)}).

\texttt{map2()} is the equivalent of \texttt{mapply()} and \texttt{pmap()} is the generalisation of \texttt{map2()} for more
than 2 arguments:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{print}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1]  1  2  3  4  5  6  7  8  9 10
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{b }\OtherTok{\textless{}{-}} \FunctionTok{seq}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{, }\AttributeTok{length.out =} \DecValTok{10}\NormalTok{)}

\FunctionTok{print}\NormalTok{(b)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] 1.000000 1.111111 1.222222 1.333333 1.444444 1.555556 1.666667 1.777778
##  [9] 1.888889 2.000000
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map2}\NormalTok{(a, b, }\StringTok{\textasciigrave{}}\AttributeTok{*}\StringTok{\textasciigrave{}}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 1
## 
## [[2]]
## [1] 2.222222
## 
## [[3]]
## [1] 3.666667
## 
## [[4]]
## [1] 5.333333
## 
## [[5]]
## [1] 7.222222
## 
## [[6]]
## [1] 9.333333
## 
## [[7]]
## [1] 11.66667
## 
## [[8]]
## [1] 14.22222
## 
## [[9]]
## [1] 17
## 
## [[10]]
## [1] 20
\end{verbatim}

Each element of \texttt{a} gets multiplied by the element of \texttt{b} that is in the same position.
Let's see what \texttt{pmap()} does. Can you guess from the code below what is going on? I will print
\texttt{a} and \texttt{b} again for clarity:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1]  1  2  3  4  5  6  7  8  9 10
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{b}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1] 1.000000 1.111111 1.222222 1.333333 1.444444 1.555556 1.666667 1.777778
##  [9] 1.888889 2.000000
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{n }\OtherTok{\textless{}{-}} \FunctionTok{seq}\NormalTok{(}\DecValTok{1}\SpecialCharTok{:}\DecValTok{10}\NormalTok{)}

\FunctionTok{pmap}\NormalTok{(}\FunctionTok{list}\NormalTok{(a, b, n), rnorm)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] -0.1758315
## 
## [[2]]
## [1] -0.2162863  1.1033912
## 
## [[3]]
## [1]  4.5731231 -0.3743379  6.8130737
## 
## [[4]]
## [1]  0.8933089  4.1930837  7.5276030 -2.3575522
## 
## [[5]]
## [1]  2.1814981 -1.7455750  5.0548288  2.7848458  0.9230675
## 
## [[6]]
## [1]  2.806217  5.667499 -5.032922  6.741065 -2.757928 12.414101
## 
## [[7]]
## [1] -3.314145 -7.912019 -3.865292  4.307842 18.022049  1.278158  1.083208
## 
## [[8]]
## [1]  6.2629161  2.1213552  0.3543566  2.1041606 -0.2643654  8.7600450  3.3616206
## [8] -7.7446668
## 
## [[9]]
## [1]  -7.609538   5.472267  -4.869374 -11.943063   4.707929  -7.730088  13.431771
## [8]   1.606800  -6.578745
## 
## [[10]]
##  [1]  -9.101480   4.404571 -16.071437   1.110689   7.168097  15.848579
##  [7]  16.710863   1.998482 -17.856521  -2.021087
\end{verbatim}

Let's take a closer look at what \texttt{a}, \texttt{b} and \texttt{n} look like, when they are place next to each other:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{cbind}\NormalTok{(a, b, n)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##        a        b  n
##  [1,]  1 1.000000  1
##  [2,]  2 1.111111  2
##  [3,]  3 1.222222  3
##  [4,]  4 1.333333  4
##  [5,]  5 1.444444  5
##  [6,]  6 1.555556  6
##  [7,]  7 1.666667  7
##  [8,]  8 1.777778  8
##  [9,]  9 1.888889  9
## [10,] 10 2.000000 10
\end{verbatim}

\texttt{rnorm()} gets first called with the parameters from the first line, meaning
\texttt{rnorm(a{[}1{]},\ b{[}1{]},\ n{[}1{]})}. The second time \texttt{rnorm()} gets called, you guessed it,
it with the parameters on the second line of the array above,
\texttt{rnorm(a{[}2{]},\ b{[}2{]},\ n{[}2{]})}, etc.

There are other functions in the \texttt{map()} family of functions, but we will discover them in the
exercises!

The \texttt{map()} family of functions does not have any more secrets for you. Let's now take a look at
the \texttt{reduce()} family of functions.

\hypertarget{reducing-with-purrr}{%
\subsubsection{\texorpdfstring{Reducing with \texttt{purrr}}{Reducing with purrr}}\label{reducing-with-purrr}}

Reducing is another important concept in functional programming. It allows going from a list of
elements, to a single element, by somehow \emph{combining} the elements into one. For instance, using
the base R \texttt{Reduce()} function, you can sum the elements of a list like so:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{Reduce}\NormalTok{(}\StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{, }\FunctionTok{seq}\NormalTok{(}\DecValTok{1}\SpecialCharTok{:}\DecValTok{100}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 5050
\end{verbatim}

using \texttt{purrr::reduce()}, this becomes:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{reduce}\NormalTok{(}\FunctionTok{seq}\NormalTok{(}\DecValTok{1}\SpecialCharTok{:}\DecValTok{100}\NormalTok{), }\StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 5050
\end{verbatim}

If you don't really get what happening, don't worry. Things should get clearer once I'll introduce
another version of \texttt{reduce()}, called \texttt{accumulate()}, which we will see below.

Sometimes, the direction from which we start to reduce is quite important. You can ``start from the
end'' of the list by using the \texttt{.dir} argument:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{reduce}\NormalTok{(}\FunctionTok{seq}\NormalTok{(}\DecValTok{1}\SpecialCharTok{:}\DecValTok{100}\NormalTok{), }\StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{, }\AttributeTok{.dir =} \StringTok{"backward"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 5050
\end{verbatim}

Of course, for commutative operations, direction does not matter. But it does matter for non-commutative
operations:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{reduce}\NormalTok{(}\FunctionTok{seq}\NormalTok{(}\DecValTok{1}\SpecialCharTok{:}\DecValTok{100}\NormalTok{), }\StringTok{\textasciigrave{}}\AttributeTok{{-}}\StringTok{\textasciigrave{}}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] -5048
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{reduce}\NormalTok{(}\FunctionTok{seq}\NormalTok{(}\DecValTok{1}\SpecialCharTok{:}\DecValTok{100}\NormalTok{), }\StringTok{\textasciigrave{}}\AttributeTok{{-}}\StringTok{\textasciigrave{}}\NormalTok{, }\AttributeTok{.dir =} \StringTok{"backward"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] -50
\end{verbatim}

Let's now take a look at \texttt{accumulate()}. \texttt{accumulate()} is very similar to \texttt{map()}, but keeps the
intermediary results. Which intermediary results? Let's try and see what happens:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \FunctionTok{seq}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{10}\NormalTok{)}

\FunctionTok{accumulate}\NormalTok{(a, }\StringTok{\textasciigrave{}}\AttributeTok{{-}}\StringTok{\textasciigrave{}}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##  [1]   1  -1  -4  -8 -13 -19 -26 -34 -43 -53
\end{verbatim}

\texttt{accumulate()} illustrates pretty well what is happening; the first element, \texttt{1}, is simply the
first element of \texttt{seq(1,\ 10)}. The second element of the result however, is the difference between
\texttt{1} and \texttt{2}, \texttt{-1}. The next element in \texttt{a} is \texttt{3}. Thus the next result is \texttt{-1-3}, \texttt{-4}, and so
on until we run out of elements in \texttt{a}.

The below illustration shows the algorithm step-by-step:

\begin{verbatim}
(1-2-3-4-5-6-7-8-9-10)
((1)-2-3-4-5-6-7-8-9-10)
((1-2)-3-4-5-6-7-8-9-10)
((-1-3)-4-5-6-7-8-9-10)
((-4-4)-5-6-7-8-9-10)
((-8-5)-6-7-8-9-10)
((-13-6)-7-8-9-10)
((-19-7)-8-9-10)
((-26-8)-9-10)
((-34-9)-10)
(-43-10)
-53
\end{verbatim}

\texttt{reduce()} only shows the final result of all these operations. \texttt{accumulate()} and \texttt{reduce()} also
have an \texttt{.init} argument, that makes it possible to start the reducing procedure from an initial
value that is different from the first element of the vector:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{reduce}\NormalTok{(a, }\StringTok{\textasciigrave{}}\AttributeTok{+}\StringTok{\textasciigrave{}}\NormalTok{, }\AttributeTok{.init =} \DecValTok{1000}\NormalTok{)}

\FunctionTok{accumulate}\NormalTok{(a, }\StringTok{\textasciigrave{}}\AttributeTok{{-}}\StringTok{\textasciigrave{}}\NormalTok{, }\AttributeTok{.init =} \DecValTok{1000}\NormalTok{, }\AttributeTok{.dir =} \StringTok{"backward"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1055
\end{verbatim}

\begin{verbatim}
##  [1]  995 -994  996 -993  997 -992  998 -991  999 -990 1000
\end{verbatim}

\texttt{reduce()} generalizes functions that only take two arguments. If you were to write a function that returns
the minimum between two numbers:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{my\_min }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(a, b)\{}
    \ControlFlowTok{if}\NormalTok{(a }\SpecialCharTok{\textless{}}\NormalTok{ b)\{}
        \FunctionTok{return}\NormalTok{(a)}
\NormalTok{    \} }\ControlFlowTok{else}\NormalTok{ \{}
        \FunctionTok{return}\NormalTok{(b)}
\NormalTok{    \}}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

You could use \texttt{reduce()} to get the minimum of a list of numbers:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{numbers2 }\OtherTok{\textless{}{-}} \FunctionTok{c}\NormalTok{(}\DecValTok{3}\NormalTok{, }\DecValTok{1}\NormalTok{, }\SpecialCharTok{{-}}\DecValTok{8}\NormalTok{, }\DecValTok{9}\NormalTok{)}

\FunctionTok{reduce}\NormalTok{(numbers2, my\_min)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] -8
\end{verbatim}

\texttt{map()} and \texttt{reduce()} are arguably the most useful higher-order functions, and perhaps also the
most famous one, true ambassadors of functional programming. You might have read about
\href{https://en.wikipedia.org/wiki/MapReduce}{MapReduce}, a programming model for processing big
data in parallel. The way MapReduce works is inspired by both these \texttt{map()} and \texttt{reduce()} functions,
which are always included in functional programming languages. This illustrates that the functional
programming paradigm is very well suited to parallel computing.

Something else that is very important to understand at this point; up until now, we only used these
functions on lists, or atomic vectors, of numbers. However, \texttt{map()} and \texttt{reduce()}, and other
higher-order functions for that matter, do not care about the contents of the list. What these
functions do, is take another functions, and make it do something to the elements of the list.
It does not matter if it's a list of numbers, of characters, of data frames, even of models. All that
matters is that the function that will be applied to these elements, can operate on them.
So if you have a list of fitted models, you can map \texttt{summary()} on this list to get summaries of
each model. Or if you have a list of data frames, you can map a function that performs several
cleaning steps. This will be explored in a future section, but it is important to keep this in mind.

\hypertarget{error-handling-with-safely-and-possibly}{%
\subsubsection{\texorpdfstring{Error handling with \texttt{safely()} and \texttt{possibly()}}{Error handling with safely() and possibly()}}\label{error-handling-with-safely-and-possibly}}

\texttt{safely()} and \texttt{possibly()} are very useful functions. Consider the following situation:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\StringTok{"a"}\NormalTok{, }\DecValTok{4}\NormalTok{, }\DecValTok{5}\NormalTok{)}

\FunctionTok{sqrt}\NormalTok{(a)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{Error }\ControlFlowTok{in} \FunctionTok{sqrt}\NormalTok{(a) }\SpecialCharTok{:}\NormalTok{ non}\SpecialCharTok{{-}}\NormalTok{numeric argument to mathematical }\ControlFlowTok{function}
\end{Highlighting}
\end{Shaded}

Using \texttt{map()} or \texttt{Map()} will result in a similar error. \texttt{safely()} is an higher-order function that
takes one function as an argument and executes it\ldots{} \emph{safely}, meaning the execution of the function
will not stop if there is an error. The error message gets captured alongside valid results.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{a }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\StringTok{"a"}\NormalTok{, }\DecValTok{4}\NormalTok{, }\DecValTok{5}\NormalTok{)}

\NormalTok{safe\_sqrt }\OtherTok{\textless{}{-}} \FunctionTok{safely}\NormalTok{(sqrt)}

\FunctionTok{map}\NormalTok{(a, safe\_sqrt)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [[1]]$result
## NULL
## 
## [[1]]$error
## <simpleError in .Primitive("sqrt")(x): non-numeric argument to mathematical function>
## 
## 
## [[2]]
## [[2]]$result
## [1] 2
## 
## [[2]]$error
## NULL
## 
## 
## [[3]]
## [[3]]$result
## [1] 2.236068
## 
## [[3]]$error
## NULL
\end{verbatim}

\texttt{possibly()} works similarly, but also allows you to specify a return value in case of an error:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{possible\_sqrt }\OtherTok{\textless{}{-}} \FunctionTok{possibly}\NormalTok{(sqrt, }\AttributeTok{otherwise =} \ConstantTok{NA\_real\_}\NormalTok{)}

\FunctionTok{map}\NormalTok{(a, possible\_sqrt)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] NA
## 
## [[2]]
## [1] 2
## 
## [[3]]
## [1] 2.236068
\end{verbatim}

Of course, in this particular example, the same effect could be obtained way more easily:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sqrt}\NormalTok{(}\FunctionTok{as.numeric}\NormalTok{(a))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Warning: NAs introduced by coercion
\end{verbatim}

\begin{verbatim}
## [1]       NA 2.000000 2.236068
\end{verbatim}

However, in some situations, this trick does not work as intended (or at all). \texttt{possibly()} and
\texttt{safely()} allow the programmer to model errors explicitly, and to then provide a consistent way
of dealing with them. For instance, consider the following example:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{data}\NormalTok{(mtcars)}

\FunctionTok{write.csv}\NormalTok{(mtcars, }\StringTok{"my\_data/mtcars.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection
In addition: Warning message:
In file(file, ifelse(append, "a", "w")) :
  cannot open file 'my_data/mtcars.csv': No such file or directory
\end{verbatim}

The folder \texttt{path/to/save/} does not exist, and as such this code produces an error. You might
want to catch this error, and create the directory for instance:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{possibly\_write.csv }\OtherTok{\textless{}{-}} \FunctionTok{possibly}\NormalTok{(write.csv, }\AttributeTok{otherwise =} \ConstantTok{NULL}\NormalTok{)}

\ControlFlowTok{if}\NormalTok{(}\FunctionTok{is.null}\NormalTok{(}\FunctionTok{possibly\_write.csv}\NormalTok{(mtcars, }\StringTok{"my\_data/mtcars.csv"}\NormalTok{))) \{}
  \FunctionTok{print}\NormalTok{(}\StringTok{"Creating folder..."}\NormalTok{)}
  \FunctionTok{dir.create}\NormalTok{(}\StringTok{"my\_data/"}\NormalTok{)}
  \FunctionTok{print}\NormalTok{(}\StringTok{"Saving file..."}\NormalTok{)}
  \FunctionTok{write.csv}\NormalTok{(mtcars, }\StringTok{"my\_data/mtcars.csv"}\NormalTok{)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
[1] "Creating folder..."
[1] "Saving file..."
Warning message:
In file(file, ifelse(append, "a", "w")) :
  cannot open file 'my_data/mtcars.csv': No such file or directory
\end{verbatim}

The warning message comes from the first time we try to write the \texttt{.csv}, inside the \texttt{if}
statement. Because this fails, we create the directory and then actually save the file.
In the exercises, you'll discover \texttt{quietly()}, which also captures warnings and messages.

To conclude this section: remember function factories? Turns out that \texttt{safely()}, \texttt{purely()} and \texttt{quietly()} are
function factories.

\hypertarget{partial-applications-with-partial}{%
\subsubsection{\texorpdfstring{Partial applications with \texttt{partial()}}{Partial applications with partial()}}\label{partial-applications-with-partial}}

Consider the following simple function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{add }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(a, b) a}\SpecialCharTok{+}\NormalTok{b}
\end{Highlighting}
\end{Shaded}

It is possible to create a new function, where one of the parameters is fixed, for instance, where
\texttt{a\ =\ 10}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{add\_to\_10 }\OtherTok{\textless{}{-}} \FunctionTok{partial}\NormalTok{(add, }\AttributeTok{a =} \DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{add\_to\_10}\NormalTok{(}\DecValTok{12}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 22
\end{verbatim}

This is equivalent to the following:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{add\_to\_10\_2 }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(b)\{}
  \FunctionTok{add}\NormalTok{(}\AttributeTok{a =} \DecValTok{10}\NormalTok{, b)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Using \texttt{partial()} is much less verbose however, and allowing you to define new functions very quickly:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{head10 }\OtherTok{\textless{}{-}} \FunctionTok{partial}\NormalTok{(head, }\AttributeTok{n =} \DecValTok{10}\NormalTok{)}

\FunctionTok{head10}\NormalTok{(mtcars)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                    mpg cyl  disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4         21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag     21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710        22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive    21.4   6 258.0 110 3.08 3.215 19.44  1  0    3    1
## Hornet Sportabout 18.7   8 360.0 175 3.15 3.440 17.02  0  0    3    2
## Valiant           18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1
## Duster 360        14.3   8 360.0 245 3.21 3.570 15.84  0  0    3    4
## Merc 240D         24.4   4 146.7  62 3.69 3.190 20.00  1  0    4    2
## Merc 230          22.8   4 140.8  95 3.92 3.150 22.90  1  0    4    2
## Merc 280          19.2   6 167.6 123 3.92 3.440 18.30  1  0    4    4
\end{verbatim}

\hypertarget{function-composition-using-compose}{%
\subsubsection{\texorpdfstring{Function composition using \texttt{compose}}{Function composition using compose}}\label{function-composition-using-compose}}

Function composition is another handy tool, which makes chaining equation much more elegant:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{compose}\NormalTok{(sqrt, log10, exp)(}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 2.083973
\end{verbatim}

You can read this expression as \emph{\texttt{exp()} after \texttt{log10()} after \texttt{sqrt()}} and is equivalent to:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{sqrt}\NormalTok{(}\FunctionTok{log10}\NormalTok{(}\FunctionTok{exp}\NormalTok{(}\DecValTok{10}\NormalTok{)))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 2.083973
\end{verbatim}

It is also possible to reverse the order the functions get called using the \texttt{.dir\ =} option:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{compose}\NormalTok{(sqrt, log10, exp, }\AttributeTok{.dir =} \StringTok{"forward"}\NormalTok{)(}\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1.648721
\end{verbatim}

One could also use the \texttt{\%\textgreater{}\%} operator to achieve the same result:

\begin{Shaded}
\begin{Highlighting}[]
\DecValTok{10} \SpecialCharTok{\%\textgreater{}\%}
\NormalTok{  sqrt }\SpecialCharTok{\%\textgreater{}\%}
\NormalTok{  log10 }\SpecialCharTok{\%\textgreater{}\%}
\NormalTok{  exp}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 1.648721
\end{verbatim}

but strictly speaking, this is not function composition.

\hypertarget{transposing-lists}{%
\subsubsection{«Transposing lists»}\label{transposing-lists}}

Another interesting function is \texttt{transpose()}. It is not an alternative to the function \texttt{t()} from
\texttt{base} but, has a similar effect. \texttt{transpose()} works on lists. Let's take a look at the example
from before:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{safe\_sqrt }\OtherTok{\textless{}{-}} \FunctionTok{safely}\NormalTok{(sqrt, }\AttributeTok{otherwise =} \ConstantTok{NA\_real\_}\NormalTok{)}

\FunctionTok{map}\NormalTok{(a, safe\_sqrt)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [[1]]$result
## [1] NA
## 
## [[1]]$error
## <simpleError in .Primitive("sqrt")(x): non-numeric argument to mathematical function>
## 
## 
## [[2]]
## [[2]]$result
## [1] 2
## 
## [[2]]$error
## NULL
## 
## 
## [[3]]
## [[3]]$result
## [1] 2.236068
## 
## [[3]]$error
## NULL
\end{verbatim}

The output is a list with the first element being a list with a result and an error message. One
might want to have all the results in a single list, and all the error messages in another list.
This is possible with \texttt{transpose()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{purrr}\SpecialCharTok{::}\FunctionTok{transpose}\NormalTok{(}\FunctionTok{map}\NormalTok{(a, safe\_sqrt))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## $result
## $result[[1]]
## [1] NA
## 
## $result[[2]]
## [1] 2
## 
## $result[[3]]
## [1] 2.236068
## 
## 
## $error
## $error[[1]]
## <simpleError in .Primitive("sqrt")(x): non-numeric argument to mathematical function>
## 
## $error[[2]]
## NULL
## 
## $error[[3]]
## NULL
\end{verbatim}

I explicitely call \texttt{purrr::transpose()} because there is also a \texttt{data.table::transpose()}, which
is not the same function. You have to be careful about that sort of thing, because it can cause
errors in your programs and debuging this type of error is a nightmare.

Now that we are familiar with functional programming, let's try to apply some of its principles
to data manipulation.

\hypertarget{list-based-workflows-for-efficiency}{%
\subsection{List-based workflows for efficiency}\label{list-based-workflows-for-efficiency}}

You can use your own functions in pipe workflows:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{double\_number }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(x)\{}
\NormalTok{  x}\SpecialCharTok{+}\NormalTok{x}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mtcars }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{head}\NormalTok{() }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{double\_mpg =} \FunctionTok{double\_number}\NormalTok{(mpg))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
##                    mpg cyl disp  hp drat    wt  qsec vs am gear carb double_mpg
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4       42.0
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4       42.0
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1       45.6
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1       42.8
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2       37.4
## Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1       36.2
\end{verbatim}

It is important to understand that your functions, and functions that are built-in into R, or that
come from packages, are exactly the same thing. Every function is a first-class object in R, no
matter where they come from. The consequence of functions being first-class objects is that
functions can take functions as arguments, functions can return functions (the function factories
from the previous chapter) and can be assigned to any variable:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{plop }\OtherTok{\textless{}{-}}\NormalTok{ sqrt}

\FunctionTok{plop}\NormalTok{(}\DecValTok{4}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [1] 2
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{bacon }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(.f)\{}

  \FunctionTok{message}\NormalTok{(}\StringTok{"Bacon is tasty"}\NormalTok{)}

\NormalTok{  .f}

\NormalTok{\}}

\FunctionTok{bacon}\NormalTok{(sqrt) }\CommentTok{\# \textasciigrave{}bacon\textasciigrave{} is a function factory, as it returns a function (alongside an informative message)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Bacon is tasty
\end{verbatim}

\begin{verbatim}
## function (x)  .Primitive("sqrt")
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{\# To actually call it:}
\FunctionTok{bacon}\NormalTok{(sqrt)(}\DecValTok{4}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Bacon is tasty
\end{verbatim}

\begin{verbatim}
## [1] 2
\end{verbatim}

Now, let's step back for a bit and think about what we learned up until now, and especially
the \texttt{map()} family of functions.

Let's read the list of datasets from the previous chapter:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{paths }\OtherTok{\textless{}{-}} \FunctionTok{Sys.glob}\NormalTok{(}\StringTok{"datasets/unemployment/*.csv"}\NormalTok{)}

\NormalTok{all\_datasets }\OtherTok{\textless{}{-}} \FunctionTok{import\_list}\NormalTok{(paths)}

\FunctionTok{str}\NormalTok{(all\_datasets)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 4
##  $ unemp_2013:'data.frame':  118 obs. of  8 variables:
##   ..$ Commune                   : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ Total employed population : int [1:118] 223407 17802 1703 844 1431 4094 2146 971 1218 3002 ...
##   ..$ of which: Wage-earners    : int [1:118] 203535 15993 1535 750 1315 3800 1874 858 1029 2664 ...
##   ..$ of which: Non-wage-earners: int [1:118] 19872 1809 168 94 116 294 272 113 189 338 ...
##   ..$ Unemployed                : int [1:118] 19287 1071 114 25 74 261 98 45 66 207 ...
##   ..$ Active population         : int [1:118] 242694 18873 1817 869 1505 4355 2244 1016 1284 3209 ...
##   ..$ Unemployment rate (in %)  : num [1:118] 7.95 5.67 6.27 2.88 4.92 ...
##   ..$ Year                      : int [1:118] 2013 2013 2013 2013 2013 2013 2013 2013 2013 2013 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2013.csv"
##  $ unemp_2014:'data.frame':  118 obs. of  8 variables:
##   ..$ Commune                   : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ Total employed population : int [1:118] 228423 18166 1767 845 1505 4129 2172 1007 1268 3124 ...
##   ..$ of which: Wage-earners    : int [1:118] 208238 16366 1606 757 1390 3840 1897 887 1082 2782 ...
##   ..$ of which: Non-wage-earners: int [1:118] 20185 1800 161 88 115 289 275 120 186 342 ...
##   ..$ Unemployed                : int [1:118] 19362 1066 122 19 66 287 91 38 61 202 ...
##   ..$ Active population         : int [1:118] 247785 19232 1889 864 1571 4416 2263 1045 1329 3326 ...
##   ..$ Unemployment rate (in %)  : num [1:118] 7.81 5.54 6.46 2.2 4.2 ...
##   ..$ Year                      : int [1:118] 2014 2014 2014 2014 2014 2014 2014 2014 2014 2014 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2014.csv"
##  $ unemp_2015:'data.frame':  118 obs. of  8 variables:
##   ..$ Commune                   : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ Total employed population : int [1:118] 233130 18310 1780 870 1470 4130 2170 1050 1300 3140 ...
##   ..$ of which: Wage-earners    : int [1:118] 212530 16430 1620 780 1350 3820 1910 920 1100 2770 ...
##   ..$ of which: Non-wage-earners: int [1:118] 20600 1880 160 90 120 310 260 130 200 370 ...
##   ..$ Unemployed                : int [1:118] 18806 988 106 29 73 260 80 41 72 169 ...
##   ..$ Active population         : int [1:118] 251936 19298 1886 899 1543 4390 2250 1091 1372 3309 ...
##   ..$ Unemployment rate (in %)  : num [1:118] 7.46 5.12 5.62 3.23 4.73 ...
##   ..$ Year                      : int [1:118] 2015 2015 2015 2015 2015 2015 2015 2015 2015 2015 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2015.csv"
##  $ unemp_2016:'data.frame':  118 obs. of  8 variables:
##   ..$ Commune                   : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ Total employed population : int [1:118] 236100 18380 1790 870 1470 4160 2160 1030 1330 3150 ...
##   ..$ of which: Wage-earners    : int [1:118] 215430 16500 1640 780 1350 3840 1900 900 1130 2780 ...
##   ..$ of which: Non-wage-earners: int [1:118] 20670 1880 150 90 120 320 260 130 200 370 ...
##   ..$ Unemployed                : int [1:118] 18185 975 91 27 66 246 76 35 70 206 ...
##   ..$ Active population         : int [1:118] 254285 19355 1881 897 1536 4406 2236 1065 1400 3356 ...
##   ..$ Unemployment rate (in %)  : num [1:118] 7.15 5.04 4.84 3.01 4.3 ...
##   ..$ Year                      : int [1:118] 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2016.csv"
\end{verbatim}

\texttt{all\_datasets} is a list with 4 elements, each of them is a \texttt{data.frame}.

The first thing we are going to do is use a function to clean the names of the datasets. These
names are not very easy to work with; there are spaces, and it would be better if the names of the
columns would be all lowercase. For this we are going to use the function \texttt{clean\_names()} from the
\texttt{janitor} package. For a single dataset, I would write this:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(janitor)}

\NormalTok{one\_dataset }\OtherTok{\textless{}{-}}\NormalTok{ one\_dataset }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{clean\_names}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

and I would get a dataset with column names in lowercase and spaces replaced by \texttt{\_} (and other
corrections). How can I apply, or map, this function to each dataset in the list? To do this I need
to use \texttt{purrr::map()}, which we've seen in the previous section:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(purrr)}

\NormalTok{all\_datasets }\OtherTok{\textless{}{-}}\NormalTok{ all\_datasets }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{map}\NormalTok{(clean\_names)}

\NormalTok{all\_datasets }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{glimpse}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 4
##  $ unemp_2013:'data.frame':  118 obs. of  8 variables:
##   ..$ commune                     : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ total_employed_population   : int [1:118] 223407 17802 1703 844 1431 4094 2146 971 1218 3002 ...
##   ..$ of_which_wage_earners       : int [1:118] 203535 15993 1535 750 1315 3800 1874 858 1029 2664 ...
##   ..$ of_which_non_wage_earners   : int [1:118] 19872 1809 168 94 116 294 272 113 189 338 ...
##   ..$ unemployed                  : int [1:118] 19287 1071 114 25 74 261 98 45 66 207 ...
##   ..$ active_population           : int [1:118] 242694 18873 1817 869 1505 4355 2244 1016 1284 3209 ...
##   ..$ unemployment_rate_in_percent: num [1:118] 7.95 5.67 6.27 2.88 4.92 ...
##   ..$ year                        : int [1:118] 2013 2013 2013 2013 2013 2013 2013 2013 2013 2013 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2013.csv"
##  $ unemp_2014:'data.frame':  118 obs. of  8 variables:
##   ..$ commune                     : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ total_employed_population   : int [1:118] 228423 18166 1767 845 1505 4129 2172 1007 1268 3124 ...
##   ..$ of_which_wage_earners       : int [1:118] 208238 16366 1606 757 1390 3840 1897 887 1082 2782 ...
##   ..$ of_which_non_wage_earners   : int [1:118] 20185 1800 161 88 115 289 275 120 186 342 ...
##   ..$ unemployed                  : int [1:118] 19362 1066 122 19 66 287 91 38 61 202 ...
##   ..$ active_population           : int [1:118] 247785 19232 1889 864 1571 4416 2263 1045 1329 3326 ...
##   ..$ unemployment_rate_in_percent: num [1:118] 7.81 5.54 6.46 2.2 4.2 ...
##   ..$ year                        : int [1:118] 2014 2014 2014 2014 2014 2014 2014 2014 2014 2014 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2014.csv"
##  $ unemp_2015:'data.frame':  118 obs. of  8 variables:
##   ..$ commune                     : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ total_employed_population   : int [1:118] 233130 18310 1780 870 1470 4130 2170 1050 1300 3140 ...
##   ..$ of_which_wage_earners       : int [1:118] 212530 16430 1620 780 1350 3820 1910 920 1100 2770 ...
##   ..$ of_which_non_wage_earners   : int [1:118] 20600 1880 160 90 120 310 260 130 200 370 ...
##   ..$ unemployed                  : int [1:118] 18806 988 106 29 73 260 80 41 72 169 ...
##   ..$ active_population           : int [1:118] 251936 19298 1886 899 1543 4390 2250 1091 1372 3309 ...
##   ..$ unemployment_rate_in_percent: num [1:118] 7.46 5.12 5.62 3.23 4.73 ...
##   ..$ year                        : int [1:118] 2015 2015 2015 2015 2015 2015 2015 2015 2015 2015 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2015.csv"
##  $ unemp_2016:'data.frame':  118 obs. of  8 variables:
##   ..$ commune                     : chr [1:118] "Grand-Duche de Luxembourg" "Canton Capellen" "Dippach" "Garnich" ...
##   ..$ total_employed_population   : int [1:118] 236100 18380 1790 870 1470 4160 2160 1030 1330 3150 ...
##   ..$ of_which_wage_earners       : int [1:118] 215430 16500 1640 780 1350 3840 1900 900 1130 2780 ...
##   ..$ of_which_non_wage_earners   : int [1:118] 20670 1880 150 90 120 320 260 130 200 370 ...
##   ..$ unemployed                  : int [1:118] 18185 975 91 27 66 246 76 35 70 206 ...
##   ..$ active_population           : int [1:118] 254285 19355 1881 897 1536 4406 2236 1065 1400 3356 ...
##   ..$ unemployment_rate_in_percent: num [1:118] 7.15 5.04 4.84 3.01 4.3 ...
##   ..$ year                        : int [1:118] 2016 2016 2016 2016 2016 2016 2016 2016 2016 2016 ...
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2016.csv"
\end{verbatim}

Remember that \texttt{map(list,\ function)} simply evaluates \texttt{function} to each element of \texttt{list}.

So now, what if I want to know, for each dataset, which \emph{communes} have an unemployment rate that is
less than, say, 3\%? For a single dataset I would do something like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{one\_dataset }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(unemployment\_rate\_in\_percent }\SpecialCharTok{\textless{}} \DecValTok{3}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

but since we're dealing with a list of data sets, we cannot simply use \texttt{filter()} on it. This is because
\texttt{filter()} expects a data frame, not a list of data frames. The way around this is to use \texttt{map()}.

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{all\_datasets }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{map}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\FunctionTok{filter}\NormalTok{(., unemployment\_rate\_in\_percent }\SpecialCharTok{\textless{}} \DecValTok{3}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## $unemp_2013
##      commune total_employed_population of_which_wage_earners
## 1    Garnich                       844                   750
## 2 Leudelange                      1064                   937
## 3       Bech                       526                   463
##   of_which_non_wage_earners unemployed active_population
## 1                        94         25               869
## 2                       127         32              1096
## 3                        63         16               542
##   unemployment_rate_in_percent year
## 1                     2.876870 2013
## 2                     2.919708 2013
## 3                     2.952030 2013
## 
## $unemp_2014
##      commune total_employed_population of_which_wage_earners
## 1    Garnich                       845                   757
## 2 Leudelange                      1102                   965
## 3       Bech                       543                   476
## 4 Flaxweiler                       879                   789
##   of_which_non_wage_earners unemployed active_population
## 1                        88         19               864
## 2                       137         34              1136
## 3                        67         15               558
## 4                        90         27               906
##   unemployment_rate_in_percent year
## 1                     2.199074 2014
## 2                     2.992958 2014
## 3                     2.688172 2014
## 4                     2.980132 2014
## 
## $unemp_2015
##   commune total_employed_population of_which_wage_earners
## 1    Bech                       520                   450
## 2    Bous                       750                   680
##   of_which_non_wage_earners unemployed active_population
## 1                        70         14               534
## 2                        70         22               772
##   unemployment_rate_in_percent year
## 1                     2.621723 2015
## 2                     2.849741 2015
## 
## $unemp_2016
##             commune total_employed_population of_which_wage_earners
## 1 Reckange-sur-Mess                       980                   850
## 2              Bech                       520                   450
## 3          Betzdorf                      1500                  1350
## 4        Flaxweiler                       910                   820
##   of_which_non_wage_earners unemployed active_population
## 1                       130         30              1010
## 2                        70         11               531
## 3                       150         45              1545
## 4                        90         24               934
##   unemployment_rate_in_percent year
## 1                     2.970297 2016
## 2                     2.071563 2016
## 3                     2.912621 2016
## 4                     2.569593 2016
\end{verbatim}

\texttt{map()} needs a function to map to each element of the list. \texttt{all\_datasets} is the list to which I
want to map the function. But what function? \texttt{filter()} is the function I need, so why doesn't:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{all\_datasets }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{map}\NormalTok{(}\FunctionTok{filter}\NormalTok{(unemployment\_rate\_in\_percent }\SpecialCharTok{\textless{}} \DecValTok{3}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

work? This is what happens if we try it:

\begin{verbatim}
Error in filter(unemployment_rate_in_percent < 3) :
  object 'unemployment_rate_in_percent' not found
\end{verbatim}

This is because \texttt{filter()} needs both the data set, and a so-called predicate (a predicate
is an expression that evaluates to \texttt{TRUE} or \texttt{FALSE}). But you need to make more explicit
what is the dataset and what is the predicate, because here, \texttt{filter()} thinks that the
dataset is \texttt{unemployment\_rate\_in\_percent}. The way to do this is to use an anonymous
function (discussed in Chapter 7), which allows you to explicitely state what is the
dataset, and what is the predicate. As we've seen, there's three ways to define
anonymous functions:

\begin{itemize}
\tightlist
\item
  Using a formula (only works within \texttt{\{tidyverse\}} functions):
\end{itemize}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{all\_datasets }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{map}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\FunctionTok{filter}\NormalTok{(., unemployment\_rate\_in\_percent }\SpecialCharTok{\textless{}} \DecValTok{3}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{glimpse}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 4
##  $ unemp_2013:'data.frame':  3 obs. of  8 variables:
##   ..$ commune                     : chr [1:3] "Garnich" "Leudelange" "Bech"
##   ..$ total_employed_population   : int [1:3] 844 1064 526
##   ..$ of_which_wage_earners       : int [1:3] 750 937 463
##   ..$ of_which_non_wage_earners   : int [1:3] 94 127 63
##   ..$ unemployed                  : int [1:3] 25 32 16
##   ..$ active_population           : int [1:3] 869 1096 542
##   ..$ unemployment_rate_in_percent: num [1:3] 2.88 2.92 2.95
##   ..$ year                        : int [1:3] 2013 2013 2013
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2013.csv"
##  $ unemp_2014:'data.frame':  4 obs. of  8 variables:
##   ..$ commune                     : chr [1:4] "Garnich" "Leudelange" "Bech" "Flaxweiler"
##   ..$ total_employed_population   : int [1:4] 845 1102 543 879
##   ..$ of_which_wage_earners       : int [1:4] 757 965 476 789
##   ..$ of_which_non_wage_earners   : int [1:4] 88 137 67 90
##   ..$ unemployed                  : int [1:4] 19 34 15 27
##   ..$ active_population           : int [1:4] 864 1136 558 906
##   ..$ unemployment_rate_in_percent: num [1:4] 2.2 2.99 2.69 2.98
##   ..$ year                        : int [1:4] 2014 2014 2014 2014
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2014.csv"
##  $ unemp_2015:'data.frame':  2 obs. of  8 variables:
##   ..$ commune                     : chr [1:2] "Bech" "Bous"
##   ..$ total_employed_population   : int [1:2] 520 750
##   ..$ of_which_wage_earners       : int [1:2] 450 680
##   ..$ of_which_non_wage_earners   : int [1:2] 70 70
##   ..$ unemployed                  : int [1:2] 14 22
##   ..$ active_population           : int [1:2] 534 772
##   ..$ unemployment_rate_in_percent: num [1:2] 2.62 2.85
##   ..$ year                        : int [1:2] 2015 2015
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2015.csv"
##  $ unemp_2016:'data.frame':  4 obs. of  8 variables:
##   ..$ commune                     : chr [1:4] "Reckange-sur-Mess" "Bech" "Betzdorf" "Flaxweiler"
##   ..$ total_employed_population   : int [1:4] 980 520 1500 910
##   ..$ of_which_wage_earners       : int [1:4] 850 450 1350 820
##   ..$ of_which_non_wage_earners   : int [1:4] 130 70 150 90
##   ..$ unemployed                  : int [1:4] 30 11 45 24
##   ..$ active_population           : int [1:4] 1010 531 1545 934
##   ..$ unemployment_rate_in_percent: num [1:4] 2.97 2.07 2.91 2.57
##   ..$ year                        : int [1:4] 2016 2016 2016 2016
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2016.csv"
\end{verbatim}

(notice the \texttt{.} in the formula, making the position of the dataset as the first argument to \texttt{filter()}
explicit) or

\begin{itemize}
\tightlist
\item
  using an anonymous function (using the \texttt{function(x)} keyword):
\end{itemize}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{all\_datasets }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{map}\NormalTok{(}\ControlFlowTok{function}\NormalTok{(x)}\FunctionTok{filter}\NormalTok{(x, unemployment\_rate\_in\_percent }\SpecialCharTok{\textless{}} \DecValTok{3}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{glimpse}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 4
##  $ unemp_2013:'data.frame':  3 obs. of  8 variables:
##   ..$ commune                     : chr [1:3] "Garnich" "Leudelange" "Bech"
##   ..$ total_employed_population   : int [1:3] 844 1064 526
##   ..$ of_which_wage_earners       : int [1:3] 750 937 463
##   ..$ of_which_non_wage_earners   : int [1:3] 94 127 63
##   ..$ unemployed                  : int [1:3] 25 32 16
##   ..$ active_population           : int [1:3] 869 1096 542
##   ..$ unemployment_rate_in_percent: num [1:3] 2.88 2.92 2.95
##   ..$ year                        : int [1:3] 2013 2013 2013
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2013.csv"
##  $ unemp_2014:'data.frame':  4 obs. of  8 variables:
##   ..$ commune                     : chr [1:4] "Garnich" "Leudelange" "Bech" "Flaxweiler"
##   ..$ total_employed_population   : int [1:4] 845 1102 543 879
##   ..$ of_which_wage_earners       : int [1:4] 757 965 476 789
##   ..$ of_which_non_wage_earners   : int [1:4] 88 137 67 90
##   ..$ unemployed                  : int [1:4] 19 34 15 27
##   ..$ active_population           : int [1:4] 864 1136 558 906
##   ..$ unemployment_rate_in_percent: num [1:4] 2.2 2.99 2.69 2.98
##   ..$ year                        : int [1:4] 2014 2014 2014 2014
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2014.csv"
##  $ unemp_2015:'data.frame':  2 obs. of  8 variables:
##   ..$ commune                     : chr [1:2] "Bech" "Bous"
##   ..$ total_employed_population   : int [1:2] 520 750
##   ..$ of_which_wage_earners       : int [1:2] 450 680
##   ..$ of_which_non_wage_earners   : int [1:2] 70 70
##   ..$ unemployed                  : int [1:2] 14 22
##   ..$ active_population           : int [1:2] 534 772
##   ..$ unemployment_rate_in_percent: num [1:2] 2.62 2.85
##   ..$ year                        : int [1:2] 2015 2015
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2015.csv"
##  $ unemp_2016:'data.frame':  4 obs. of  8 variables:
##   ..$ commune                     : chr [1:4] "Reckange-sur-Mess" "Bech" "Betzdorf" "Flaxweiler"
##   ..$ total_employed_population   : int [1:4] 980 520 1500 910
##   ..$ of_which_wage_earners       : int [1:4] 850 450 1350 820
##   ..$ of_which_non_wage_earners   : int [1:4] 130 70 150 90
##   ..$ unemployed                  : int [1:4] 30 11 45 24
##   ..$ active_population           : int [1:4] 1010 531 1545 934
##   ..$ unemployment_rate_in_percent: num [1:4] 2.97 2.07 2.91 2.57
##   ..$ year                        : int [1:4] 2016 2016 2016 2016
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2016.csv"
\end{verbatim}

\begin{itemize}
\tightlist
\item
  or, since R 4.1, using the shorthand \texttt{\textbackslash{}(x)}:
\end{itemize}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{all\_datasets }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{map}\NormalTok{(\textbackslash{}(x)}\FunctionTok{filter}\NormalTok{(x, unemployment\_rate\_in\_percent }\SpecialCharTok{\textless{}} \DecValTok{3}\NormalTok{)) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{glimpse}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 4
##  $ unemp_2013:'data.frame':  3 obs. of  8 variables:
##   ..$ commune                     : chr [1:3] "Garnich" "Leudelange" "Bech"
##   ..$ total_employed_population   : int [1:3] 844 1064 526
##   ..$ of_which_wage_earners       : int [1:3] 750 937 463
##   ..$ of_which_non_wage_earners   : int [1:3] 94 127 63
##   ..$ unemployed                  : int [1:3] 25 32 16
##   ..$ active_population           : int [1:3] 869 1096 542
##   ..$ unemployment_rate_in_percent: num [1:3] 2.88 2.92 2.95
##   ..$ year                        : int [1:3] 2013 2013 2013
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2013.csv"
##  $ unemp_2014:'data.frame':  4 obs. of  8 variables:
##   ..$ commune                     : chr [1:4] "Garnich" "Leudelange" "Bech" "Flaxweiler"
##   ..$ total_employed_population   : int [1:4] 845 1102 543 879
##   ..$ of_which_wage_earners       : int [1:4] 757 965 476 789
##   ..$ of_which_non_wage_earners   : int [1:4] 88 137 67 90
##   ..$ unemployed                  : int [1:4] 19 34 15 27
##   ..$ active_population           : int [1:4] 864 1136 558 906
##   ..$ unemployment_rate_in_percent: num [1:4] 2.2 2.99 2.69 2.98
##   ..$ year                        : int [1:4] 2014 2014 2014 2014
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2014.csv"
##  $ unemp_2015:'data.frame':  2 obs. of  8 variables:
##   ..$ commune                     : chr [1:2] "Bech" "Bous"
##   ..$ total_employed_population   : int [1:2] 520 750
##   ..$ of_which_wage_earners       : int [1:2] 450 680
##   ..$ of_which_non_wage_earners   : int [1:2] 70 70
##   ..$ unemployed                  : int [1:2] 14 22
##   ..$ active_population           : int [1:2] 534 772
##   ..$ unemployment_rate_in_percent: num [1:2] 2.62 2.85
##   ..$ year                        : int [1:2] 2015 2015
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2015.csv"
##  $ unemp_2016:'data.frame':  4 obs. of  8 variables:
##   ..$ commune                     : chr [1:4] "Reckange-sur-Mess" "Bech" "Betzdorf" "Flaxweiler"
##   ..$ total_employed_population   : int [1:4] 980 520 1500 910
##   ..$ of_which_wage_earners       : int [1:4] 850 450 1350 820
##   ..$ of_which_non_wage_earners   : int [1:4] 130 70 150 90
##   ..$ unemployed                  : int [1:4] 30 11 45 24
##   ..$ active_population           : int [1:4] 1010 531 1545 934
##   ..$ unemployment_rate_in_percent: num [1:4] 2.97 2.07 2.91 2.57
##   ..$ year                        : int [1:4] 2016 2016 2016 2016
##   ..- attr(*, "filename")= chr "datasets/unemployment/unemp_2016.csv"
\end{verbatim}

As you see, everything is starting to come together: lists, to hold complex objects, over which anonymous
functions are mapped using higher-order functions. Let's continue cleaning this dataset.

Before merging these datasets together, we would need them to have a \texttt{year} column indicating the
year the data was measured in each data frame. It would also be helpful if gave names to these datasets, meaning
converting the list to a named list. For this task, we can use \texttt{purrr::set\_names()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{all\_datasets }\OtherTok{\textless{}{-}} \FunctionTok{set\_names}\NormalTok{(all\_datasets, }\FunctionTok{as.character}\NormalTok{(}\FunctionTok{seq}\NormalTok{(}\DecValTok{2013}\NormalTok{, }\DecValTok{2016}\NormalTok{)))}
\end{Highlighting}
\end{Shaded}

Let's take a look at the list now:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{str}\NormalTok{(all\_datasets)}
\end{Highlighting}
\end{Shaded}

As you can see, each \texttt{data.frame} object contained in the list has been renamed. You can thus
access them with the \texttt{\$} operator:

\includegraphics[width=3.75in]{pics/all_datasets_names}

Using \texttt{map()} we now know how to apply a function to each dataset of a list. But maybe it would be
easier to merge all the datasets first, and then manipulate them? This can be the case sometimes,
but not always.
As long as you provide a function and a list of elements to \texttt{reduce()}, you will get a single
output. So how could \texttt{reduce()} help us with merging all the datasets that are in the list? \texttt{dplyr}
comes with a lot of function to merge \emph{two} datasets. Remember that I said before that \texttt{reduce()}
allows you to generalize a function of two arguments? Let's try it with our list of datasets:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux }\OtherTok{\textless{}{-}} \FunctionTok{reduce}\NormalTok{(all\_datasets, full\_join)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Joining, by = c("commune", "total_employed_population", "of_which_wage_earners", "of_which_non_wage_earners", "unemployed",
## "active_population", "unemployment_rate_in_percent", "year")
## Joining, by = c("commune", "total_employed_population", "of_which_wage_earners", "of_which_non_wage_earners", "unemployed",
## "active_population", "unemployment_rate_in_percent", "year")
## Joining, by = c("commune", "total_employed_population", "of_which_wage_earners", "of_which_non_wage_earners", "unemployed",
## "active_population", "unemployment_rate_in_percent", "year")
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{glimpse}\NormalTok{(unemp\_lux)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Rows: 472
## Columns: 8
## $ commune                      <chr> "Grand-Duche de Luxembourg", "Canton Cape~
## $ total_employed_population    <int> 223407, 17802, 1703, 844, 1431, 4094, 214~
## $ of_which_wage_earners        <int> 203535, 15993, 1535, 750, 1315, 3800, 187~
## $ of_which_non_wage_earners    <int> 19872, 1809, 168, 94, 116, 294, 272, 113,~
## $ unemployed                   <int> 19287, 1071, 114, 25, 74, 261, 98, 45, 66~
## $ active_population            <int> 242694, 18873, 1817, 869, 1505, 4355, 224~
## $ unemployment_rate_in_percent <dbl> 7.947044, 5.674773, 6.274078, 2.876870, 4~
## $ year                         <int> 2013, 2013, 2013, 2013, 2013, 2013, 2013,~
\end{verbatim}

\texttt{full\_join()} is one of the \texttt{dplyr} function that merges data. There are others that might be
useful depending on the kind of join operation you need. Let's write this data to disk as we're
going to keep using it for the next chapters:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{export}\NormalTok{(unemp\_lux, }\StringTok{"datasets/unemp\_lux.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\hypertarget{functional-programming-and-plotting}{%
\subsubsection{Functional programming and plotting}\label{functional-programming-and-plotting}}

In this section, we are going to learn how to use the possibilities offered by the \texttt{purrr} package
and how it can work together with \texttt{ggplot2} to generate many plots. This is a more advanced topic,
but what comes next is also what makes R, and the functional programming paradigm so powerful.

For example, suppose that instead of wanting a single plot with the unemployment rate of each
commune, you need one unemployment plot, per commune:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{==} \StringTok{"Luxembourg"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme\_minimal}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Luxembourg"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-731-1.pdf}

and then you would write the same for ``Esch-sur-Alzette'' and also for ``Wiltz''. If you only have to
make to make these 3 plots, copy and pasting the above lines is no big deal:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{==} \StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme\_minimal}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Esch{-}sur{-}Alzette"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-732-1.pdf}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{==} \StringTok{"Wiltz"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{ggplot}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =}\NormalTok{ division)) }\SpecialCharTok{+}
  \FunctionTok{theme\_minimal}\NormalTok{() }\SpecialCharTok{+}
  \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \StringTok{"Unemployment in Esch{-}sur{-}Alzette"}\NormalTok{, }\AttributeTok{x =} \StringTok{"Year"}\NormalTok{, }\AttributeTok{y =} \StringTok{"Rate"}\NormalTok{) }\SpecialCharTok{+}
  \FunctionTok{geom\_line}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-732-2.pdf}

But copy and pasting is error prone. Can you spot the copy-paste mistake I made? And what if you
have to create the above plots for all 108 Luxembourguish communes? That's a lot of copy pasting.
What if, once you are done copy pasting, you have to change something, for example, the theme? You
could use the search and replace function of RStudio, true, but sometimes search and replace can
also introduce bugs and typos. You can avoid all these issues by using \texttt{purrr::map()}. What do you
need to map over? The commune names. So let's create a vector of commune names:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{communes }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\StringTok{"Luxembourg"}\NormalTok{, }\StringTok{"Esch{-}sur{-}Alzette"}\NormalTok{, }\StringTok{"Wiltz"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Now we can create the graphs using \texttt{map()}, or \texttt{map2()} to be exact:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{plots\_tibble }\OtherTok{\textless{}{-}}\NormalTok{ unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%}\NormalTok{ communes) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(division) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{nest}\NormalTok{() }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{plot =} \FunctionTok{map2}\NormalTok{(}\AttributeTok{.x =}\NormalTok{ data, }\AttributeTok{.y =}\NormalTok{ division, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{ggplot}\NormalTok{(}\AttributeTok{data =}\NormalTok{ .x) }\SpecialCharTok{+}
       \FunctionTok{theme\_minimal}\NormalTok{() }\SpecialCharTok{+}
       \FunctionTok{geom\_line}\NormalTok{(}\FunctionTok{aes}\NormalTok{(year, unemployment\_rate\_in\_percent, }\AttributeTok{group =} \DecValTok{1}\NormalTok{)) }\SpecialCharTok{+}
       \FunctionTok{labs}\NormalTok{(}\AttributeTok{title =} \FunctionTok{paste}\NormalTok{(}\StringTok{"Unemployment in"}\NormalTok{, .y))))}
\end{Highlighting}
\end{Shaded}

Let's study this line by line: the first line is easy, we simply use \texttt{filter()} to keep only the
communes we are interested in. Then we group by \texttt{division} and use \texttt{tidyr::nest()}. As a refresher,
let's take a look at what this does:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{unemp\_lux\_data }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{filter}\NormalTok{(division }\SpecialCharTok{\%in\%}\NormalTok{ communes) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{group\_by}\NormalTok{(division) }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{nest}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 3 x 2
## # Groups:   division [3]
##   division         data             
##   <chr>            <list>           
## 1 Esch-sur-Alzette <tibble [15 x 7]>
## 2 Luxembourg       <tibble [15 x 7]>
## 3 Wiltz            <tibble [15 x 7]>
\end{verbatim}

This creates a tibble with two columns, \texttt{division} and \texttt{data}, where each individual (or
commune in this case) is another tibble with all the original variables. This is very useful,
because now we can pass these tibbles to \texttt{map2()}, to generate the plots. But why \texttt{map2()} and
what's the difference with \texttt{map()}? \texttt{map2()} works the same way as \texttt{map()}, but maps over two
inputs:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{numbers1 }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\DecValTok{1}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{4}\NormalTok{, }\DecValTok{5}\NormalTok{)}

\NormalTok{numbers2 }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\DecValTok{9}\NormalTok{, }\DecValTok{8}\NormalTok{, }\DecValTok{7}\NormalTok{, }\DecValTok{6}\NormalTok{, }\DecValTok{5}\NormalTok{)}

\FunctionTok{map2}\NormalTok{(numbers1, numbers2, }\StringTok{\textasciigrave{}}\AttributeTok{*}\StringTok{\textasciigrave{}}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## [1] 9
## 
## [[2]]
## [1] 16
## 
## [[3]]
## [1] 21
## 
## [[4]]
## [1] 24
## 
## [[5]]
## [1] 25
\end{verbatim}

In our example with the graphs, the two inputs are the data, and the names of the communes. This is
useful to create the title with \texttt{labs(title\ =\ paste("Unemployment\ in",\ .y))))} where \texttt{.y} is the
second input of \texttt{map2()}, the commune names contained in variable \texttt{division}.

So what happened? We now have a tibble called \texttt{plots\_tibble} that looks like this:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{print}\NormalTok{(plots\_tibble)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 3 x 3
## # Groups:   division [3]
##   division         data              plot  
##   <chr>            <list>            <list>
## 1 Esch-sur-Alzette <tibble [15 x 7]> <gg>  
## 2 Luxembourg       <tibble [15 x 7]> <gg>  
## 3 Wiltz            <tibble [15 x 7]> <gg>
\end{verbatim}

This tibble contains three columns, \texttt{division}, \texttt{data} and now a new one called \texttt{plot}, that we
created before using the last line \texttt{mutate(plot\ =\ ...)} (remember that \texttt{mutate()} adds columns to
tibbles). \texttt{plot} is a list-column, with elements\ldots{} being plots! Yes you read that right, the
elements of the column \texttt{plot} are literally plots. This is what I meant with list columns.
Let's see what is inside the \texttt{data} and the \texttt{plot} columns exactly:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{plots\_tibble }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{pull}\NormalTok{(data)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## # A tibble: 15 x 7
##     year active_population of_which_non_wage_e~1 of_wh~2 total~3 unemp~4 unemp~5
##    <int>             <dbl>                 <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
##  1  2001              11.3                   665    10.1    10.8  561       4.95
##  2  2002              11.7                   677    10.3    11.0  696       5.96
##  3  2003              11.7                   674    10.2    10.9  813       6.94
##  4  2004              12.2                   659    10.6    11.3  899       7.38
##  5  2005              11.9                   654    10.3    11.0  952       7.97
##  6  2006              12.2                   657    10.5    11.2    1.07    8.71
##  7  2007              12.6                   634    10.9    11.5    1.03    8.21
##  8  2008              12.9                   638    11.0    11.6    1.28    9.92
##  9  2009              13.2                   652    11.0    11.7    1.58   11.9 
## 10  2010              13.6                   638    11.2    11.8    1.73   12.8 
## 11  2011              13.9                   630    11.5    12.1    1.77   12.8 
## 12  2012              14.3                   684    11.8    12.5    1.83   12.8 
## 13  2013              14.8                   694    12.0    12.7    2.05   13.9 
## 14  2014              15.2                   703    12.5    13.2    2.00   13.2 
## 15  2015              15.3                   710    12.6    13.3    2.03   13.2 
## # ... with abbreviated variable names 1: of_which_non_wage_earners,
## #   2: of_which_wage_earners, 3: total_employed_population, 4: unemployed,
## #   5: unemployment_rate_in_percent
## 
## [[2]]
## # A tibble: 15 x 7
##     year active_population of_which_non_wage_e~1 of_wh~2 total~3 unemp~4 unemp~5
##    <int>             <dbl>                 <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
##  1  2001              34.4                  2.89    30.4    33.2    1.14    3.32
##  2  2002              34.8                  2.94    30.3    33.2    1.56    4.5 
##  3  2003              35.2                  3.03    30.1    33.2    2.04    5.78
##  4  2004              35.6                  3.06    30.1    33.2    2.39    6.73
##  5  2005              35.6                  3.13    29.8    33.0    2.64    7.42
##  6  2006              35.5                  3.12    30.3    33.4    2.03    5.72
##  7  2007              36.1                  3.25    31.1    34.4    1.76    4.87
##  8  2008              37.5                  3.39    31.9    35.3    2.23    5.95
##  9  2009              37.9                  3.49    31.6    35.1    2.85    7.51
## 10  2010              38.6                  3.54    32.1    35.7    2.96    7.66
## 11  2011              40.3                  3.66    33.6    37.2    3.11    7.72
## 12  2012              41.8                  3.81    34.6    38.4    3.37    8.07
## 13  2013              43.4                  3.98    35.5    39.5    3.86    8.89
## 14  2014              44.6                  4.11    36.7    40.8    3.84    8.6 
## 15  2015              45.2                  4.14    37.5    41.6    3.57    7.9 
## # ... with abbreviated variable names 1: of_which_non_wage_earners,
## #   2: of_which_wage_earners, 3: total_employed_population, 4: unemployed,
## #   5: unemployment_rate_in_percent
## 
## [[3]]
## # A tibble: 15 x 7
##     year active_population of_which_non_wage_e~1 of_wh~2 total~3 unemp~4 unemp~5
##    <int>             <dbl>                 <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
##  1  2001              2.13                   223    1.79    2.01     122    5.73
##  2  2002              2.14                   220    1.78    2.00     134    6.27
##  3  2003              2.18                   223    1.79    2.02     163    7.48
##  4  2004              2.24                   227    1.85    2.08     156    6.97
##  5  2005              2.26                   229    1.85    2.08     187    8.26
##  6  2006              2.20                   206    1.82    2.02     181    8.22
##  7  2007              2.27                   198    1.88    2.08     197    8.67
##  8  2008              2.30                   200    1.90    2.10     201    8.75
##  9  2009              2.36                   201    1.94    2.15     216    9.14
## 10  2010              2.42                   195    1.97    2.17     256   10.6 
## 11  2011              2.48                   190    2.02    2.21     269   10.9 
## 12  2012              2.59                   188    2.10    2.29     301   11.6 
## 13  2013              2.66                   195    2.15    2.34     318   12.0 
## 14  2014              2.69                   185    2.19    2.38     315   11.7 
## 15  2015              2.77                   180    2.27    2.45     321   11.6 
## # ... with abbreviated variable names 1: of_which_non_wage_earners,
## #   2: of_which_wage_earners, 3: total_employed_population, 4: unemployed,
## #   5: unemployment_rate_in_percent
\end{verbatim}

each element of data is a tibble for the specific country with columns \texttt{year}, \texttt{active\_population},
etc, the original columns. But obviously, there is no \texttt{division} column. So to plot the data, and
join all the dots together, we need to add \texttt{group\ =\ 1} in the call to \texttt{ggplot2()} (whereas if you
plot multiple lines in the same graph, you need to write \texttt{group\ =\ division}).

But more interestingly, how can you actually see the plots? If you want to simply look at them, it
is enough to use \texttt{pull()}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{plots\_tibble }\SpecialCharTok{\%\textgreater{}\%}
  \FunctionTok{pull}\NormalTok{(plot)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-739-1.pdf}

\begin{verbatim}
## 
## [[2]]
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-739-2.pdf}

\begin{verbatim}
## 
## [[3]]
\end{verbatim}

\includegraphics{modern_R_files/figure-latex/unnamed-chunk-739-3.pdf}

And if we want to save these plots, we can do so using \texttt{map2()}:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{map2}\NormalTok{(}\FunctionTok{paste0}\NormalTok{(plots\_tibble}\SpecialCharTok{$}\NormalTok{division, }\StringTok{".pdf"}\NormalTok{), plots\_tibble}\SpecialCharTok{$}\NormalTok{plot, ggsave)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
Saving 7 x 5 in image
Saving 6.01 x 3.94 in image
Saving 6.01 x 3.94 in image
\end{verbatim}

This was probably the most advanced topic we have studied yet; but you probably agree with me that
it is among the most useful ones. This section is a perfect illustration of the power of functional
programming; you can mix and match functions as long as you give them the correct arguments.
You can pass data to functions that use data and then pass these functions to other functions that
use functions as arguments, such as \texttt{map()}.\footnote{Functions that have other functions as input are
  called \emph{higher order functions}} \texttt{map()} does not care if the functions you pass to it produces tables,
graphs or even another function. \texttt{map()} will simply map this function to a list of inputs, and as
long as these inputs are correct arguments to the function, \texttt{map()} will do its magic. If you
combine this with list-columns, you can even use \texttt{map()} alongside \texttt{dplyr} functions and map your
function by first grouping, filtering, etc\ldots{}

\hypertarget{modeling-with-functional-programming}{%
\subsubsection{Modeling with functional programming}\label{modeling-with-functional-programming}}

As written just above, \texttt{map()} simply applies a function to a list of inputs, and in the previous
section we mapped \texttt{ggplot()} to generate many plots at once. This approach can also be used to
map any modeling functions, for instance \texttt{lm()} to a list of datasets.

For instance, suppose that you wish to perform a Monte Carlo simulation. Suppose that you are
dealing with a binary choice problem; usually, you would use a logistic regression for this.

However, in certain disciplines, especially in the social sciences, the so-called Linear Probability
Model is often used as well. The LPM is a simple linear regression, but unlike the standard setting
of a linear regression, the dependent variable, or target, is a binary variable, and not a continuous
variable. Before you yell ``Wait, that's illegal'', you should know that in practice LPMs do a good
job of estimating marginal effects, which is what social scientists and econometricians are often
interested in. Marginal effects are another way of interpreting models, giving how the outcome
(or the target) changes given a change in a independent variable (or a feature). For instance,
a marginal effect of 0.10 for age would mean that probability of success would increase by 10\% for
each added year of age. We already discussed marginal effects in Chapter 6.

There has been a lot of discussion on logistic regression vs LPMs, and there are pros and cons
of using LPMs. Micro-econometricians are still fond of LPMs, even though the pros of LPMs are
not really convincing. However, quoting Angrist and Pischke:

``While a nonlinear model may fit the CEF (population conditional expectation function) for LDVs
(limited dependent variables) more closely than a linear model, when it comes to marginal effects,
this probably matters little'' (source: \emph{Mostly Harmless Econometrics})

so LPMs are still used for estimating marginal effects.

Let us check this assessment with one example. First, we simulate some data, then
run a logistic regression and compute the marginal effects, and then compare with a LPM:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{set.seed}\NormalTok{(}\DecValTok{1234}\NormalTok{)}
\NormalTok{x1 }\OtherTok{\textless{}{-}} \FunctionTok{rnorm}\NormalTok{(}\DecValTok{100}\NormalTok{)}
\NormalTok{x2 }\OtherTok{\textless{}{-}} \FunctionTok{rnorm}\NormalTok{(}\DecValTok{100}\NormalTok{)}
  
\NormalTok{z }\OtherTok{\textless{}{-}}\NormalTok{ .}\DecValTok{5} \SpecialCharTok{+} \DecValTok{2}\SpecialCharTok{*}\NormalTok{x1 }\SpecialCharTok{+} \DecValTok{4}\SpecialCharTok{*}\NormalTok{x2}

\NormalTok{p }\OtherTok{\textless{}{-}} \DecValTok{1}\SpecialCharTok{/}\NormalTok{(}\DecValTok{1} \SpecialCharTok{+} \FunctionTok{exp}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{z))}

\NormalTok{y }\OtherTok{\textless{}{-}} \FunctionTok{rbinom}\NormalTok{(}\DecValTok{100}\NormalTok{, }\DecValTok{1}\NormalTok{, p)}

\NormalTok{df }\OtherTok{\textless{}{-}} \FunctionTok{tibble}\NormalTok{(}\AttributeTok{y =}\NormalTok{ y, }\AttributeTok{x1 =}\NormalTok{ x1, }\AttributeTok{x2 =}\NormalTok{ x2)}
\end{Highlighting}
\end{Shaded}

This data generating process generates data from a binary choice model. Fitting the model using a
logistic regression allows us to recover the structural parameters:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{logistic\_regression }\OtherTok{\textless{}{-}} \FunctionTok{glm}\NormalTok{(y }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ df, }\AttributeTok{family =} \FunctionTok{binomial}\NormalTok{(}\AttributeTok{link =} \StringTok{"logit"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

Let's see a summary of the model fit:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{summary}\NormalTok{(logistic\_regression)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Call:
## glm(formula = y ~ ., family = binomial(link = "logit"), data = df)
## 
## Deviance Residuals: 
##      Min        1Q    Median        3Q       Max  
## -2.91941  -0.44872   0.00038   0.42843   2.55426  
## 
## Coefficients:
##             Estimate Std. Error z value Pr(>|z|)    
## (Intercept)   0.0960     0.3293   0.292 0.770630    
## x1            1.6625     0.4628   3.592 0.000328 ***
## x2            3.6582     0.8059   4.539 5.64e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 138.629  on 99  degrees of freedom
## Residual deviance:  60.576  on 97  degrees of freedom
## AIC: 66.576
## 
## Number of Fisher Scoring iterations: 7
\end{verbatim}

We do recover the parameters that generated the data, but what about the marginal effects? We can
get the marginal effects easily using the \texttt{\{margins\}} package:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{library}\NormalTok{(margins)}

\FunctionTok{margins}\NormalTok{(logistic\_regression)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Average marginal effects
\end{verbatim}

\begin{verbatim}
## glm(formula = y ~ ., family = binomial(link = "logit"), data = df)
\end{verbatim}

\begin{verbatim}
##      x1     x2
##  0.1598 0.3516
\end{verbatim}

Or, even better, we can compute the \emph{true} marginal effects, since we know the data
generating process:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{meffects }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(dataset, coefs)\{}
\NormalTok{  X }\OtherTok{\textless{}{-}}\NormalTok{ dataset }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{select}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{y) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{as.matrix}\NormalTok{()}
  
\NormalTok{  dydx\_x1 }\OtherTok{\textless{}{-}} \FunctionTok{mean}\NormalTok{(}\FunctionTok{dlogis}\NormalTok{(X}\SpecialCharTok{\%*\%}\FunctionTok{c}\NormalTok{(coefs[}\DecValTok{2}\NormalTok{], coefs[}\DecValTok{3}\NormalTok{]))}\SpecialCharTok{*}\NormalTok{coefs[}\DecValTok{2}\NormalTok{])}
\NormalTok{  dydx\_x2 }\OtherTok{\textless{}{-}} \FunctionTok{mean}\NormalTok{(}\FunctionTok{dlogis}\NormalTok{(X}\SpecialCharTok{\%*\%}\FunctionTok{c}\NormalTok{(coefs[}\DecValTok{2}\NormalTok{], coefs[}\DecValTok{3}\NormalTok{]))}\SpecialCharTok{*}\NormalTok{coefs[}\DecValTok{3}\NormalTok{])}
  
  \FunctionTok{tribble}\NormalTok{(}\SpecialCharTok{\textasciitilde{}}\NormalTok{term, }\SpecialCharTok{\textasciitilde{}}\NormalTok{true\_effect,}
          \StringTok{"x1"}\NormalTok{, dydx\_x1,}
          \StringTok{"x2"}\NormalTok{, dydx\_x2)}
\NormalTok{\}}

\NormalTok{(true\_meffects }\OtherTok{\textless{}{-}} \FunctionTok{meffects}\NormalTok{(df, }\FunctionTok{c}\NormalTok{(}\FloatTok{0.5}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{4}\NormalTok{)))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 2 x 2
##   term  true_effect
##   <chr>       <dbl>
## 1 x1          0.175
## 2 x2          0.350
\end{verbatim}

Ok, so now what about using this infamous Linear Probability Model to estimate the marginal effects?

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{lpm }\OtherTok{\textless{}{-}} \FunctionTok{lm}\NormalTok{(y }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ df)}

\FunctionTok{summary}\NormalTok{(lpm)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## 
## Call:
## lm(formula = y ~ ., data = df)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.83953 -0.31588 -0.02885  0.28774  0.77407 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  0.51340    0.03587  14.314  < 2e-16 ***
## x1           0.16771    0.03545   4.732 7.58e-06 ***
## x2           0.31250    0.03449   9.060 1.43e-14 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.3541 on 97 degrees of freedom
## Multiple R-squared:  0.5135, Adjusted R-squared:  0.5034 
## F-statistic: 51.18 on 2 and 97 DF,  p-value: 6.693e-16
\end{verbatim}

It's not too bad, but maybe it could have been better in other circumstances. Perhaps if we had more
observations, or perhaps for a different set of structural parameters the results of the LPM
would have been closer. The LPM estimates the marginal effect of \texttt{x1} to be
0.1677134 vs 0.1597956
for the logistic regression and for \texttt{x2}, the LPM estimation is 0.3124966
vs 0.351607. The \emph{true} marginal effects are
0.1750963 and 0.3501926 for \texttt{x1} and \texttt{x2} respectively.

Just as to assess the accuracy of a model data scientists perform cross-validation, a Monte Carlo
study can be performed to asses how close the estimation of the marginal effects using a LPM is
to the marginal effects derived from a logistic regression. It will allow us to test with datasets
of different sizes, and generated using different structural parameters.

First, let's write a function that generates data. The function below generates 10 datasets of size
100 (the code is inspired by this \href{https://stats.stackexchange.com/a/46525}{StackExchange answer}):

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{generate\_datasets }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(}\AttributeTok{coefs =} \FunctionTok{c}\NormalTok{(.}\DecValTok{5}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{4}\NormalTok{), }\AttributeTok{sample\_size =} \DecValTok{100}\NormalTok{, }\AttributeTok{repeats =} \DecValTok{10}\NormalTok{)\{}

\NormalTok{  generate\_one\_dataset }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(coefs, sample\_size)\{}
\NormalTok{  x1 }\OtherTok{\textless{}{-}} \FunctionTok{rnorm}\NormalTok{(sample\_size)}
\NormalTok{  x2 }\OtherTok{\textless{}{-}} \FunctionTok{rnorm}\NormalTok{(sample\_size)}
  
\NormalTok{  z }\OtherTok{\textless{}{-}}\NormalTok{ coefs[}\DecValTok{1}\NormalTok{] }\SpecialCharTok{+}\NormalTok{ coefs[}\DecValTok{2}\NormalTok{]}\SpecialCharTok{*}\NormalTok{x1 }\SpecialCharTok{+}\NormalTok{ coefs[}\DecValTok{3}\NormalTok{]}\SpecialCharTok{*}\NormalTok{x2}

\NormalTok{  p }\OtherTok{\textless{}{-}} \DecValTok{1}\SpecialCharTok{/}\NormalTok{(}\DecValTok{1} \SpecialCharTok{+} \FunctionTok{exp}\NormalTok{(}\SpecialCharTok{{-}}\NormalTok{z))}

\NormalTok{  y }\OtherTok{\textless{}{-}} \FunctionTok{rbinom}\NormalTok{(sample\_size, }\DecValTok{1}\NormalTok{, p)}

\NormalTok{  df }\OtherTok{\textless{}{-}} \FunctionTok{tibble}\NormalTok{(}\AttributeTok{y =}\NormalTok{ y, }\AttributeTok{x1 =}\NormalTok{ x1, }\AttributeTok{x2 =}\NormalTok{ x2)}
\NormalTok{  \}}

\NormalTok{  simulations }\OtherTok{\textless{}{-}} \FunctionTok{rerun}\NormalTok{(}\AttributeTok{.n =}\NormalTok{ repeats, }\FunctionTok{generate\_one\_dataset}\NormalTok{(coefs, sample\_size))}
 
  \FunctionTok{tibble}\NormalTok{(}\StringTok{"coefs"} \OtherTok{=} \FunctionTok{list}\NormalTok{(coefs), }\StringTok{"sample\_size"} \OtherTok{=}\NormalTok{ sample\_size, }\StringTok{"repeats"} \OtherTok{=}\NormalTok{ repeats, }\StringTok{"simulations"} \OtherTok{=} \FunctionTok{list}\NormalTok{(simulations))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Let's first generate one dataset:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{one\_dataset }\OtherTok{\textless{}{-}} \FunctionTok{generate\_datasets}\NormalTok{(}\AttributeTok{repeats =} \DecValTok{1}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Let's take a look at \texttt{one\_dataset}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{one\_dataset}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 4
##   coefs     sample_size repeats simulations
##   <list>          <dbl>   <dbl> <list>     
## 1 <dbl [3]>         100       1 <list [1]>
\end{verbatim}

As you can see, the tibble with the simulated data is inside a list-column called \texttt{simulations}.
Let's take a closer look:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{str}\NormalTok{(one\_dataset}\SpecialCharTok{$}\NormalTok{simulations)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## List of 1
##  $ :List of 1
##   ..$ : tibble [100 x 3] (S3: tbl_df/tbl/data.frame)
##   .. ..$ y : int [1:100] 0 1 1 1 0 1 1 0 0 1 ...
##   .. ..$ x1: num [1:100] 0.437 1.06 0.452 0.663 -1.136 ...
##   .. ..$ x2: num [1:100] -2.316 0.562 -0.784 -0.226 -1.587 ...
\end{verbatim}

The structure is quite complex, and it's important to understand this, because it will have an
impact on the next lines of code; it is a list, containing a list, containing a dataset! No worries
though, we can still map over the datasets directly, by using \texttt{modify\_depth()} instead of \texttt{map()}.

Now, let's fit a LPM and compare the estimation of the marginal effects with the \emph{true} marginal
effects. In order to have some confidence in our results,
we will not simply run a linear regression on that single dataset, but will instead simulate hundreds,
then thousands and ten of thousands of data sets, get the marginal effects and compare
them to the true ones (but here I won't simulate more than 500 datasets).

Let's first generate 10 datasets:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{many\_datasets }\OtherTok{\textless{}{-}} \FunctionTok{generate\_datasets}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

Now comes the tricky part. I have this object, \texttt{many\_datasets} looking like this:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{many\_datasets}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 4
##   coefs     sample_size repeats simulations
##   <list>          <dbl>   <dbl> <list>     
## 1 <dbl [3]>         100      10 <list [10]>
\end{verbatim}

I would like to fit LPMs to the 10 datasets. For this, I will need to use all the power of functional
programming and the \texttt{\{tidyverse\}}. I will be adding columns to this data frame using \texttt{mutate()}
and mapping over the \texttt{simulations} list-column using \texttt{modify\_depth()}. The list of data frames is
at the second level (remember, it's a list containing a list containing data frames).

I'll start by fitting the LPMs, then using \texttt{broom::tidy()} I will get a nice data frame of the
estimated parameters. I will then only select what I need, and then bind the rows of all the
data frames. I will do the same for the \emph{true} marginal effects.

I highly suggest that you run the following lines, one after another. It is complicated to understand
what's going on if you are not used to such workflows. However, I hope to convince you that once
it will click, it'll be much more intuitive than doing all this inside a loop. Here's the code:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{results }\OtherTok{\textless{}{-}}\NormalTok{ many\_datasets }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lpm =} \FunctionTok{modify\_depth}\NormalTok{(simulations, }\DecValTok{2}\NormalTok{, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{lm}\NormalTok{(y }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ .x))) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lpm =} \FunctionTok{modify\_depth}\NormalTok{(lpm, }\DecValTok{2}\NormalTok{, broom}\SpecialCharTok{::}\NormalTok{tidy)) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lpm =} \FunctionTok{modify\_depth}\NormalTok{(lpm, }\DecValTok{2}\NormalTok{, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{select}\NormalTok{(., term, estimate))) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lpm =} \FunctionTok{modify\_depth}\NormalTok{(lpm, }\DecValTok{2}\NormalTok{, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{filter}\NormalTok{(., term }\SpecialCharTok{!=} \StringTok{"(Intercept)"}\NormalTok{))) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lpm =} \FunctionTok{map}\NormalTok{(lpm, bind\_rows)) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{true\_effect =} \FunctionTok{modify\_depth}\NormalTok{(simulations, }\DecValTok{2}\NormalTok{, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{meffects}\NormalTok{(., }\AttributeTok{coefs =}\NormalTok{ coefs[[}\DecValTok{1}\NormalTok{]]))) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{true\_effect =} \FunctionTok{map}\NormalTok{(true\_effect, bind\_rows))}
\end{Highlighting}
\end{Shaded}

This is how results looks like:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{results}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 1 x 6
##   coefs     sample_size repeats simulations lpm               true_effect      
##   <list>          <dbl>   <dbl> <list>      <list>            <list>           
## 1 <dbl [3]>         100      10 <list [10]> <tibble [20 x 2]> <tibble [20 x 2]>
\end{verbatim}

Let's take a closer look to the \texttt{lpm} and \texttt{true\_effect} columns:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{results}\SpecialCharTok{$}\NormalTok{lpm}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## # A tibble: 20 x 2
##    term  estimate
##    <chr>    <dbl>
##  1 x1       0.228
##  2 x2       0.353
##  3 x1       0.180
##  4 x2       0.361
##  5 x1       0.165
##  6 x2       0.374
##  7 x1       0.182
##  8 x2       0.358
##  9 x1       0.125
## 10 x2       0.345
## 11 x1       0.171
## 12 x2       0.331
## 13 x1       0.122
## 14 x2       0.309
## 15 x1       0.129
## 16 x2       0.332
## 17 x1       0.102
## 18 x2       0.374
## 19 x1       0.176
## 20 x2       0.410
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{results}\SpecialCharTok{$}\NormalTok{true\_effect}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## [[1]]
## # A tibble: 20 x 2
##    term  true_effect
##    <chr>       <dbl>
##  1 x1          0.183
##  2 x2          0.366
##  3 x1          0.166
##  4 x2          0.331
##  5 x1          0.174
##  6 x2          0.348
##  7 x1          0.169
##  8 x2          0.339
##  9 x1          0.167
## 10 x2          0.335
## 11 x1          0.173
## 12 x2          0.345
## 13 x1          0.157
## 14 x2          0.314
## 15 x1          0.170
## 16 x2          0.340
## 17 x1          0.182
## 18 x2          0.365
## 19 x1          0.161
## 20 x2          0.321
\end{verbatim}

Let's bind the columns, and compute the difference between the \emph{true} and estimated marginal
effects:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simulation\_results }\OtherTok{\textless{}{-}}\NormalTok{ results }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{difference =} \FunctionTok{map2}\NormalTok{(}\AttributeTok{.x =}\NormalTok{ lpm, }\AttributeTok{.y =}\NormalTok{ true\_effect, full\_join)) }\SpecialCharTok{\%\textgreater{}\%}  
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{difference =} \FunctionTok{map}\NormalTok{(difference, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{mutate}\NormalTok{(., }\AttributeTok{difference =}\NormalTok{ true\_effect }\SpecialCharTok{{-}}\NormalTok{ estimate))) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{mutate}\NormalTok{(}\AttributeTok{difference =} \FunctionTok{map}\NormalTok{(difference, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{select}\NormalTok{(., term, difference))) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{pull}\NormalTok{(difference) }\SpecialCharTok{\%\textgreater{}\%} 
\NormalTok{  .[[}\DecValTok{1}\NormalTok{]]}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Joining, by = "term"
\end{verbatim}

Let's take a look at the simulation results:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{simulation\_results }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{group\_by}\NormalTok{(term) }\SpecialCharTok{\%\textgreater{}\%} 
  \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean =} \FunctionTok{mean}\NormalTok{(difference), }
            \AttributeTok{sd =} \FunctionTok{sd}\NormalTok{(difference))}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## # A tibble: 2 x 3
##   term     mean     sd
##   <chr>   <dbl>  <dbl>
## 1 x1     0.0122 0.0368
## 2 x2    -0.0141 0.0311
\end{verbatim}

Already with only 10 simulated datasets, the difference in means is not significant. Let's rerun
the analysis, but for difference sizes. In order to make things easier, we can put all the code
into a nifty function:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{monte\_carlo }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(coefs, sample\_size, repeats)\{}
\NormalTok{  many\_datasets }\OtherTok{\textless{}{-}} \FunctionTok{generate\_datasets}\NormalTok{(coefs, sample\_size, repeats)}
  
\NormalTok{  results }\OtherTok{\textless{}{-}}\NormalTok{ many\_datasets }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lpm =} \FunctionTok{modify\_depth}\NormalTok{(simulations, }\DecValTok{2}\NormalTok{, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{lm}\NormalTok{(y }\SpecialCharTok{\textasciitilde{}}\NormalTok{ ., }\AttributeTok{data =}\NormalTok{ .x))) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lpm =} \FunctionTok{modify\_depth}\NormalTok{(lpm, }\DecValTok{2}\NormalTok{, broom}\SpecialCharTok{::}\NormalTok{tidy)) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lpm =} \FunctionTok{modify\_depth}\NormalTok{(lpm, }\DecValTok{2}\NormalTok{, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{select}\NormalTok{(., term, estimate))) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lpm =} \FunctionTok{modify\_depth}\NormalTok{(lpm, }\DecValTok{2}\NormalTok{, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{filter}\NormalTok{(., term }\SpecialCharTok{!=} \StringTok{"(Intercept)"}\NormalTok{))) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{lpm =} \FunctionTok{map}\NormalTok{(lpm, bind\_rows)) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{true\_effect =} \FunctionTok{modify\_depth}\NormalTok{(simulations, }\DecValTok{2}\NormalTok{, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{meffects}\NormalTok{(., }\AttributeTok{coefs =}\NormalTok{ coefs[[}\DecValTok{1}\NormalTok{]]))) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{true\_effect =} \FunctionTok{map}\NormalTok{(true\_effect, bind\_rows))}

\NormalTok{  simulation\_results }\OtherTok{\textless{}{-}}\NormalTok{ results }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{difference =} \FunctionTok{map2}\NormalTok{(}\AttributeTok{.x =}\NormalTok{ lpm, }\AttributeTok{.y =}\NormalTok{ true\_effect, full\_join)) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{difference =} \FunctionTok{map}\NormalTok{(difference, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{mutate}\NormalTok{(., }\AttributeTok{difference =}\NormalTok{ true\_effect }\SpecialCharTok{{-}}\NormalTok{ estimate))) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{difference =} \FunctionTok{map}\NormalTok{(difference, }\SpecialCharTok{\textasciitilde{}}\FunctionTok{select}\NormalTok{(., term, difference))) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{pull}\NormalTok{(difference) }\SpecialCharTok{\%\textgreater{}\%} 
\NormalTok{    .[[}\DecValTok{1}\NormalTok{]]}

\NormalTok{  simulation\_results }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{group\_by}\NormalTok{(term) }\SpecialCharTok{\%\textgreater{}\%} 
    \FunctionTok{summarise}\NormalTok{(}\AttributeTok{mean =} \FunctionTok{mean}\NormalTok{(difference), }
              \AttributeTok{sd =} \FunctionTok{sd}\NormalTok{(difference))}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

And now, let's run the simulation for different parameters and sizes:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{monte\_carlo}\NormalTok{(}\FunctionTok{c}\NormalTok{(.}\DecValTok{5}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{4}\NormalTok{), }\DecValTok{100}\NormalTok{, }\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Joining, by = "term"
\end{verbatim}

\begin{verbatim}
## # A tibble: 2 x 3
##   term      mean     sd
##   <chr>    <dbl>  <dbl>
## 1 x1    -0.00826 0.0318
## 2 x2    -0.00732 0.0421
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{monte\_carlo}\NormalTok{(}\FunctionTok{c}\NormalTok{(.}\DecValTok{5}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{4}\NormalTok{), }\DecValTok{100}\NormalTok{, }\DecValTok{100}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Joining, by = "term"
\end{verbatim}

\begin{verbatim}
## # A tibble: 2 x 3
##   term     mean     sd
##   <chr>   <dbl>  <dbl>
## 1 x1    0.00360 0.0408
## 2 x2    0.00517 0.0459
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{monte\_carlo}\NormalTok{(}\FunctionTok{c}\NormalTok{(.}\DecValTok{5}\NormalTok{, }\DecValTok{2}\NormalTok{, }\DecValTok{4}\NormalTok{), }\DecValTok{100}\NormalTok{, }\DecValTok{500}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Joining, by = "term"
\end{verbatim}

\begin{verbatim}
## # A tibble: 2 x 3
##   term       mean     sd
##   <chr>     <dbl>  <dbl>
## 1 x1    -0.00152  0.0388
## 2 x2    -0.000701 0.0462
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{monte\_carlo}\NormalTok{(}\FunctionTok{c}\NormalTok{(pi, }\DecValTok{6}\NormalTok{, }\DecValTok{9}\NormalTok{), }\DecValTok{100}\NormalTok{, }\DecValTok{10}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Joining, by = "term"
\end{verbatim}

\begin{verbatim}
## # A tibble: 2 x 3
##   term      mean     sd
##   <chr>    <dbl>  <dbl>
## 1 x1    -0.00829 0.0421
## 2 x2     0.00178 0.0397
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{monte\_carlo}\NormalTok{(}\FunctionTok{c}\NormalTok{(pi, }\DecValTok{6}\NormalTok{, }\DecValTok{9}\NormalTok{), }\DecValTok{100}\NormalTok{, }\DecValTok{100}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Joining, by = "term"
\end{verbatim}

\begin{verbatim}
## # A tibble: 2 x 3
##   term     mean     sd
##   <chr>   <dbl>  <dbl>
## 1 x1    0.0107  0.0576
## 2 x2    0.00831 0.0772
\end{verbatim}

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{monte\_carlo}\NormalTok{(}\FunctionTok{c}\NormalTok{(pi, }\DecValTok{6}\NormalTok{, }\DecValTok{9}\NormalTok{), }\DecValTok{100}\NormalTok{, }\DecValTok{500}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{verbatim}
## Joining, by = "term"
\end{verbatim}

\begin{verbatim}
## # A tibble: 2 x 3
##   term     mean     sd
##   <chr>   <dbl>  <dbl>
## 1 x1    0.00879 0.0518
## 2 x2    0.0113  0.0687
\end{verbatim}

We see that, at least for this set of parameters, the LPM does a good job of estimating marginal
effects.

Now, this study might in itself not be very interesting to you, but I believe the general approach
is quite useful and flexible enough to be adapted to all kinds of use-cases.

\hypertarget{exercises-6}{%
\subsection{Exercises}\label{exercises-6}}

\hypertarget{exercise-1-6}{%
\subsubsection*{Exercise 1}\label{exercise-1-6}}
\addcontentsline{toc}{subsubsection}{Exercise 1}

Suppose you have an Excel workbook that contains data on three sheets. Create a function that
reads entire workbooks, and that returns a list of tibbles, where each tibble is the data of one
sheet (download the example Excel workbook, \texttt{example\_workbook.xlsx}, from the \texttt{assets} folder on
the books github).

\hypertarget{exercise-2-4}{%
\subsubsection*{Exercise 2}\label{exercise-2-4}}
\addcontentsline{toc}{subsubsection}{Exercise 2}

Use one of the \texttt{map()} functions to combine two lists into one. Consider the following two lists:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mediterranean }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\StringTok{"starters"} \OtherTok{=} \FunctionTok{list}\NormalTok{(}\StringTok{"humous"}\NormalTok{, }\StringTok{"lasagna"}\NormalTok{), }\StringTok{"dishes"} \OtherTok{=} \FunctionTok{list}\NormalTok{(}\StringTok{"sardines"}\NormalTok{, }\StringTok{"olives"}\NormalTok{))}

\NormalTok{continental }\OtherTok{\textless{}{-}} \FunctionTok{list}\NormalTok{(}\StringTok{"starters"} \OtherTok{=} \FunctionTok{list}\NormalTok{(}\StringTok{"pea soup"}\NormalTok{, }\StringTok{"terrine"}\NormalTok{), }\StringTok{"dishes"} \OtherTok{=} \FunctionTok{list}\NormalTok{(}\StringTok{"frikadelle"}\NormalTok{, }\StringTok{"sauerkraut"}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

The result we'd like to have would look like this:

\begin{Shaded}
\begin{Highlighting}[]
\SpecialCharTok{$}\NormalTok{starters}
\SpecialCharTok{$}\NormalTok{starters[[}\DecValTok{1}\NormalTok{]]}
\NormalTok{[}\DecValTok{1}\NormalTok{] }\StringTok{"humous"}

\SpecialCharTok{$}\NormalTok{starters[[}\DecValTok{2}\NormalTok{]]}
\NormalTok{[}\DecValTok{1}\NormalTok{] }\StringTok{"olives"}

\SpecialCharTok{$}\NormalTok{starters[[}\DecValTok{3}\NormalTok{]]}
\NormalTok{[}\DecValTok{1}\NormalTok{] }\StringTok{"pea soup"}

\SpecialCharTok{$}\NormalTok{starters[[}\DecValTok{4}\NormalTok{]]}
\NormalTok{[}\DecValTok{1}\NormalTok{] }\StringTok{"terrine"}


\SpecialCharTok{$}\NormalTok{dishes}
\SpecialCharTok{$}\NormalTok{dishes[[}\DecValTok{1}\NormalTok{]]}
\NormalTok{[}\DecValTok{1}\NormalTok{] }\StringTok{"sardines"}

\SpecialCharTok{$}\NormalTok{dishes[[}\DecValTok{2}\NormalTok{]]}
\NormalTok{[}\DecValTok{1}\NormalTok{] }\StringTok{"lasagna"}

\SpecialCharTok{$}\NormalTok{dishes[[}\DecValTok{3}\NormalTok{]]}
\NormalTok{[}\DecValTok{1}\NormalTok{] }\StringTok{"frikadelle"}

\SpecialCharTok{$}\NormalTok{dishes[[}\DecValTok{4}\NormalTok{]]}
\NormalTok{[}\DecValTok{1}\NormalTok{] }\StringTok{"sauerkraut"}
\end{Highlighting}
\end{Shaded}

\hypertarget{package-development}{%
\section{Package development}\label{package-development}}

\hypertarget{why-you-need-to-write-your-own-package}{%
\subsection{Why you need to write your own package}\label{why-you-need-to-write-your-own-package}}

One of the reasons you might have tried R in the first place is the abundance of packages. As I'm
writing these lines (in November 2020) 16523 packages are available on CRAN (in August 2019, there
were 14762, and in August 2016, when I first wrote the number of packages down for my first ebook,
it was 8922 packages).

This is a staggering amount of packages and to help you look for the right ones, you can check
out \href{https://cran.r-project.org/}{CRAN Task Views}.

You might wonder why the heck should you write your own packages? After all, with so many packages
you're sure to find something that suits your needs, right? Well, it depends. Of course, you will
not need to write you own function to perform non-linear regression, or to train a neural network.
But as time will go, you will start writing your own functions, functions that fit your needs, and
that you use daily. It may be functions that prepare and shape data that you use at work for
analysis. Or maybe you want to deliver an analysis to a client, with data and source code, so
you decide to deliver a package that contains everything (something I've already done in the
past). Maybe you want to develop a Shiny applications using the \texttt{\{golem\}} framework, which allows
you to build apps as packages.

Ok, but is it necessary to write a package? Why not just write functions inside some scripts and
then simply run or share these scripts (and in the case of Shiny, you don't have to use \texttt{\{golem\}})?
This seems like a valid solution at first. However, it quickly becomes tedious, especially if you
have multiple scripts scattered around your computer or inside different subfolders. You'll also
have to write the documentation on separate files and these can easily get lost or become outdated.
Relying on scripts does not scale well; even if you are not sharing your code outside of your
computer (maybe you're working on super secret projects at NASA), you always have to think about
future you. And in general, future you thinks that past you is an asshole, exactly because you put
0 effort in documenting, testing and making your code easy to use. Having everything inside a
package takes care of these headaches for you, and will make future you proud of past you. And if
you have to share your code, or deliver to a client, believe me, it will make things a thousand
times easier.

Code that is inside packages is very easy to document and test, especially if you're using Rstudio.
It also makes it possible to use the wonderful \texttt{\{covr\}} package, which tells you which lines in
which functions are called by your tests. If some lines are missing, write tests that invoke them and
increase the coverage of your tests! Documenting and testing your code is very important; it gives
you assurance that the code your writing works, but most importantly, it gives \emph{others} assurance
that what you wrote works. And I include future you in these \emph{others} too.

In order to share this package with these \emph{others} we are going to use Git. If you're familiar with
Git, great, you'll be able to skip some sections. If not, then buckle up, you're in for a wild ride.

As I mentioned in the introduction, if you want to learn much more than I'll show about packages
read \citet{wickham2015}. I will only show you the basics, but it should be enough to get you productive.

\hypertarget{starting-easy-creating-a-package-to-share-data}{%
\subsection{Starting easy: creating a package to share data}\label{starting-easy-creating-a-package-to-share-data}}

We will start a package from scratch, in order to share data with the world. For this, we are first
going to scrape a table off Wikipedia, prepare the data and then include it in a package. To make
distributing this package easy, we're going to put it up on Github, so you'll need a Github account.

Let's start by creating a Github account.

\hypertarget{setting-up-a-github-account}{%
\subsubsection{Setting up a Github account}\label{setting-up-a-github-account}}

Setting up a Github account is very easy; just go over to \url{https://github.com/}
and simply sign up!

Then you will need to generate a ssh key on your computer. This is a way for you to securely
interact with your Github account, and push your code to the repository without having to always
type your password. I will assume you never created any ssh
keys before, because if you already did, you could skip these steps. I will also assume that you are
on a GNU+Linux or macOS system; if you're using windows, the instructions are very similar, but
you'll first need to install Git available \href{https://git-scm.com/downloads}{here}. Git is available
by default on any GNU+Linux system, and as far as I know also on macOS, but I might be wrong and
you might also need to install git on macOS (but then the instructions are the same whether
you're using GNU+Linux or macOS). If you have trouble installing git, read the following section
from the \href{https://git-scm.com/book/en/v2/Getting-Started-Installing-Git}{Pro Git book}.

Then, open a terminal (or the git command line on Windows) and type the following:

\begin{verbatim}
ssh-keygen
\end{verbatim}

This command will generate several files in the \texttt{.ssh} directory inside your \texttt{HOME} directory. Look
for the file that ends with the \texttt{.pub} extension, and copy its contents. You will need to paste
these contents on Github.

So now sign in to Github; once you are signed in, go to settings and then \texttt{SSH\ and\ GPG\ keys}:

\includegraphics[width=14.31in]{pics/github_ssh}

In the screenshot above, you see my ssh key associated with my account; this will be empty for you.
Click on the top right, \emph{New SSH key}:

\includegraphics[width=14.32in]{pics/github_add_ssh}

Give your key a name, and paste the key you generated before. You're done! You can now configure
git a bit more by telling it who you are. Open a terminal, adapt and type the following commands:

\begin{verbatim}
git config --global user.name "Harold Zurcher"
git config --global user.email harold.zurcher@madisonbus.com
\end{verbatim}

You're ready to go!\footnote{If you know who Harold Zurcher is, you da real MVP}
You can now push code to github to share it with the world. Or if you do not want
to share you package (for confidentiality reasons for instance), you can still benefit from using
git, as it possible to have an internal git server that could be managed by your company's IT team.
There is also the possibility to set up corporate, and thus private git servers by buying the service
from github, or other providers such as gitlab.

\hypertarget{starting-your-package}{%
\subsubsection{Starting your package}\label{starting-your-package}}

\includegraphics{pics/new_package.gif}

To start writing a package, the easiest way is to load up Rstudio and start a new project, under the
\emph{File} menu. If you're starting from scratch, just choose the first option, \emph{New Directory} and then
\emph{R package}. Give a new to your package, for example \texttt{arcade} (you'll see why in a bit) and you can
also choose to use git for version control. Now if you check the folder where you chose to save
your package, you will see a folder with the same name as your package, and inside this folder a
lot of new files and other folders. The most important folder for now is the \texttt{R} folder. This is
the folder that will hold your \texttt{.R} source code files. You can also see these files and folders
inside the \emph{Files} panel from within Rstudio. Rstudio will also have \texttt{hello.R} opened, which is a
single demo source file inside the \texttt{R} folder. You can get rid of this file, or keep it and edit it.
I would advise you keep it and even distribute it inside your package. You can save this file
in a special directory called \texttt{data-raw}. You don't need to manually create this folder now, we will
do so in a bit. For now, just follow along.

Now, to start working on your package, the best is to use a package called \texttt{\{usethis\}}. \texttt{\{usethis\}}
is a package that makes writing packages very easy; it includes functions that create the required
subfolders and necessary template files so that you do not need to constantly check how file so-and-so
should be placed or named.

Let's start by adding a readme file. This is easily achieved by using the following function from
\texttt{\{usethis\}}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{usethis}\SpecialCharTok{::}\FunctionTok{use\_readme\_md}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

This creates a template README.md file in the root directory of your package. You can now edit this
file accordingly, and that's it.

The next step could be setting up your package to work with \texttt{\{roxygen2\}}, which will help write
the documentation of your package:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{usethis}\SpecialCharTok{::}\FunctionTok{use\_roxygen\_md}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

The output tells you to run \texttt{devtools::document()}, we will do this later.

Since you have learned about the tidyverse by reading this book, I am willing to bet that you will
want to use the \texttt{\%\textgreater{}\%} operator inside the functions contained in your package. To do this without issues,
which wil become apparent later, use the following command:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{usethis}\SpecialCharTok{::}\FunctionTok{use\_pipe}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

This will make the \texttt{\%\textgreater{}\%} operator available internally to your package's functions, but also to the
user that will load the package.

We are almost done setting up the package. If you plan on distributing data with your package,
you might want to also share the code that prepared the data. For instance, if you receive the
data from your finance department, but this data needs some cleaning before being useful, you could
write a script to do so and then distribute this script also with the package, for reproducibility
purposes. These scripts, while not central to the package, could still be of interest to the users.
The directory to place them is called \texttt{data-raw}:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{usethis}\SpecialCharTok{::}\FunctionTok{use\_data\_raw}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

One final folder is \texttt{inst}. You can add files to this folder, and they will be available to the users
that install the package. Users can find the files in the folder where packages get installed. On
GNU+Linux systems, that would be somewhere like: \texttt{/home/user/R/amd64-linux-gnu-library/3.6}. There,
you will find the installation folders of all the packages. If the package you make is called \texttt{\{spam\}},
you will find the files you put inside the \texttt{inst} folder on the root of the installation folder of
\texttt{spam}. You can simply create the \texttt{inst} folder yourself, or use the following command:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{usethis}\SpecialCharTok{::}\FunctionTok{use\_directory}\NormalTok{(}\StringTok{"inst"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Finally, the last step is to give your package a license; this again is only useful if you plan on
distributing it to the world. If you are writing your own package for yourself, or for purposes
internal to your company, this is probably superfluous. I won't discuss the particularities of
licenses, so let's just say that for the sake of this example package we are writing, we are going
to use the MIT license:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{usethis}\SpecialCharTok{::}\FunctionTok{use\_mit\_license}\NormalTok{()}
\end{Highlighting}
\end{Shaded}

This again creates the right file at the right spot. There are other interesting functions inside
the \texttt{\{usethis\}} package, and we will come back to it later.

\hypertarget{including-data-inside-the-package}{%
\subsection{Including data inside the package}\label{including-data-inside-the-package}}

Many packages include data and we are going to learn how to do it. I'll assume that we already
have a dataset on hand that we have to share. This is quite simple to do, first let's simply
load the data:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{arcade }\OtherTok{\textless{}{-}}\NormalTok{ readr}\SpecialCharTok{::}\FunctionTok{read\_csv}\NormalTok{(}\StringTok{"\textasciitilde{}/path/to/data/arcade.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

and then use, once again, \texttt{\{usethis\}} comes to our rescue:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{usethis}\SpecialCharTok{::}\FunctionTok{use\_data}\NormalTok{(arcade, }\AttributeTok{compress =} \StringTok{"xz"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

that's it! Well almost. We still need to write a little script that will allow users of your
package to load the data. This script is simply called \texttt{data.R} and contains the following lines:

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{\#\textquotesingle{} List of highest{-}grossing games}
\CommentTok{\#\textquotesingle{}}
\CommentTok{\#\textquotesingle{} Source: https://en.wikipedia.org/wiki/Arcade\_game\#List\_of\_highest{-}grossing\_games}
\CommentTok{\#\textquotesingle{}}
\CommentTok{\#\textquotesingle{} @format A data frame with 6 variables: \textbackslash{}code\{game\}, \textbackslash{}code\{release\_year\},}
\CommentTok{\#\textquotesingle{}   \textbackslash{}code\{hardware\_units\_sold\}, \textbackslash{}code\{comment\_hardware\}, \textbackslash{}code\{estimated\_gross\_revenue\}, }
\CommentTok{\#\textquotesingle{}   \textbackslash{}code\{comment\_revenue\}}
\CommentTok{\#\textquotesingle{} \textbackslash{}describe\{}
\CommentTok{\#\textquotesingle{} \textbackslash{}item\{game\}\{The name of the game\}}
\CommentTok{\#\textquotesingle{} \textbackslash{}item\{release\_year\}\{The year the game was released\}}
\CommentTok{\#\textquotesingle{} \textbackslash{}item\{hardware\_units\_sold\}\{The amount of hardware units sold\}}
\CommentTok{\#\textquotesingle{} \textbackslash{}item\{comment\_hardware\}\{Comment accompanying the amount of hardware units sold\}}
\CommentTok{\#\textquotesingle{} \textbackslash{}item\{estimated\_gross\_revenue\}\{Estimated gross revenue in US$ with 2019 inflation\}}
\CommentTok{\#\textquotesingle{} \textbackslash{}item\{comment\_revenue\}\{Comment accompanying the amount of hardware units sold\}}
\CommentTok{\#\textquotesingle{} \}}
\StringTok{"arcade"}
\end{Highlighting}
\end{Shaded}

Basically this is a description of the data, and the name with which the user will invoke the data. To
conclude this part, remember the \texttt{data-raw} folder? If you used a script to scrape/get the data
from somewhere, or if you had to write code to prepare the data to make it fit for sharing, this
is where you can put that script. I have written such a script, I will discuss it in the next
chapter, where I'll show you how to scrape data from the internet. You can also save the file
where you wrote all your calls to \texttt{\{usethis\}} functions if you want.

\hypertarget{adding-functions-to-your-package}{%
\subsection{Adding functions to your package}\label{adding-functions-to-your-package}}

Functions will be added inside the \texttt{R} package. In there, you will find the \texttt{hello.R} file. You can
edit this file if you kept it or you can create a new script. This script can hold one function, or
several functions.
Let's start with the simplest case; one function inside one script.

\hypertarget{one-function-inside-one-script}{%
\subsubsection{One function inside one script}\label{one-function-inside-one-script}}

Create a new R script, or edit the \texttt{hello.R} file, and add in the following code:

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{\#\textquotesingle{} Compute descriptive statistics for the numeric columns of a data frame.}
\CommentTok{\#\textquotesingle{} @param df The data frame to summarise.}
\CommentTok{\#\textquotesingle{} @param ... Optional. Columns in the data frame}
\CommentTok{\#\textquotesingle{} @return A data frame with descriptive statistics. If you are only interested in certain columns}
\CommentTok{\#\textquotesingle{} you can add these columns.}
\CommentTok{\#\textquotesingle{} @import dplyr}
\CommentTok{\#\textquotesingle{} @importFrom tidyr gather}
\CommentTok{\#\textquotesingle{} @export}
\CommentTok{\#\textquotesingle{} @examples}
\CommentTok{\#\textquotesingle{} \textbackslash{}dontrun\{}
\CommentTok{\#\textquotesingle{} describe(dataset)}
\CommentTok{\#\textquotesingle{} describe(dataset, col1, col2)}
\CommentTok{\#\textquotesingle{} \}}
\NormalTok{describe\_numeric }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(df, ...)\{}

    \ControlFlowTok{if}\NormalTok{ (}\FunctionTok{nargs}\NormalTok{() }\SpecialCharTok{\textgreater{}} \DecValTok{1}\NormalTok{) df }\OtherTok{\textless{}{-}} \FunctionTok{select}\NormalTok{(df, ...)}

\NormalTok{    df }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{select\_if}\NormalTok{(is.numeric) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{gather}\NormalTok{(variable, value) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{group\_by}\NormalTok{(variable) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{summarise\_all}\NormalTok{(}\FunctionTok{list}\NormalTok{(}\AttributeTok{mean =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{mean}\NormalTok{(., }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                           \AttributeTok{sd =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{sd}\NormalTok{(., }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                           \AttributeTok{nobs =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{length}\NormalTok{(.),}
                           \AttributeTok{min =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{min}\NormalTok{(., }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                           \AttributeTok{max =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{max}\NormalTok{(., }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                           \AttributeTok{q05 =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{quantile}\NormalTok{(., }\FloatTok{0.05}\NormalTok{, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                           \AttributeTok{q25 =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{quantile}\NormalTok{(., }\FloatTok{0.25}\NormalTok{, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                           \AttributeTok{mode =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{as.character}\NormalTok{(brotools}\SpecialCharTok{::}\FunctionTok{sample\_mode}\NormalTok{(.), }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                           \AttributeTok{median =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{quantile}\NormalTok{(., }\FloatTok{0.5}\NormalTok{, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                           \AttributeTok{q75 =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{quantile}\NormalTok{(., }\FloatTok{0.75}\NormalTok{, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                           \AttributeTok{q95 =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{quantile}\NormalTok{(., }\FloatTok{0.95}\NormalTok{, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                           \AttributeTok{n\_missing =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{sum}\NormalTok{(}\FunctionTok{is.na}\NormalTok{(.)))) }\SpecialCharTok{\%\textgreater{}\%}
        \FunctionTok{mutate}\NormalTok{(}\AttributeTok{type =} \StringTok{"Numeric"}\NormalTok{)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Save the script under the name \texttt{describe.R}.

This function shows you pretty much you need to know when writing functions for packages. First,
there's the comment lines, that start with \texttt{\#\textquotesingle{}} and not with \texttt{\#}. These lines will be converted
into the function's documentation which you and your package's users will be able to read in
Rstudio's \emph{Help} pane. Notice the keywords that start with \texttt{@}. These are quite important:

\begin{itemize}
\tightlist
\item
  \texttt{@param}: used to define the function's parameters;
\item
  \texttt{@return}: used to define the object returned by the function;
\item
  \texttt{@import}: if the function needs functions from another package, in the present case \texttt{\{dplyr\}};
  then this is where you would define these. Separate several package with a space;
\item
  \texttt{@importFrom}: if the function only needs one function from a package, define it here. Read it as
  \emph{from tidyr import gather}, very similar to how it is done in Python;
\item
  \texttt{@export}: makes the function available to the users. If you omit this, this function will not
  be available to the users and only available internally to the other functions of the package. Not
  making functions available to users can be useful if you need to write functions that are used by
  other functions but never be used by anyone directly. It is still possible to access these internal,
  private, functions by using \texttt{:::}, as in, \texttt{package:::private\_function()};
\item
  \texttt{@examples}: lists examples in the documentation. The \texttt{\textbackslash{}dontrun\{\}} tag is used for when you do
  not want these examples to run when building the package.
\end{itemize}

As explained before, if the function depends on function from other packages, then \texttt{@import} or
\texttt{@importFrom} must be used. But it is also possible to use the \texttt{package::function()} syntax like
I did on the following line:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{mode }\OtherTok{=} \ErrorTok{\textasciitilde{}}\FunctionTok{as.character}\NormalTok{(brotools}\SpecialCharTok{::}\FunctionTok{sample\_mode}\NormalTok{(.), }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
\end{Highlighting}
\end{Shaded}

This function uses the \texttt{sample\_mode()} function from my \texttt{\{brotools\}} package. Since it is the only
function that I am using, I don't import the whole package with \texttt{@import}. I could have done the
same for \texttt{gather()} from \texttt{\{tidyr\}} instead of using \texttt{@importFrom}, but I wanted to showcase
\texttt{@importFrom}, which can also be use to import several functions:

\begin{verbatim}
@importFrom package function_1 function_2 function_3
\end{verbatim}

The way I'm doing this however is not optimal. If your package depends on many functions from
other packages that are not available on CRAN, but rather on Github, you might want to do that
in a cleaner way. The cleaner way is to add a ``Remotes'' field in the package's NAMESPACE (this is
a very important file that gets generated automatically by \texttt{devtools::document()}) I won't
cover this here, but you can read more about it \href{https://cran.r-project.org/web/packages/devtools/vignettes/dependencies.html}{here}.
What I will cover is how to declare dependencies to other CRAN packages. These dependencies also
get declared inside the `Description' file, which we will cover in the next section.

Because I'm doing that in this hacky way, my \texttt{\{brotools\}} package should be installed:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{devtools}\SpecialCharTok{::}\FunctionTok{install\_github}\NormalTok{(}\StringTok{"b{-}rodrigues/brotools"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

Again, I want to emphasize that this is not the best way of doing it. However, using the ``REMOTES''
field as described in the document I linked above is not complicated.

Now comes the function itself. The function is written in pretty much the same way as usual, but
there are some particularities. First of all, the second argument of the function is the \texttt{...}, which
were already covered in Chapter 7. I want to give the option to my users to specify any columns to
summarise only these columns, instead of all of them, which is the default behaviour. But because
I cannot know how many columns the user wants to summarize beforehand, and also because I do not
want to limit the user to 2 or 3 columns, I use the \texttt{...}.
But what if the user wants to summarize all the columns? This is taken care of in this line:

\begin{Shaded}
\begin{Highlighting}[]
  \ControlFlowTok{if}\NormalTok{ (}\FunctionTok{nargs}\NormalTok{() }\SpecialCharTok{\textgreater{}} \DecValTok{1}\NormalTok{) df }\OtherTok{\textless{}{-}} \FunctionTok{select}\NormalTok{(df, ...)}
\end{Highlighting}
\end{Shaded}

\texttt{nargs()} counts the number of arguments of the function. If the user calls the function like so:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{describe\_numeric}\NormalTok{(mtcars)}
\end{Highlighting}
\end{Shaded}

\texttt{nargs()} will return 1. If, instead, the user calls the function with one or more columns:

\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{describe\_numeric}\NormalTok{(mtcars, hp, mpg)}
\end{Highlighting}
\end{Shaded}

then \texttt{nargs()} will return 2 (in this case). And does, this piece of code will be executed:

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{df }\OtherTok{\textless{}{-}} \FunctionTok{select}\NormalTok{(df, ...)}
\end{Highlighting}
\end{Shaded}

which selects the columns \texttt{hp} and \texttt{mpg} from the \texttt{mtcars} dataset. This reduced data set is then
the one that is being summarized.

\hypertarget{many-functions-inside-a-script}{%
\subsubsection{Many functions inside a script}\label{many-functions-inside-a-script}}

If you need to add more functions, you can add more in the same
script, or create one script per function. The advantage of writing more than one function per
script is that you can keep functions that are conceptually similar in the same place. For instance,
if you want to add a function called \texttt{describe\_character()} to your package, adding it to the same
script where \texttt{describe\_numeric()} is might be a good idea, so let's do just that:

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{\#\textquotesingle{} Compute descriptive statistics for the numeric columns of a data frame.}
\CommentTok{\#\textquotesingle{} @param df The data frame to summarise.}
\CommentTok{\#\textquotesingle{} @param ... Optional. Columns in the data frame}
\CommentTok{\#\textquotesingle{} @return A data frame with descriptive statistics. If you are only interested in certain columns}
\CommentTok{\#\textquotesingle{} you can add these columns.}
\CommentTok{\#\textquotesingle{} @import dplyr}
\CommentTok{\#\textquotesingle{} @importFrom tidyr gather}
\CommentTok{\#\textquotesingle{} @export}
\CommentTok{\#\textquotesingle{} @examples}
\CommentTok{\#\textquotesingle{} \textbackslash{}dontrun\{}
\CommentTok{\#\textquotesingle{} describe(dataset)}
\CommentTok{\#\textquotesingle{} describe(dataset, col1, col2)}
\CommentTok{\#\textquotesingle{} \}}
\NormalTok{describe\_numeric }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(df, ...)\{}

  \ControlFlowTok{if}\NormalTok{ (}\FunctionTok{nargs}\NormalTok{() }\SpecialCharTok{\textgreater{}} \DecValTok{1}\NormalTok{) df }\OtherTok{\textless{}{-}} \FunctionTok{select}\NormalTok{(df, ...)}

\NormalTok{  df }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{select}\NormalTok{(is.numeric) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{pivot\_longer}\NormalTok{(}\AttributeTok{cols =} \FunctionTok{everything}\NormalTok{(),}
                 \AttributeTok{names\_to =} \StringTok{"variable"}\NormalTok{, }\AttributeTok{values\_to =} \StringTok{"value"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(variable) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{everything}\NormalTok{(),}
\NormalTok{                     tibble}\SpecialCharTok{::}\FunctionTok{lst}\NormalTok{(}\AttributeTok{mean =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{mean}\NormalTok{(., }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                       \AttributeTok{sd =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{sd}\NormalTok{(., }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                       \AttributeTok{nobs =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{length}\NormalTok{(.),}
                       \AttributeTok{min =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{min}\NormalTok{(., }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                       \AttributeTok{max =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{max}\NormalTok{(., }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                       \AttributeTok{q05 =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{quantile}\NormalTok{(., }\FloatTok{0.05}\NormalTok{, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                       \AttributeTok{q25 =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{quantile}\NormalTok{(., }\FloatTok{0.25}\NormalTok{, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                       \AttributeTok{mode =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{as.character}\NormalTok{(brotools}\SpecialCharTok{::}\FunctionTok{sample\_mode}\NormalTok{(.), }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                       \AttributeTok{median =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{quantile}\NormalTok{(., }\FloatTok{0.5}\NormalTok{, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                       \AttributeTok{q75 =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{quantile}\NormalTok{(., }\FloatTok{0.75}\NormalTok{, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                       \AttributeTok{q95 =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{quantile}\NormalTok{(., }\FloatTok{0.95}\NormalTok{, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                       \AttributeTok{n\_missing =} \SpecialCharTok{\textasciitilde{}}\FunctionTok{sum}\NormalTok{(}\FunctionTok{is.na}\NormalTok{(.))))) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{type =} \StringTok{"Numeric"}\NormalTok{)}
\NormalTok{\}}

\CommentTok{\#\textquotesingle{} Compute descriptive statistics for the character or factor columns of a data frame.}
\CommentTok{\#\textquotesingle{} @param df The data frame to summarise.}
\CommentTok{\#\textquotesingle{} @return A data frame with a description of the character or factor columns.}
\CommentTok{\#\textquotesingle{} @import dplyr}
\CommentTok{\#\textquotesingle{} @importFrom tidyr gather}
\NormalTok{describe\_character\_or\_factors }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(df, type)\{}
\NormalTok{  df }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{pivot\_longer}\NormalTok{(}\AttributeTok{cols =} \FunctionTok{everything}\NormalTok{(),}
                 \AttributeTok{names\_to =} \StringTok{"variable"}\NormalTok{, }\AttributeTok{values\_to =} \StringTok{"value"}\NormalTok{) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{group\_by}\NormalTok{(variable) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{summarise}\NormalTok{(}\FunctionTok{across}\NormalTok{(}\FunctionTok{everything}\NormalTok{(),}
                     \FunctionTok{funs}\NormalTok{(}\AttributeTok{mode =}\NormalTok{ brotools}\SpecialCharTok{::}\FunctionTok{sample\_mode}\NormalTok{(value, }\AttributeTok{na.rm =} \ConstantTok{TRUE}\NormalTok{),}
                     \AttributeTok{nobs =} \FunctionTok{length}\NormalTok{(value),}
                     \AttributeTok{n\_missing =} \FunctionTok{sum}\NormalTok{(}\FunctionTok{is.na}\NormalTok{(value)),}
                     \AttributeTok{n\_unique =} \FunctionTok{length}\NormalTok{(}\FunctionTok{unique}\NormalTok{(value))))) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{mutate}\NormalTok{(}\AttributeTok{type =}\NormalTok{ type)}
\NormalTok{\}}

\CommentTok{\#\textquotesingle{} Compute descriptive statistics for the character columns of a data frame.}
\CommentTok{\#\textquotesingle{} @param df The data frame to summarise.}
\CommentTok{\#\textquotesingle{} @return A data frame with a description of the character columns.}
\CommentTok{\#\textquotesingle{} @import dplyr}
\CommentTok{\#\textquotesingle{} @export}
\CommentTok{\#\textquotesingle{} @examples}
\CommentTok{\#\textquotesingle{} \textbackslash{}dontrun\{}
\CommentTok{\#\textquotesingle{} describe(dataset)}
\CommentTok{\#\textquotesingle{} \}}
\NormalTok{describe\_character }\OtherTok{\textless{}{-}} \ControlFlowTok{function}\NormalTok{(df)\{}
\NormalTok{  df }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{select}\NormalTok{(}\FunctionTok{where}\NormalTok{(is.character)) }\SpecialCharTok{\%\textgreater{}\%}
    \FunctionTok{describe\_character\_or\_factors}\NormalTok{(}\AttributeTok{type =} \StringTok{"Character"}\NormalTok{)}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

Let's now continue on to the next section, where we will learn to document the package.

\hypertarget{documenting-your-package}{%
\subsection{Documenting your package}\label{documenting-your-package}}

There are several files that you must edit to fully document the package; for now, only the functions
are documented. The first of these files is the \texttt{DESCRIPTION} file.

\hypertarget{description}{%
\subsubsection{Description}\label{description}}

By default, the \texttt{DESCRIPTION} file, which you can find in the root of your package project, contains
the following lines:

\begin{verbatim}
Package: arcade
Type: Package
Title: What the Package Does (Title Case)
Version: 0.1.0
Author: Who wrote it
Maintainer: The package maintainer <yourself@somewhere.net>
Description: More about what it does (maybe more than one line)
    Use four spaces when indenting paragraphs within the Description.
License: What license is it under?
Encoding: UTF-8
LazyData: true
RoxygenNote: 7.0.2
\end{verbatim}

Each section is quite self-explanatory. This is how it could look like once you're done editing it:

\begin{verbatim}
Package: arcade
Type: Package
Title: List of highest-grossing Arcade Games
Version: 0.1.0
Author: person("Harold", "Zurcher", email = "harold.zurcher@madisonbus.com", role = c("aut", "cre"))
Description: This package contains data about the highest-grossing arcade games from the 70's until
    2010's. Also contains some functions to summarize data.
License: CC0
Encoding: UTF-8
LazyData: true
RoxygenNote: 7.0.2
\end{verbatim}

The \texttt{Author} and \texttt{Maintainer} need some further explanations; I have added Harold Zurcher as
the athor and creator, with the \texttt{role\ =\ c("aut",\ "cre")} bit. \texttt{"cre"} can also be used for
maintainer, so I removed the \texttt{Maintainer} line.

\hypertarget{unit-testing-your-package}{%
\subsection{Unit testing your package}\label{unit-testing-your-package}}

\hypertarget{further-topics}{%
\section{Further topics}\label{further-topics}}

This chapter is a collection of short section that show some of the very nice things you can use
R for. These sections are based on past blog posts.

\hypertarget{using-python-from-r-with-reticulate}{%
\subsection{\texorpdfstring{Using Python from R with \texttt{\{reticulate\}}}{Using Python from R with \{reticulate\}}}\label{using-python-from-r-with-reticulate}}

There is a lot of discussion online about the benefits of Python over and vice-versa. When it comes
to data science, they are for the most part interchangeable. I would say that R has an advantage
over Python when it comes to offering specialized packages for certain topics such as
econometrics, bioinformatics, actuarial sciences, etc\ldots{} while Python seems to offer more possibilities
when it comes to integrating a machine learning model into an app.
However, if most of your work is data analysis/machine learning, both languages are practically
interchangeable. But it can happen that you need access to a very specific library with no R
equivalent. Well, in that case, no need to completely switch to Python, as you can call Python code
from R using the \texttt{\{reticulate\}} package.

\texttt{\{reticulate\}} allows you to seamlessly call Python functions from an R session. An easy way to use
\texttt{\{reticulate\}} is to start a a new notebook, but you can also use \texttt{\{reticulate\}} and the included
functions interactively. However, I find that Rstudio notebooks work very well for this particular
use-case, because you can write R and Python chunks, and thus differentiate the different
specific lines of code really well.

Let's see how this works. First of all, you might need to specify the path to your Python executable,
in my case, because I've installed Python using Anaconda, I need to specify it:

\begin{verbatim}
# This is an R chunk
use_python("~/miniconda3/bin/python")
\end{verbatim}

\hypertarget{generating-pdf-or-word-reports-with-r}{%
\subsection{Generating Pdf or Word reports with R}\label{generating-pdf-or-word-reports-with-r}}

\hypertarget{scraping-the-internet}{%
\subsection{Scraping the internet}\label{scraping-the-internet}}

\hypertarget{regular-expressions}{%
\subsection{Regular expressions}\label{regular-expressions}}

\hypertarget{setting-up-a-blog-with-blogdown}{%
\subsection{\texorpdfstring{Setting up a blog with \texttt{\{blogdown\}}}{Setting up a blog with \{blogdown\}}}\label{setting-up-a-blog-with-blogdown}}

  \bibliography{book.bib,packages.bib}

\end{document}