Create Testing chapter

_quarto.yml

@@ -21,17 +21,18 @@ book:
     - chapters/issue_tracking.qmd
     - chapters/reporting_bugs.qmd
     - chapters/submitting_feature_requests.qmd
-    - chapters/reviewing_bugs.qmd                
-    - chapters/finding_the_source.qmd             
+    - chapters/reviewing_bugs.qmd
+    - chapters/finding_the_source.qmd
     - chapters/lifecycle_of_a_patch.qmd
-    - chapters/documenting.qmd                    
-    - chapters/message_translations.qmd         
+    - chapters/documenting.qmd
+    - chapters/message_translations.qmd
+    - chapters/testing_in_r.qmd
     - chapters/testing_pre_release_r_versions.qmd
-    - chapters/where_to_get_help.qmd              
-    - chapters/news_and_announcements.qmd  
-    - chapters/developer_tools.qmd    
-    - chapters/additional_resources.qmd 
-    - chapters/r_core_developers.qmd      
+    - chapters/where_to_get_help.qmd
+    - chapters/news_and_announcements.qmd
+    - chapters/developer_tools.qmd
+    - chapters/additional_resources.qmd
+    - chapters/r_core_developers.qmd
 
 format:
   html:

chapters/testing_in_r.qmd

@@ -0,0 +1,71 @@
+# Testing in R {#TestR}
+
+This chapter discusses about writing tests for R and extending the test suite for R. Usually, when one contributes a patch, one might also want to contribute tests that are able to cover the new code.
+
+## When and why to write a test?
+
+Whenever you add new functionality to the core code or any of the packages distributed with R, you may be asked to add test(s) corresponding to the new code. 
+
+If the new code requires GUI interaction or accesses the Internet, then it is essential to add its name to the `stop list` in `tests/no-segfault.Rin`.
+
+## Writing tests for R
+
+Writing tests for R is much like writing tests for your own code. Tests need to be thorough, fast, isolated, consistently repeatable, and as simple as possible. When you are adding tests to an existing test file, it is also recommended that you study the other tests in that file; it will give you an idea of which precautions you have to take to make your tests robust and portable. 
+
+Tests for base R live in the [`tests` directory of the R sources](https://svn.r-project.org/R/trunk/tests/). Contributors mainly contribute to the "regression tests" which are in the files `reg-tests-1a.R`, `reg-tests-1b.R`,  etc. These tests are added after fixing a bug, to ensure that future changes do not cause a regression that reintroduces the bug. If you have been asked to add tests for base functionality (something that is not in one of the add-on packages in the base distribution), you should add it to the latest regression test file, with the highest numbering (currently `reg-tests-3.R`). 
+
+## Test files
+
+Most tests in base R are `.R` files with R code implementing one of two types of test:
+
+1. Tests that produce an error on failure. These often involve a call to `stopifnot()`.
+2. Tests that produce output that is compared to reference output.
+
+If a test file `.R` includes tests that produce output for comparison, running the tests will create an `.Rout` file that should be compared to an `.Rout.save` file with the same name. The tests are usually run as part of the check commands `make check` (for a basic check of R) and `make check-all` or `make check-devel` (for a thorough check with/without the recommended packages). Alternatively, you can run a test file directly, for example if you have set the environment variable `TOP_SRCDIR` as the path for the top source directory:
+
+```sh
+cd $TOP_SRCDIR/tests
+R CMD BATCH reg-tests-3.R
+```
+
+will run `reg-tests-3.R` and create `reg-tests-3.Rout` in the same directory. If you have added a test or made a change that means you need to update the reference output. You can create or overwrite the reference output with, e.g. 
+
+```sh
+mv reg-tests-3.Rout reg-tests-3.Rout.save
+```
+
+Note that only substantive changes are important - the check will not fail due to differences in the version of R, the locale, or output from `proc.time()`, so if these are the only differences the `.Rout.save`  does not need to be updated.
+
+Sometimes test `.R` files are generated from a `.Rin` file, for example `no-segfault.Rin` generates tests for functions in the base add-on packages, calling each function with a range of inputs (character, logical, `NULL`, etc).
+
+## Tests of base R packages
+
+The check commands (`make check` etc) run `R CMD check` on the add-on packages in the base distribution, so they also will run any package-specific tests. If a package has tests, they will be in the `tests` directory of the sources for that package. Here is an example of the structure of the [R package `methods`](https://github.com/r-devel/r-svn/tree/main/src/library/methods) with the `tests` directory highlighted with yellow colour:
+
+![Screenshot of the structure of the R package `methods` with the `tests` directory highlighted with yellow colour](img/structure-methods.png)
+
+The base add-on packages use the same form of tests that are used for base R as described in the [Test files]{#test-files} section.
+
+When the package tests are run, the `tests` directory is copied to the check area, and the tests are run with the copy as the working directory and with `R_LIBS` set so that the copy of the package installed during testing will be found by `library(packagename)`.
+
+All the package-specific tests are run in a vanilla R session without setting the random-number seed. Hence, for tests which use random numbers one needs to set the seed to obtain reproducible results.
+
+## Checking R packages
+
+For more detail on the package checker `R CMD check` see the [Checking packages](https://cran.r-project.org/doc/manuals/r-devel/R-exts.html#Checking-packages-1) section of the Writing R Extensions manual.
+
+## Testing practices
+
+Tests for R should follow the general good practice documented in the [Writing R Extensions](https://cran.r-project.org/doc/manuals/r-devel/R-exts.html) manual. In particular:
+
+- Testing the time taken by a command is not reasonable. This is because, one cannot know how fast or how heavily loaded an R platform might be. At best one can test a ratio of times, however even that is difficult and not advisable.
+
+- The exact format of messages from R or from add-on packages can change and may be translated. Hence, it is not a good idea to test their exact format.
+
+- For tests comparing against reference output, use `options(warn = 1)` to print warnings as they occur or `options(warn = 2)` to turn off warnings. In the latter case `tools::assertWarning()` may be used to assert when a warning is expected, see for example [tests/d-p-q-r-tst-2.R](https://svn.r-project.org/R/trunk/tests/d-p-q-r-tests.R)
+
+## See also
+
+1. [Testing R Code](https://cran.r-project.org/doc/manuals/r-release/R-ints.html#Testing-R-code)
+
+2. [Writing R Extensions](https://cran.r-project.org/doc/manuals/r-release/R-exts.html)

chapters/testing_pre_release_r_versions.qmd

@@ -1,14 +1,14 @@
-# Testing Pre-release R Versions {#TestRVer}
+# Testing Pre-release R Versions
 
-This chapter is inspired from the blog on [testing R before release](https://blog.r-project.org/2021/04/28/r-can-use-your-help-testing-r-before-release/index.html) and discusses how you can help with testing of pre-release versions of R. 
+This chapter is inspired from the blog on [testing R before release](https://developer.r-project.org/Blog/public/2021/04/28/r-can-use-your-help-testing-r-before-release/index.html) and discusses how you can help with testing of pre-release versions of R. 
 
 ## Where to test?
 
 Whenever possible use a fresh package library for testing, even better would be to use [virtual machines](#VM) for the testing. This would ensure that you do not damage your existing R installation.
 
 ### Virtual machine {#VM}
 
-A free Windows 10 virtual machine is provided by Microsoft (with a 90-day limit) for building, testing, and checking R packages and R itself. Package maintainers who work on Linux and MacOS can use it to test their packages on Windows. Read the [instructions](https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/vm.html) on how to automatically set up the machine to check R packages. Tomas Kalibera describes the details of using virtual machine in the blog [Virtual Windows machine for checking R packages](https://blog.r-project.org/2021/03/18/virtual-windows-machine-for-checking-r-packages/index.html). 
+A free Windows 10 virtual machine is provided by Microsoft (with a 90-day limit) for building, testing, and checking R packages and R itself. Package maintainers who work on Linux and MacOS can use it to test their packages on Windows. Read the [instructions](https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/vm.html) on how to automatically set up the machine to check R packages. Tomas Kalibera describes the details of using virtual machine in the blog [Virtual Windows machine for checking R packages](https://developer.r-project.org/Blog/public/2021/03/18/virtual-windows-machine-for-checking-r-packages/index.html). 
 
 ## What can you test?
 
@@ -26,20 +26,10 @@ You can test:
 
 Details of performing testing on various operating systems:
 
-  * [Windows](https://blog.r-project.org/2021/04/28/r-can-use-your-help-testing-r-before-release/index.html#on-windows)
+  * [Windows](https://developer.r-project.org/Blog/public/2021/04/28/r-can-use-your-help-testing-r-before-release/index.html#on-windows)
 
-  * [macOS](https://blog.r-project.org/2021/04/28/r-can-use-your-help-testing-r-before-release/index.html#on-macos)
+  * [macOS](https://developer.r-project.org/Blog/public/2021/04/28/r-can-use-your-help-testing-r-before-release/index.html#on-macos)
 
-  * [Linux](https://blog.r-project.org/2021/04/28/r-can-use-your-help-testing-r-before-release/index.html#on-linux)
+  * [Linux](https://developer.r-project.org/Blog/public/2021/04/28/r-can-use-your-help-testing-r-before-release/index.html#on-linux)
 
-  * [Solaris](https://blog.r-project.org/2021/04/28/r-can-use-your-help-testing-r-before-release/index.html#on-solaris)
-
-## Writing tests for R
-
-Writing tests for R is much like writing tests for your own code. Tests need to be thorough, fast, isolated, consistently repeatable, and as simple as possible. 
-
-When you are adding tests to an existing test file, it is also recommended that you study the other tests in that file; it will teach you which precautions you have to take to make your tests robust and portable. We try to have tests both for normal behaviour and for error conditions. Tests live in the `tests` directory.
-
-## Benchmarks
-
-Benchmarking is useful to test that a change does not degrade performance.
+  * [Solaris](https://developer.r-project.org/Blog/public/2021/04/28/r-can-use-your-help-testing-r-before-release/index.html#on-solaris)