RPR-Literate programming

Literate Programming with R

((Draft) Literate programming principles; R Markdown; R Notebooks)

Abstract:

Documentation of results using R markdown and R notebooks.

Objectives:
This unit will ...

... introduce the philosophy behind "Literate Programming";
... teach the practice with an example that uses knitr in the RStuio environment;
... point you to R notebooks;

Outcomes:
After working through this unit you ...

... can produce your own "Literate Programs" with knitr or in an R notebook.

Deliverables:

Time management: Before you begin, estimate how long it will take you to complete this unit. Then, record in your course journal: the number of hours you estimated, the number of hours you worked on the unit, and the amount of time that passed between start and completion of this unit.

Journal: Document your progress in your Course Journal. Some tasks may ask you to include specific items in your journal. Don't overlook these.

Insights: If you find something particularly noteworthy about this unit, make a note in your insights! page.

Prerequisites:

RPR-Introduction (Introduction to R)

This page is tagged for revision; expect changes and proceed with caution.

Literate programming is an idea that software is best described in a natural language, focussing on the logic of the program, i.e. the why of code, not the what. The goal is to ensure that model, code, and documentation become a single unit, and that all this information is stored in one and only one location. The product should be consistent between its described goals and its implementation, seamless in capturing the process from start (data input) to end (visualization, interpretation), and reversible (between analysis, design and implementation).

In literate programming, narrative and computer code are kept in the same file. This source document is typically written in Markdown or LaTeX syntax and includes the programming code as well as text annotations, tables, formulas etc. The supporting software can weave human-readable documentation from this, or tangle executable code. Literate programming with both Markdown and LaTex is supported by R Studio and this makes the R Studio interface a useful development environment for this paradigm. While it is easy to edit source files with a different editor and process files in base R after loading the Sweave() and Stangle() functions or the knitr package. In our context here we will use R Studio because it conveniently integrates the functionality we need.

knitr is an R package for literate programming. It is integrated with R Studio.

RMarkdown

Markdown is an extremely simple and informal way of structuring documents that is useful if for some reason you feel html is too complicated. That's really all it does: format documents in a simple way so they can be displayed as Web pages. For Markdown documentation, see here.. The concept is quite similar to Wiki markup syntax, the syntax is (regrettably) different, and for a number of features there there are (regrettably) several different ways to achieve the same results.

RMarkdown is an R package that is integrated with R Studio and allows integrating R code with Markdown documents. knitr can work with Markdown files, and this gives additional output options, such as PDF and MSWord documents.

Let's give it a try: we'll write and document an R function that will find us a random phobia to ponder on.

Task:

Open an R Studio session.
Select Session → Set Working Directory → Choose Directory... and choose some project directory.
- Note that there is a bug in R Studio that will prevent the knitr interface from working correctly if your home directory contains an .Rprofile file that issues a setwd() command to a directory other than your project directory. If you run into an error when weaving your file, remove any setwd() command you might find in such a profile.
Select File → New File → RMarkdown.... When you do this the first time, R Studio will ask you whether you want to install/update a number of required packages. Click Yes.
Enter "Random Phobia" as the Title and your name as the Author, select to create a Document, and check HTML as the default output option.

R Studio will load some default text and markup into the script pane which we can edit.

Choose Help → Cheatssheets → R Markdown Cheat Sheet and R Markdown Reference Guide to download two PDFs via your browser. Browse the contents to get an idea where you can clarify concepts as you go through this example.

Let's introduce our plan: copy/paste the following text into the document to replace the two sections with the headers ## R Markdown and ## Including Plots.

##Phobias!
We all have some, but we could always use more. How to know them all? With this code we access the Oxford English Dictionary's Website - the most authoritative source on the English language, and scrape a list of phobias. A function is supplied to retrieve a random phobia, which we can subsequently ponder on - either to delight in the fact that we don't have that fear, or to add to our daily quota of anxieties (like our well-founded [fear of bad programming practice](http://xkcd.com/292/)).

To load the list, we will "screenscrape" a list of Phobias from the [OED Phobia list](https://en.oxforddictionaries.com/explore/phobias-list). First, we install the `rvest` and `xml2` packages (or install them from CRAN, if we don't have them).

Note the following Markdown elements in the code:

- a tag Text... to set text to a smaller font size (we could have used Text... instead because Markdown respects HTML elelements);

- a Web link [Text...](URL)added to text;

- backticks that cause text to be rendered as "code".

The filename in the script pane tab (Untitled1) is red, because the file contains unsaved changes. Save the file in your project directory under the nameRandomPhobia, note that the extension .Rmd is automatically added.

Now click on the wool and knitting needle icon in the menu bar to compile this code to HTML and view the current state in the viewer pane. See how the elements are rendered.

Then it's time to add our first bit of R code

Copy and paste the following:

```{r loadLibrary}
if (! requireNamespace("rvest", quietly=TRUE)) {
  install.packages("rvest")
}
if (! requireNamespace("xml2", quietly=TRUE)) {
  install.packages("xml2")
}
```

This is what is know as a "code chunk". It is delimited by three backticks ``` and has directives and options for the chunk in the first line. It is labelled as R code, and note that after the {r we have added an (optional) label for the chunk. That is useful, because we can rapidly navigate between chunks (click on the navigation menu at the bottom of the script pane), and we can refer to the labels to execute chunks that are coded later in the document at an earlier stage. This is an important idea of literate programming: the flow of the document should not be determined by the requirements of the code, but by the logic of the narrative. TLDR; label your chunks. It's useful.

Other options can be added after a comma, for example we can suppress printing of a chunk into the document altogether, if we think it is not relevant for the document, by adding the option echo=FALSE^[1].

To execute a particular chunk, simply place the cursor into the chunk and select Chunks → Run Current Chunk from the menu at the top of the script pane. Try this and check the console pane, the library should load without error.

Let's add more text and code: copy and paste this into the document to add more comment and a second chunk.

Retrieving the HTML source of a Web document is done with `xml2::read_html()`. The `rvest` package is designed for screenscraping and has functions to make our life very easy: it can find all HTML formatted tables in an HTML document, parse them with an XPATH expression and return them as lists from which we can get data frames. There may be several tables in the source page, each one is returned as a list element. Since we know (hope?) the OED page contains only one table, we use only the first list element.

```{r getPageData, cache=TRUE}
myURL <- "https://en.oxforddictionaries.com/explore/phobias-list"
phobias <- xml2::read_html(myURL)
phobias <- rvest::html_nodes(phobias, xpath = '//*[@id="content"]/div[1]/div[2]/div/div/div/div/div[4]/table')
phobias <- html_table(phobias)[[1]]
```

Some things to note here:

Enclosing a piece of text in "backticks" `Text...` formats that text as "code" - typically in a fixed-width font.
For this chunk we have set the option cache as TRUE. This is a very useful and well thought out mechanism that avoids recomputing code that takes a long time or should otherwise be limited. The results of a cached chunk of code are stored locally and retrieved when the file is weaved. Only if anything within the chunk is changed (or cache is set to FALSE), is the chunk evaluated again. This prevents us from excessively pounding on the OED as we develop our script, which is a question of good manners in the context of this example, but can save a lot of time as our projects become large and the calculations become complex.
rvest needs an XPATH expression to parse the document. Writing XPATH expressions can be a bit gnarly - the RBloggers article linked from the Further Reading section demonstrates a nifty way to get the expression from within a Chrome browser window. The interface has slightly changed since the article was written, but it's easy enough to figure out.
If you examine the phobias dataframe, you may note that the columns are named x1 and x2, and that the column headers are in the first row of the dataframe. This is because the OED authors did not put their column headers in <th> tags, so html_table() assumed they are part of the contents. You should fix this, use the first row as columnames and remove the first row. If you don't know how, you can peek below^[2].

In order to make sure everything has worked, we'll print a sample from the table to our documentation file. RMarkdown provides a shorthand notation for tables - just like Wiki markup. I never use these. HTML tables are easy enough to format and remember and they provide many more options. In the example below, we customize the row background-color for alternating rows. That is something we could not do with simple markdown.

Paste the following:

**Table**: seven random phobias
```{r renderPhobiaTable, echo=FALSE, results='asis'}
cat("<table border=\"1\", width=\"50%\">\n")
cat("<tr style=\"background-color:#CCFFF0;\"><th>Phobia</th><th>Fear of...</th></tr>\n")
for (i in 1:7) {
  r <- randRow(phobias)
  if (i %% 2) {
    cat("<tr style=\"background-color:#F9F9F9;\">")
  }
  else {
    cat("<tr style=\"background-color:#EEFFF9;\">")
  }
  cat(paste("<td>", r[2], "</td><td>", r[1], "</td></tr>\n", sep=""))
}
cat("</table>\n")
```

This is now a mix of markup code and R code. Two important options in the chunk header:

echo=FALSE prevents the contents of the chunk to be printed. We don't want this code in our output, we only want the result.
results='asis' prevents the results from being marked up. The raw HTML is sent to the output document.

But note the following: this piece of code calls a function randRow(phobiaFrame) that we have not defined yet. In an R script this would not work. But in a knitr document we can reference a chunk of code anywhere in (and outside) of the document and thus define our function before the renderPhobiaTable chunk is executed. This is important for literate programming, where we don't want to be constrained by the requirements of the code.

Therefore, paste the following before the previous chunk:

```{r , ref.label="randRow", echo=FALSE}
```

This executes the code chunk with the label randRow (and - you guessed it - the function will be defined in that chunk) without giving any output.

To finish off, paste the following:

To pick a single random phobia from the list, we take a (pseudo) random sample of size 1 from the number of rows in the `phobiaFrame` object. Our function thus returns a random row from a matrix or dataframe, and it uses an optional argument: `seed`. This can either be Boolean `FALSE` (the default), or an integer that is used in R's `set.seed()` function.

```{r randRow}
randRow <- function(M, seed = FALSE) {
  # Return a random row from a dataframe M.
  if (seed) {
    set.seed(as.integer(seed))
  }
  return(M[sample(1:nrow(M), 1), ])
}
```

With this useful tool we can ponder on our favourite phobia of the day. For today, let it be **`r randRow(phobias, seed=1123581321)[2]`**, the fear of `r randRow(phobias, seed=1123581321)[1]`.

Reptiles! Awful.

This piece now contains the function definition for randRow, which it prints to the document after our comments. It also contains inline R code that is executed as the document is built.

That should be all. You should be able to save the document and select (from the menu bar of the script pane) Knit → Knit to HTML to execute the code, build, and load a Webpage with the document we just wrote. If your code has errors in the chunks, they will be reported in the console.

If all the pasting of bits and chunks was confusing, the final .Rmd file is here.

R Notebooks

R Notebooks take the concpet into the RStudio editor itself, rather than constructing a Webpage. On one hand, you become dependent on the RStudio editor, on the other hand, you directly edit and comment as you are developing. This is true "Literate Programming".

Task:
Read about the concept here and follow along with the exercise.

Self-evaluation

Question 1

Add a histogram to the code that prints the length distribution of the phobia names. (Hint: use the nchar() function.)

Answer ...

```{r showHist}

hist(nchar(phobias[ , 1]),
     main = "Length of phobia-names",
     xlab = "number of letters",
     ylab = "counts",
     col="paleturquoise")

```

Done.

Notes

↑ For a complete list of chunk options, see the documentation by knitr's author, Xie Yihui.

↑ Add the follwing bit into the getPageData chunk:

colnames(phobias) <- phobias[1, ]
phobias <- phobias[-1, ]

About ...

Author:

Boris Steipe (boris.steipe@utoronto.ca)

Created:

2017-09-17

Modified:

2019-01-07

Version:

1.2

Version history:

1.2 Change from require() to requireNamespace() and use <package>::<function>() idiom.
1.1 bugfix, comment on header tags in a table, add an eval question
1.0 First live version
0.1 First stub

This copyrighted material is licensed under a Creative Commons Attribution 4.0 International License. Follow the link to learn more.

This page is tagged for revision; expect changes and proceed with caution.

[1] For a complete list of chunk options, see the documentation by knitr's author, Xie Yihui.

[2] Add the follwing bit into the getPageData chunk:
colnames(phobias) <- phobias[1, ] phobias <- phobias[-1, ]

[1]

[2]

RPR-Literate programming

Contents

Evaluation

Contents

RMarkdown

R Notebooks

Self-evaluation

Question 1

Further reading, links and resources

Notes

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Sections

Tools