RPR-Literate programming

Literate Programming with R

 

Keywords:  (Draft) Literate programming principles; R Markdown; R Notebooks

 

 

Caution!

This unit is under development. There is some contents here but it is incomplete and/or may change significantly: links may lead to nowhere, the contents is likely going to be rearranged, and objectives, deliverables etc. may be incomplete or missing. Do not work with this material until it is updated to "live" status.

 

Abstract

Documentation of results using R markdown and R notebooks. Example needs debugging since the OED site has changed.

 

This unit ...

Prerequisites

• RPR-Introduction (Introduction to R)

 

Objectives

This unit will ...

• ... introduce the philosophy behind "Literate Programming";
• ... teach the practice with an example that uses the R knitr package;
• ... demonstrate R notebooks;

 

Outcomes

After working through this unit you ...

• ... can produce your own "Literate Programs" with knitr or in an R notebook.

 

Deliverables

• Time management: Before you begin, estimate how long it will take you to complete this unit. Then, record in your course journal: the number of hours you estimated, the number of hours you worked on the unit, and the amount of time that passed between start and completion of this unit.
• Journal: Document your progress in your Course Journal. Some tasks may ask you to include specific items in your journal. Don't overlook these.
• Insights: If you find something particularly noteworthy about this unit, make a note in your insights! page.

 

Evaluation

Evaluation: NA

This unit is not evaluated for course marks.

 

Contents

Literate programming is an idea that software is best described in a natural language, focussing on the logic of the program, i.e. the why of code, not the what. The goal is to ensure that model, code, and documentation become a single unit, and that all this information is stored in one and only one location. The product should be consistent between its described goals and its implementation, seamless in capturing the process from start (data input) to end (visualization, interpretation), and reversible (between analysis, design and implementation).

In literate programming, narrative and computer code are kept in the same file. This source document is typically written in Markdown or LaTeX syntax and includes the programming code as well as text annotations, tables, formulas etc. The supporting software can weave human-readable documentation from this, or tangle executable code. Literate programming with both Markdown and LaTex is supported by R Studio and this makes the R Studio interface a useful development environment for this paradigm. While it is easy to edit source files with a different editor and process files in base R after loading the Sweave() and Stangle() functions or the knitr package. In our context here we will use R Studio because it conveniently integrates the functionality we need.

knitr is an R package for literate programming. It is integrated with R Studio.

 

RMarkdown

Markdown is an extremely simple and informal way of structuring documents that is useful if for some reason you feel html is too complicated. That's really all it does: format documents in a simple way so they can be displayed as Web pages. For Markdown documentation, see here.. The concept is quite similar to Wiki markup syntax, the syntax is (regrettably) different, and for a number of features there there are (regrettably) several different ways to achieve the same results.

RMarkdown is an R package that is integrated with R Studio and allows integrating R code with Markdown documents. knitr can work with Markdown files, and this gives additional output options, such as PDF and MSWord documents.

Let's give it a try: we'll write and document an R function that will find us a random phobia to ponder on.

Task:

• Open an R Studio session.
• Select Session → Set Working Directory → Choose Directory... and choose some project directory.
  • Note that there is a bug in R Studio that will prevent the knitr interface from working correctly if your home directory contains an .Rprofile file that issues a setwd() command to a directory other than your project directory. If you run into an error when weaving your file, remove any setwd() command you might find in such a profile.
• Select File → New File → RMarkdown.... When you do this the first time, R Studio will ask you whether you want to install/update a number of required packages. Click Yes.
• Enter "Random Phobia" as the Title and your name as the Author, select to create a Document, and check HTML as the default output option.

R Studio will load some default text and markup into the script pane which we can edit.

Let's introduce our plan: copy/paste the following text into the document.

##Phobias!
We all have some, but we could always use more. How to know them all? With this code we access the Oxford English Dictionary's Website - the most authoritative source on the English language, and scrape a list of phobias. A function is supplied to retrieve a random phobia, which we can subsequently ponder on - either to delight in the fact that we don't have that fear, or to add to our daily quota of anxieties <small>(like our well-founded [fear of bad programming practice](http://xkcd.com/292/))</small>.

To load the list, we will "screenscrape" a list of Phobias from the [OED Phobia list](http://www.oxforddictionaries.com/words/phobias-list). First, we load the XML library (or install it from CRAN, if we don't have it).

Note the following Markdown elements in the code:

- a tag <small>Text...</small> to set text to a smaller font size (we could have used <span style="font-size:85%;">Text...</span> instead because Markdown respects HTML elelements).

- a Web link [Text...](URL)added to text

• Click on the green question mark of the menu of your script pane. There is a link to an overview of RMarkdown use and to a quick reference. Load the quick reference (it will appear in the Help pane) and scan it.

• The filename in the script pane tab is red, because it contains unsaved changes. Save the file in your project directory, note that the extension .Rmd is automatically added.

Time to add our first bit of R code

• Copy and paste the following:

```{r loadLibrary}
if (!require(XML, quiet=TRUE)) {
  install.packages("XML")
  library(XML)
}
```

This is what is know as a "code chunk". It is delimited by three backticks ``` and has directives and options for the chunk in the first line. It is labelled as R code, and note that after the {r we have added an (optional) label for the chunk. That is useful, because we can rapidly navigate between chunks (click on the navigation menu at the bottom of the script pane), and we can refer to the labels to execute chunks that are coded later in the document at an earlier stage. This is an important idea of literal programming: the flow of the document should not be determined by the requirements of the code, but by the logic of the narrative. TLDR; label your chunks. It's useful.

Other options can be added after a comma, for example we can suppress printing of a chunk into the document altogether, if we think it is not relevant for the document, by adding the option echo=FALSE.

• To execute a particular chunk, simply place the cursor into the chunk and select Chunks → Run Current Chunk from the menu at the top of the script pane. Try this and check the console pane, the library should load without error.

• Let's add more text and code: copy and paste this into the document to add more comment and a second chunk.

The XML package provides a function -- `readHTMLTable()` -- that makes our life very easy: it accesses an URL, looks for all HTML formatted tables, parses them and returns them as lists. Internally, by default `readHTMLTable` reads the data into a dataframe, so to avoid converting all the text into factors we set the option `stringsAsFactors=FALSE`. There may be several tables in the source page, each one is returned as a list element. Since we know (hope?) the OED page contains only one table, we use only the first list element.

```{r getPageData, cache=TRUE}
phobiaFrame <- readHTMLTable("http://www.oxforddictionaries.com/words/phobias-list"
                             stringsAsFactors=FALSE)[[1]]
```

Two things to note here:

• Enclosing a piece of text in "backticks" `Text...` formats that text as "code" - typically in a fixed-width font.
• For this chunk we have set the option cache as TRUE. This is a very useful and well thought out mechanism that avoids recomputing code that takes a long time or should otherwise be limited. The results of a cached chunk of code are stored locally and retrieved when the file is weaved. Only if anything within the chunk is changed (or cache is set to FALSE), is the chunk evaluated again. This prevents us from excessively pounding on the OED as we develop our script, which is a question of good manners in the context of this example, but can save a lot of time as our projects become large and the calculations become complex.

In order to make sure everything has worked, we'll print a sample from the table to our documentation file. RMarkdown provides a shorthand notation for tables - just like Wiki markup. I never use these. HTML tables are easy enough to format and remember and they provide many more options. In the example below, we customize the row background-color for alternating rows. That is something we could not do with simple markdown.

• Paste the following:

**Table**: seven random phobias
```{r renderPhobiaTable, echo=FALSE, results='asis'}
cat("<table border=\"1\", width=\"50%\">\n")
cat("<tr style=\"background-color:#CCFFF0;\"><th>Phobia</th><th>Fear of...</th></tr>\n")
for (i in 1:7) {
  r <- randRow(phobiaFrame)
  if (i %% 2) {
    cat("<tr style=\"background-color:#F9F9F9;\">")
  }
  else {
    cat("<tr style=\"background-color:#EEFFF9;\">")
  }
  cat(paste("<td>", r[2], "</td><td>", r[1], "</td></tr>\n", sep=""))
}
cat("</table>\n")
```

This is now a mix of markup code and R code. Two important options in the chunk header:

• echo=FALSE prevents the contents of the chunk to be printed. We don't want this code in our output, we only want the result.
• results='asis' prevents the results from being marked up. The raw HTML is sent to the output document.

But note the following: this piece of code calls a function randRow(phobiaFrame) that we have not defined yet. In an R script this would not work. But in a knitr document we can reference a chunk of code anywhere in (and outside) of the document and thus define our function before the renderPhobiaTable chunk is executed. This is important for literate programming, where we don't want to be constrained by the requirements of the code.

Therefore, paste the following before the previous chunk:

```{r , ref.label="randRow", echo=FALSE}
```

This executes the code chunk with the label randRow without giving any output.

To finish off, paste the following:

<p>&nbsp;
<p>
To pick a single random phobia from the list, we take a (pseudo) random sample of size 1 from the number of rows in the `phobiaFrame` object. Our function thus returns a random row from a matrix or dataframe, and it uses an optional argument: `seed`. This can either be Boolean `FALSE` (the default), or an integer that is used in R's `set.seed()` function.

```{r randRow}
randRow <- function(M, seed=FALSE) {
  if (seed) set.seed(as.integer(seed))
  return(M[sample(1:nrow(M), 1),])
}
```

With this useful tool we can ponder on our favourite phobia of the day. For today, let it be **`r randRow(phobiaFrame, seed=1123581321)[2]`**, the fear of `r randRow(phobiaFrame, seed=1123581321)[1]`.

Reptiles! Awful.

This piece now contains the function definition for randRow, which it prints to the document after our comments. It also contains inline R code that is executed as the document is built.

• That should be all. You should be able to save the document and click the Knit HTML button to execute the code, build, and load a Webpage with the document we just wrote. Please get in touch if you run into problems.

If all the pasting of bits and chunks was confusing, the final .Rmd file is here.

 

R Notebooks

R Notebooks take the concpet into the RStudio editor itself, rather than constructing a Webpage. On one hand, you become dependent on the RStudio editor, on the other hand, you directly edit and comment as you are developing. This is true "Lietarte Programming".

Task:
Read about the concept here and follow along with the exercise.

 

 

Further reading, links and resources

 

Notes

 

Self-evaluation

 

 

---

 

If in doubt, ask! If anything about this learning unit is not clear to you, do not proceed blindly but ask for clarification. Post your question on the course mailing list: others are likely to have similar problems. Or send an email to your instructor.

---

 

About ...
 
Author:

Boris Steipe (boris.steipe@utoronto.ca)

Created:

2017-09-17

Modified:

2017-10-05

Version:

1.0

Version history:

• 1.0 First live version
• 0.1 First stub

---

This copyrighted material is licensed under a Creative Commons Attribution 4.0 International License. Follow the link to learn more.