Expected Preparations:
|
|||||||||||
|
|||||||||||
Keywords: Techniques for accessing databases and downloading data | |||||||||||
|
|||||||||||
Objectives:
This unit will …
|
Outcomes:
After working through this unit you …
|
||||||||||
|
|||||||||||
Deliverables: Time management: Before you begin, estimate how long it will take you to complete this unit. Then, record in your course journal: the number of hours you estimated, the number of hours you worked on the unit, and the amount of time that passed between start and completion of this unit. Journal: Document your progress in your Course Journal. Some tasks may ask you to include specific items in your journal. Don’t overlook these. Insights: If you find something particularly noteworthy about this unit, make a note in your insights! page. |
|||||||||||
|
|||||||||||
Evaluation: NA: This unit is not evaluated for course marks. |
Often we need to automate access to databases that provide their data via Web interfaces, or are only designed to be viewed in Web browsers. This unit discussess three strategies: working with text data that is accessed through GET and POST commands, and parsing simple XML formatted data.
Many databases provide download links of their holdings, and/or covenient subsets of data, but sometimes we need to access the data piece by piece - either because no bulk download is available, or because the full dataset is unreasonably large. For the odd protein here or there, we may be able to get the information from a Web-page by hand, but this is tedious, and it is easy to make mistakes. Much better to learn how to script data downloads.
In this unit we will cover three download strategies. Our first example is the UniProt interface from which we will retrieve the FASTA sequence of a protein with a simple GET request for a text file. The second example is to retieve motif annotations from PROSITE - a POST request, with subsequent parsing of a table. The final example is to retrieve XML data from the NCBI via their E-utils interface.
Task…
ABC-units
R project. If you
have loaded it before, choose File ▸ Recent
projects ▸ ABC-Units. If you have not loaded
it before, follow the instructions in the RPR-Introduction
unit.init()
if requested.RPR-UniProt_GET.R
and follow the
instructions.
Note: take care that you understand all of the code in the script. Evaluation in this course is cumulative and you may be asked to explain any part of code.
ScanProsite is a tool to search for the occurrence of expert-curated motifs in the PROSITE database in a sequence of interest.
Task…
ScanProsite uses UniProt IDs. The UniProt ID for yeast Mbp1 is
P39678
.
P39678
into the text field, select Table
output from the dropdown menu in the STEP 3 section, and START
THE SCAN.You should see four feature hits: the APSES domain, and three ankyrin
domain sequences that partially overlap. We could copy and paste the
start and end numbers and IDs but that would be lame. Let’s get them
directly from Prosite instead, because later we will want to fetch a few
of these annotations. Prosite does not have a nice API interface like
UniProt, but the principles of using R’s
httr
package to send POST requests and retrieve the results
are the same. The parameters for the POST request are hidden in the
so-called “form” element” that your browser sends to the PROSITE Web
server. In order to construct our request correctly, we need to use the
correct parameter names in our script, that the Web page assigns when it
constructs input. The first step to capture the data from this page via
screenscraping is to look into the HTML code of the page.
(I am writing this section from the perspective of the Chrome browser - I don’t think other browsers have all of the functionality that I am describing here. You may need to install Chrome to try this…)
Use the menu and access View ▸ Developer ▸ View Source. Scroll through the page. You should easily be able to identify the data table. That’s fair enough: each of the lines contain the UniProt ID and we should be able to identify them. But how to send the request to get this page in the first place?
Use the browser’s back button to go back to the original query
form, and again: View ▸ Developer ▸
View Source. This is the page that accepts user input
in a so called form
via several different types of
elements: “radio-buttons”, a “text-box”, “check-boxes”, a “drop down
menu” and a “submit” button. We need to figure out what each of the
values are so that we can construct a valid POST
request.
If we get them wrong, in the wrong order, or have parts missing, it is
likely that the server will simply ignore our request. These elements
are harder to identify than the lines of feature information, and it’s
really easy to get them wrong, miss something and get no output. But
Chrome has a great tool to help us: it allows you to see the exact,
assembled POST
header that it sent to the Prosite
server!
PSScan.cgi
. This contains the form data. Then click
on the Headers tab and scroll down until you see the
Form Data. This has all the the required
POST
elements nicely spelled out. What you are looking for
are key value pairs like:meta: opt1
opt1
P39678
These are the field keys, and the required values. You have now reverse-engineered a Web form. Armed with this knowledge we can script it: what worked from the browser should work the same way from an R script.
Task…
ABC-units
R project. If you
have loaded it before, choose File ▸ Recent
projects ▸ ABC-Units. If you have not loaded
it before, follow the instructions in the RPR-Introduction
unit.init()
if requested.RPR-PROSITE_POST.R
and follow the
instructions.
Note: take care that you understand all of the code in the script. Evaluation in this course is cumulative and you may be asked to explain any part of code.
It is has become unreasonably difficult to screenscrape the NCBI site since the actual page contents are dynamically loaded via AJAX. This may be intentional, or just overengineering. While NCBI offers a subset of their data via the eutils API and that works well enough, some of the data that is available to the Web browser’s eyes is not served to a program.
The eutils API returns data in XML format. Have a look at the following URL in your browser to see what that looks like:
[http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=protein&<span style="color:#CC0000;">term=NP_010227</span>](http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=protein&term=NP_010227)
Look at the contents of the
<ID>...</ID>
tag, and follow the next
query:
[http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=protein&<span style="color:#CC0000;">id=6320147</span>&version=2.0](http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=protein&id=6320147&version=2.0)
Note the conceptual difference between search “term” and retrieval “id”.
An API to NCBIs Entrez system is provided through eUtils.
Task…
Browse through the E-utilities Quick Start chapter of the NCBI’s Entrez Programming Utilites Handbook for a quick overview.
Task…
ABC-units
R project. If you
have loaded it before, choose File ▸ Recent
projects ▸ ABC-Units. If you have not loaded
it before, follow the instructions in the RPR-Introduction
unit.init()
if requested.RPR-eUtils_XML.R
and follow the
instructions.
Note: take care that you understand all of the code in the script. Evaluation in this course is cumulative and you may be asked to explain any part of code.
If in doubt, ask! If anything about this contents is not clear to you, do not proceed but ask for clarification. If you have ideas about how to make this material better, let’s hear them. We are aiming to compile a list of FAQs for all learning units, and your contributions will count towards your participation marks.
Improve this page! If you have questions or comments, please post them on the Quercus Discussion board with a subject line that includes the name of the unit.
[END]