API

You’ve experienced an HTTP redirect (or URL redirect, or URL forwarding) even if you haven’t noticed. We all use browsers (I assume, since you are reading this), either on a phone or laptop/desktop computer. Browsers don’t show all the HTTP requests going on in the background, some of which are redirects. Redirection is used for various reasons, including to prevent broken links when web pages are moved, for privacy protection, to allow multiple domains to refer to a single web page, and more. ...

Acquiring full text articles fulltext is an R package I maintain to obtain full text versions of research articles for text mining. It’s a hard problem, with a spaghetti web of code. One of the hard problems is figuring out what the URL is for the full text version of an article. Publishers do not have consistent URL patterns through time, and so you can not set rules once and never revisit them. ...

I was listening to a Bike Shed podcast episode 189, “It’s Gonna Work, Definitely, No Problems Whatsoever”, and starting at 27:44 there was a conversation about exception handling. Specifically it was about exception handling in control flow when doing web API requests. This topic piqued my interest straight away as I do a lot of API stuff (making and wrapping). The part of the conversation that I want to address is their conclusion that exceptions in control flow are an anti-pattern. Seems this is a general pattern in programming languages, e.g., this SO thread. But on the contrary there are some languages in which exceptions in control flow are considered normal behavior; e.g., Python (this, this). ...

The problem The R community has a package distribution thing called CRAN just like Ruby has Rubygems, and Python has Pypi, etc. On all packages on CRAN, the CRAN maintainers run checks on each package on multiple versions of R and on many operating systems. They report those results on a page associated with the package, like this one. You might be thinking: okay, but we have Travis-CI and friends, so who cares about that? Well, it’s these checks that CRAN runs that will determine if your package on CRAN leads to emails to you asking for changes, and possibly the package being taken down if e.g., they email and you don’t respond for a period of time. ...

If you maintain an R package, or even use R packages, you may have looked at CRAN check results. These are essentially the results of running R CMD CHECK on a package. They do these for each package for each of a few different operating systems (debian, fedora, solaris, windows, osx) and different R versions (devel, release and patched). src: https://github.com/ropensci/cchecksapi base api url: https://cranchecks.info CRAN maintainers look at these, and eventually will email maintainers if checks are bad enough. ...

The USDA maintains a database of plant information, some of it trait data, some of it life history. Check it out at https://plants.usda.gov/java/ They’ve been talking about releasing an API for a long time, but have not done so. Thus, since at least some version of their data is in the public web, I’ve created a RESTful API for the data: source code: https://github.com/sckott/usdaplantsapi/ base URL: https://plantsdb.xyz Check out the API, and open issues for bugs/feature requests in the github repo. ...

GBIDS API is back Back in March this year I wrote a post about a new API for working with GenBank IDs. I had to take the API down because it was too expensive to keep up. Expensive because the dump of data is very large (3.8 GB compressed), and I need disk space on the server to uncompress that to I think about 18 GB, then load into MySQL, which is another maybe 30 GB or so. Anyway, it’s not expensive because of high traffic - although I wish that was the case - but because of needing lots of disk space. ...

GenBank IDs, accession numbers and GI identifiers, are the two types of identifiers for entries in GenBank. (see this page for why there are two types of identifiers). Actually, recent news from NCBI is that GI identifiers will be phased out by September this year, which affects what I’ll talk about below. There are a lot of sequences in GenBank. Sometimes you have identifiers and you want to check if they exist in GenBank, or want to get one type from another (accession from GI, or vice versa; although GI phase out will make this use case no longer needed), or just get a bunch of identifiers for software testing purposes perhaps. ...

request is DSL for http requests for R, and is inspired by the CLI tool httpie. It’s built on httr. The following were driving principles for this package: The web is increasingly a JSON world, so we assume applications/json by default, but give back other types if not The workflow follows logically, or at least should, from, hey, I got this url, to i need to add some options, to execute request - and functions support piping so that you can execute functions in this order Whenever possible, we transform output to data.frame’s - facilitating downstream manipulation via dplyr, etc. We do GET requests by default. Specify a different type if you don’t want GET. Given GET by default, this client is optimized for consumption of data, rather than creating new things on servers You can use non-standard evaluation to easily pass in query parameters without worrying about &’s, URL escaping, etc. (see api_query()) Same for body params (see api_body()) The following is a brief demo of some of the package functionality: ...

I’ve recently made some improvements to the functions that work with ISD (Integrated Surface Database) data. isd data The isd() function now caches more intelligently. We now cache using .rds files via saveRDS/readRDS, whereas we used to use .csv files, which take up much more disk space, and we have to worry about not changing data formats on reading data back into an R session. This has the downside that you can’t just go directly to open up a cached file in your favorite spreadsheet viewer, but you can do that manually after reading in to R. In addition, isd() now has a function cleanup, if TRUE after downloading the data file from NOAA’s ftp server and processing, we delete the file. That’s fine since we have the cached processed file. But you can choose not to cleanup the original data files. Data processing in isd() is improved as well. We convert key variables to appropriate classes to be more useful. isd stations ...