#+hugo_base_dir: ~/dev/blog #+hugo_section: ./posts #+hugo_weight: 2001 #+hugo_auto_set_lastmod: t #+title: Literate Programming against REST APIs #+date: <2019-02-02 Sat> #+author: Justin Barclay #+hugo_tags: emacs node ruby restclient twitter google #+hugo_categories: emacs #+hugo_draft: false * Rest Call Me Maybe #+BEGIN_QUOTE TL;DR Other HTTP Clients aren’t that great. Here we use Emacs and restclient, with public APIs, to identify plants and share on Twitter. Emacs and restclient offer a great user experience and workflow when documenting and exploring APIs. #+END_QUOTE Lately, I've spent a lot of time exploring web APIs. I've tried a couple of tools over the years: Postman back when it was a chrome extension, Curl, and HTTPie. They were all, ok — they got the job done — but they all ended up missing some feature or some UX. That's when I found [[https://github.com/pashky/restclient.el][restclient]], a package for Emacs with a simple DSL, and could convert the restclient request to an equivalent curl request. Let's take a look at an example of what restclient looks like when authenticating with Twitter. Here we will post our base64 encoded secrets to get a bearer token: #+BEGIN_SRC restclient POST https://api.twitter.com/oauth2/token Authorization: Basic QmRUTklkSEI5WWZHbVhwSkZ6NTZManpUNjpBMk1rOHJjaVdEWFdva3FCN1pmemZFdEk3WjRNd1lpM3JFSjhzN1JoVm9xMXhZY2pMbQ== Content-Type: application/x-www-form-urlencoded;charset=UTF-8 grant_type=client_credentials #+END_SRC Then, once you've POSTed the request a new buffer opens up with the response. #+NAME: restclient response #+begin_example { "token_type": "bearer", "access_token": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2FAAAAAAAAAAAAAAAAAAAA%3DAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" } // POST https://api.twitter.com/oauth2/token?grant_type=client_credentials // HTTP/1.1 200 OK // cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0 // content-disposition: attachment; filename=json.json // content-length: 155 // content-type: application/json;charset=utf-8 // date: Thu, 24 Jan 2019 05:54:09 GMT // expires: Tue, 31 Mar 1981 05:00:00 GMT // last-modified: Thu, 24 Jan 2019 05:54:09 GMT // ml: S // pragma: no-cache // server: tsa_a // status: 200 OK // strict-transport-security: max-age=631138519 // x-connection-hash: b93d89db0ee8f4ea90991c99c8d58449 // x-content-type-options: nosniff // x-frame-options: DENY // x-response-time: 20 // x-transaction: 005403e60014e9ee // x-twitter-response-tags: BouncerCompliant // x-ua-compatible: IE=edge,chrome=1 // x-xss-protection: 1; mode=block; report=https://twitter.com/i/xss_report // Request duration: 0.079710s #+end_example Using [[https://www.youtube.com/watch?v=fTvQTMOGJaw][restclient]] as my primary means to play with APIs worked for me for quite a while. It was a easy-to-read document, that housed all the requests in a single file, and lived alongside the rest of my code in git. However, I eventually ran into three complications when using it: 1. My restclient files would become a mess and navigating to find a specific endpoint or a set of endpoints was a pain. 2. I am lazy and dealing with authenticated APIs was a pain. - I'd need to enter my credentials each time I needed to get a new token - Once I authenticated, I would have to copy and paste that token from the response buffer into a variable, to use throughout the restclient file. 3. For exploring APIs I like to take notes, record responses, and record the input. There was a simple solution for my first and last points, which was to use [[https://www.youtube.com/watch?v=GK3fij-D1G8][org-mode]]. The power and flexibility of Emacs, and its suite of packages, constantly amazes me. I've been a long-time fan of org-mode. In my opinion it is a great replacement for Markdown and a number of other document formats in Emacs. I've found that org-mode is a great place to practice [[https://www.offerzen.com/blog/literate-programming-empower-your-writing-with-emacs-org-mode][literate programming]], but it's also a great place to an interactive style of programming, like a REPL or a Jupyter Notebook. Org-mode has a [[https://orgmode.org/worg/org-contrib/babel/][babel]] extension for [[https://github.com/alf/ob-restclient.el][restclient]]. Babel is an extension that runs code contained in SRC block: ~#+BEGIN_SRC...#+END_SRC~ and saves anything sent to STDOUT or value returned, and then can be used in other SRC code blocks. If you've never heard of org-mode, here's three key things to know before going forward: 1. Here is a [[https://www.youtube.com/watch?v=lsYdK0C2RvQ][short video]] exploring some of its features and here is a much [[https://www.youtube.com/watch?v=GK3fij-D1G8][longer video]] on literate programming in Emacs 2. This entire document was written using org-mode and babel 3. You can execute this document yourself, given you change your credentials ;) #+CAPTION: Tweeting using literate programming and Emacs [[./images/org-mode-low-quality.gif]] ** Follow along at home! If you're curious what this post looks like in org-mode you can find a copy [[https://raw.githubusercontent.com/justinbarclay/justinbarclay.me/master/content/posts/literate%5Fprogramming%5Fagainst%5Frest%5Fapis.org][here]]. Or, if you would like to run arbitrary elisp at home, you can download the following file in Emacs or your favourite [[http://spacemacs.org/][flavours]] of [[https://github.com/hlissner/doom-emacs][Vim]]. #+BEGIN_SRC elisp (browse-url-emacs "https://raw.githubusercontent.com/justinbarclay/justinbarclay.me/master/content/posts/literate_programming_against_rest_apis.org") #+END_SRC Of course, you need to go out and download restclient and ob-restclient and thats it, thats all the Emacs packages you need. Oh... also NodeJS and Ruby, don't @ me. Ok @ me, I'm lonely. * Twitter A good way to demonstrate the power of restclient and org-mode would be to post some tweets from Emacs. First, we need to be able to authenticate with Twitter. So, let’s see how we can use org-mode and restclient to authenticate with an OAuth endpoint. ** Helper functions First, we need to define a few functions that we are going to use during OAuth authentication. I could use a library or package for this, but when I am writing I become somewhat of a masochist. #+BEGIN_SRC emacs-lisp (defun twitter-signing-key (consumer-secret token-secret) "Creates a signing key by combining the consumer-secret and the token secret and percent encoding the result" (concat (url-encode-url consumer-secret) "&" (url-encode-url token-secret))) (defun twitter-signature-string (method base params) "Builds a hex encoded string of the format METHOF&BASE&PARAM1=VALUE1..." (let ((sorted-params (sort params (lambda (first second) (string< (car first) (car second)))))) (concat method "&" (url-hexify-string base) "&" (url-hexify-string (mapconcat (lambda (entry) (let ((key (car entry)) (value (cdr entry))) (concat (url-hexify-string key) "=" (url-hexify-string value)))) sorted-params "&"))))) (defun build-twitter-header-string (header oauth-headers) "Takes in a list of cons cells that represent HTTP headers, as well as the information needed to define the OAUTH response for a Twitter request, and build a restclient style header string" (concat "<<\n" (mapconcat (lambda (entry) (let ((key (car entry)) (value (cdr entry))) (concat key ": " value " "))) header "") "\nAuthorization: OAuth " (string-trim-right (mapconcat (lambda (entry) (let ((key (car entry)) (value (cdr entry))) (concat key "=" "\"" value "\"" ","))) oauth-headers " ") ",") "\n#")) #+END_SRC ** Shhh It's a Secret I don't need to store the authentication information in files, and I am to lazy to remember or copy and paste them! I can just use the information that is stored in my environment. #+NAME: twitter-consumer-key #+BEGIN_SRC bash echo $TWITTER_CONSUMER_KEY #+END_SRC #+NAME: twitter-consumer-secret #+BEGIN_SRC bash echo $TWITTER_CONSUMER_SECRET #+END_SRC #+NAME: twitter-access-token #+BEGIN_SRC bash echo $TWITTER_ACCESS_TOKEN #+END_SRC #+NAME: twitter-access-secret #+BEGIN_SRC bash echo $TWITTER_ACCESS_SECRET #+END_SRC ** Let's Auth it *** Step 1: Generate a body Before we can do all the fun authentication bits that is OAuth, we need to have some content. So, I feel like I need to be on brand for an Emacs user and let everyone know I am using Emacs for something that isn't editing text. #+NAME: hello-world #+BEGIN_SRC emacs-lisp (setq twitter-body (list (cons "status" "Hello world! I'm tweeting from Emacs"))) #+END_SRC *** Step 2: Generating some header data Ok, now that we have our Twitter status, we need to autogenerate a few more pieces of information; a nonce, a timestamp, and the signature. Emacs doesn't really have a built in crypto library, but do you know who does? Ruby! It is a fun language with a pretty full featured standard library; let's use it to generate our nonce. #+NAME: nonce #+BEGIN_SRC ruby require 'securerandom' nonce = SecureRandom.uuid nonce.gsub(/\W/, "") #+END_SRC Our request is going to need a time signature. #+NAME: oauth-time #+BEGIN_SRC emacs-lisp (format-time-string "%s") #+END_SRC We need to define the headers that we need for this request. #+NAME: twitter-headers #+BEGIN_SRC emacs-lisp (list (cons "Content-Type" "application/x-www-form-urlencoded")) #+END_SRC Did I mention Emacs built-in cryptography is kind of lacking? Well, we'll need to let another language do the heavy lifting when signing the request. I like Node and Node has a decent crypto library built into it. In the example below I am defining a code block as a function that I am going to call later and use it in an emacs-lisp source block. #+NAME: createSignature #+BEGIN_SRC js :var signature_string="" :var key="" let crypto = require('crypto') let createSignature = (key, text) => { return crypto.createHmac('sha1', key).update(signature_string).digest(); } return createSignature(key, signature_string).toString('base64'); #+END_SRC Now before we can sign anything, and we *do* need to sign things, we need to create a signing key. We can use our consumer-secret and our access-secret to build a Twitter signing key. #+NAME: signing-key #+BEGIN_SRC emacs-lisp :var consumer-secret=twitter-consumer-secret token-secret=twitter-access-secret (twitter-signing-key consumer-secret token-secret) #+END_SRC *** Step 3: Creating The Header Next up, we need to build the header, create a string to sign, sign that string, and then add that signature to our header. Simple. #+NAME: twitter-oauth-headers #+BEGIN_SRC emacs-lisp :var nonce=nonce consumer-key=twitter-consumer-key access-token=twitter-access-token oauth-time=oauth-time body=hello-world signing-key=signing-key (let* ((twitter-oauth-headers (list (cons "oauth_consumer_key" consumer-key) (cons "oauth_nonce" nonce) (cons "oauth_signature_method" "HMAC-SHA1") (cons "oauth_timestamp" oauth-time) (cons "oauth_token" access-token) (cons "oauth_version" "1.0"))) (signature-string (twitter-signature-string "POST" "https://api.twitter.com/1.1/statuses/update.json" (append twitter-oauth-headers twitter-body))) (signature (org-sbe createSignature (signature_string (eval signature-string)) (key (eval signing-key))))) ;; Here I am calling that signing function that I defined in NodeJS (append twitter-oauth-headers (list (cons "oauth_signature" (url-hexify-string signature))))) #+END_SRC *** Step 4: Encoding Data and Posting To Twitter Up next, our headers need to be in a string format that ~restclient~ knows how to read. #+NAME: twitter-restclient-headers #+BEGIN_SRC emacs-lisp :var header=twitter-headers twitter-oauth-headers=twitter-oauth-headers (build-twitter-header-string header (sort twitter-oauth-headers (lambda (first second) (string< (car first) (car second))))) #+END_SRC We need to encode our body as a post parameter string. #+NAME: twitter-post-body #+BEGIN_SRC emacs-lisp :var twitter-body=hello-world (setq twitter-post-body (concat "" (mapconcat (lambda (entry) (concat (car entry) "=" (url-hexify-string (cdr entry)))) twitter-body "&") "")) #+END_SRC Finally, now that we've done all that work to format and sign things, we can finish it off by tweeting to the world how much we love Emacs. #+BEGIN_SRC restclient :var twitter-headers=twitter-restclient-headers twitter-body=twitter-post-body # :body := (concat twitter-post-body) POST https://api.twitter.com/1.1/statuses/update.json?:body :twitter-headers #+END_SRC * Taking things to 11 [[./images/to_11.gif]] I think using org-mode and restclient to authenticate and post on Twitter is a little too mundane. Can we do anything more elaborate? Why, of course we can! This is Emacs, we pretty much have to do something overly complicated. I'm a big fan of science and I want to share my enthusiasm with the world. So, we're going to use our newly learned skills to talk across several APIs. We're going to: 1. Grab a plant name from [[https://www.trefle.io][Trefle]] 2. Find a picture of that plant, using [[https://cse.google.com/cse/][Google Custom Search]] 3. Make sure that the picture we have is of that plant, using [[https://cloud.google.com/vision/][Google Vision]] 4. Tag someone on Twitter and share the plant name and picture with them ** More Helper Functions We need a function to sanitize the response we get from restclient #+BEGIN_SRC emacs-lisp (defun sanitize-restclient-response (string) "Trim down a restclient response to JSON, removing the org source block and header information" (string-trim (replace-regexp-in-string "^#\\+BEGIN_SRC js\\|^#\\+END_SRC\\|^//[[:print:]]+" "" string))) #+END_SRC Let's be able to execute an arbitrary source code block #+BEGIN_SRC elisp (defun run-org-block (&optional code-block-name) (save-excursion (let ((code-block (or code-block-name (completing-read "Code Block: " (org-babel-src-block-names)))) (goto-char (org-babel-find-named-block code-block)) (org-babel-execute-src-block-maybe))))) #+END_SRC Here's a couple of functions we're going to use to help us parse a response from Google's API. #+BEGIN_SRC emacs-lisp (defun parse-ml-response (responses) "Extracts a Google AI response down to a list of label annotations" (let* ((json-response (json-read-from-string responses)) (label-annotations (cdr (assoc 'labelAnnotations (elt (cdr (assoc 'responses json-response)) 0))))) label-annotations)) (defun contains-description-p (annotations descriptions) "Checks to see if any of the items in the sequence ANNOTATIONS has a description that matches one of the items in DESCRIPTIONS" (let ((annotated-descriptions (mapcar (lambda (item) (cdr (assoc 'description item))) annotations))) (reduce (lambda (predicate description) (if predicate predicate (if (member (downcase description) descriptions) 't nil))) annotated-descriptions :initial-value nil))) #+END_SRC ** Harvesting a name Let's give our source block a name, ~#+NAME: trefle~, so we can easily reference it throughout the rest of our notebook. I am using my Mac's keychain to store and retrieve an access token I have stored for trefle.io. #+NAME: trefle #+BEGIN_SRC bash :results output security find-generic-password -gws trefle.io #+END_SRC To import a variable from earlier in the file you can use ~:var token=trefle~ where :var token specifies that you want to insert a variable called token into the proceeding block and the contents of that variable are pulled from a source code block named ~trefle~. Now we just need to build the HTTP headers we're going to use for our interaction with Trefle. #+NAME: trefle-headers #+BEGIN_SRC emacs-lisp :var token=trefle (concat "<< Content-Type: application/json Accept: application/json Authorization: Bearer " token) #+END_SRC As of the last time I looked, [[https://www.trefle.io][Trefle]] has over 4000 pages of plants, so we want to get a random plant off of a random page. So to start, we'll generate a page number from 0 to 4000... #+NAME: plants #+BEGIN_SRC restclient :var headers=trefle-headers :results value drawer # :page := (random 4000) GET https://trefle.io/api/plants?page=:page :headers # #+END_SRC Before we can do anything with the output we need to clean it up. Restclient likes to have all the headers for the response at the bottom of the buffer, so we need to filter those out of the response. #+NAME: sanitized-response #+BEGIN_SRC emacs-lisp :var response=plants (sanitize-restclient-response response) #+END_SRC Now we could use emacs-lisp, but everyone has Node installed and Node is pretty much built for parsing JSON, so it only makes sense to use that. We'll grab a random plant from the results and return its name. #+NAME: plant-name #+BEGIN_SRC js :var plants=sanitized-response :results value drawer let index = Math.floor(Math.random() * 30); return JSON.parse(plants)[index].scientific_name; #+END_SRC ** Imagine I need to get my Google API key. For this, I've been lazy and have just been storing it in my environment. #+NAME: google-api-key #+BEGIN_SRC bash echo $GOOGLE_API_KEY #+END_SRC We've got a plant name, now we need an image of the plant. #+NAME: google-images #+BEGIN_SRC restclient :var api-key=google-api-key plant-name=plant-name GET https://content.googleapis.com/customsearch/v1?cx=009341007550343915479%3Afg_hsgzltxw&q=:plant-name&searchType=image&key=:api-key #+END_SRC Much like our search for a plant name, we need to clean up the response from Google API so it's easily parsable as JSON. #+NAME: flower-images #+BEGIN_SRC emacs-lisp :var google-images=google-images (sanitize-restclient-response google-images) #+END_SRC We have a nice list of plant images. Let's play Google roulette and use the first image from the search. #+NAME: plant-image #+BEGIN_SRC js :var plant_images=flower-images :results value drawer return "" + JSON.parse(plant_images).items[0].link #+END_SRC ** What shall we (machine) learn today? When we're running our code we don't have time to make sure all it does what it is supposed to do and everyone knows you can't trust Google. Instead, we'll use machine learning provided by the fabulous Google Vision API to validate our choice. We'll ask Google for the top 3 labels for an image and see if those labels contain the words "Flower", "Plant", or "Tree". #+NAME: plant-ml-results #+BEGIN_SRC restclient :var api-key=google-api-key plant-image=plant-image POST https://vision.googleapis.com/v1/images:annotate?key=:api-key { "requests":[ { "image":{ "source":{ "imageUri": :plant-image } }, "features":[ { "type":"LABEL_DETECTION", "maxResults":3 } ] } ] } #+END_SRC Next, clean the response. #+NAME: sanitized-ml-results #+BEGIN_SRC emacs-lisp :var response=plant-ml-results (sanitize-restclient-response response) #+END_SRC If you're curious what talking to the Google Vision API looks like, here it is. #+NAME: google-vision-response #+begin_example json { "responses": [ { "labelAnnotations": [ { "mid": "/m/04_tb", "description": "map", "score": 0.9684097, "topicality": 0.9684097 }, { "mid": "/m/03scnj", "description": "line", "score": 0.734654, "topicality": 0.734654 }, { "mid": "/m/07j7r", "description": "tree", "score": 0.7276011, "topicality": 0.7276011 } ] } ] } #+end_example Let's check to see if the first three descriptors come back as plant, tree, or a flower. If it doesn't match these descriptors, then we rerun them code block below. When we rerun ths code block ~image-is-plant-p~ this will have a cascading effect and call all previous code blocks. So, we'll keep querying Trefle and Google until we find a pciture that meets our criteria. Warning: this could trap us into an infinite loop. #+NAME: image-is-plant-p #+BEGIN_SRC emacs-lisp :var response=sanitized-ml-results (while (not (contains-description-p (parse-ml-response response) '("plant" "tree" "flower"))) (org-sbe image-is-plant-p)) #+END_SRC ** A rose by any other name We need one last piece of information before we can demonstrate our love of plants to the world: someone to tweet at. Let's ask ourselves for some input. #+NAME: twitter-handle #+BEGIN_SRC emacs-lisp (read-string "What is the twitter handle of someone you want to tweet? ") #+END_SRC ** Content is king Now we need to [[https://www.youtube.com/watch?v=6BxAJNywkmg][build our body]] into something we can process later... #+NAME: twitter-plant-body #+BEGIN_SRC emacs-lisp :var plant_name=plant-name plant_image=plant-image twitter_handle=twitter-handle valid-image=image-is-plant-p (setq twitter-body (list (cons "status" (concat "" twitter_handle " " plant_name " " (replace-regexp-in-string "'" "" plant_image))))) #+END_SRC ** A real header scratcher: creating a new Twitter header We can use all of the source blocks we created back when we were professing our love for Emacs. However, we need to change a few references. In the source block below we need to change the reference from ~body=hello-world~ to ~body=twitter-plant-body~. #+NAME: twitter-oauth-headers-plants #+BEGIN_SRC emacs-lisp :var nonce=nonce consumer-key=twitter-consumer-key access-token=twitter-access-token oauth-time=oauth-time body=twitter-plant-body signing-key=signing-key (let* ((twitter-oauth-headers (list (cons "oauth_consumer_key" consumer-key) (cons "oauth_nonce" nonce) (cons "oauth_signature_method" "HMAC-SHA1") (cons "oauth_timestamp" oauth-time) (cons "oauth_token" access-token) (cons "oauth_version" "1.0"))) (signature-string (twitter-signature-string "POST" "https://api.twitter.com/1.1/statuses/update.json" (append twitter-oauth-headers twitter-body))) (signature (org-sbe createSignature (signature_string (eval signature-string)) (key (eval signing-key))))) (append twitter-oauth-headers (list (cons "oauth_signature" (url-hexify-string signature))))) #+END_SRC ** Repeat: Encoding Data and Posting To Twitter Similiarly, we need to reassign ~twitter-oauth-headers=twitter-oauth-headers~ to ~twitter-oauth-headers=twitter-oauth-headers-plants~ #+NAME: twitter-restclient-headers-plants #+BEGIN_SRC emacs-lisp :var header=twitter-headers twitter-oauth-headers=twitter-oauth-headers-plants (build-twitter-header-string header (sort twitter-oauth-headers (lambda (first second) (string< (car first) (car second))))) #+END_SRC Again, we encode our body... #+NAME: twitter-post-body-1 #+BEGIN_SRC emacs-lisp (setq twitter-post-body (concat "" (mapconcat (lambda (entry) (concat (car entry) "=" (url-hexify-string (cdr entry)))) twitter-body "&") "")) #+END_SRC Voila! We've can post a cute plant...or tree...or flower...to Twitter! #+BEGIN_SRC restclient :var twitter-headers=twitter-restclient-headers-1 body=twitter-post-body-1 # :body := (concat twitter-post-body) POST https://api.twitter.com/1.1/statuses/update.json?:body :twitter-headers #+END_SRC ** In conclusive You don't need to use Emacs to enjoy the benefits of literate programming across 4 disparate APIs. To do that all you really need is bubblegum and determination. However, literate programming is a great paradigm to create and share your work for others to play with. I think Emacs and in particular, org-mode, is great for literate programming because it's a lot like [[https://jupyter.org/][Jupyter Notebooks]], it supports a lot more languages and supports more than one language per notebook. Plus, org-mode works wonderfully with the two best [[http://spacemacs.org/][flavours]] of [[https://github.com/hlissner/doom-emacs][Vim]]! * References + https://developer.twitter.com/en/docs/basics/authentication/overview/application-only + https://cloud.google.com/vision/docs/request + https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/post-statuses-update.html + https://developers.google.com/custom-search/docs/overview + http://lti.tools/oauth/