Brad Parker

Books, Badgers and Big Numbers

hi@bradparker.id.au (Brad Parker) — Sat, 21 Jun 2025 00:00:00 UTC

I stumbled into what might be some combinatorial esoterica and I can't stop thinking about it.

Books

Up until April 2023 we had a pretty special second hand book shop up the road, next to the local café¹. While it was there my mid morning coffee runs usually involved having a browse and I often picked something up.

A lot of the books I was buying were non-fiction, and it's this I'd like to blame on my over-thinking their organisation. I wanted everything grouped by subject, rather than author or title or year or anything like that. So in the end I signed up for a LibraryThing account and used their codes to sort everything.

Badgers

Some time last year LibraryThing let me know they'd made me a Year In Review. You know, like Spotify does. It's not really my jam, but I clicked through anyway and ended up reading this sentence:

And make sure your floors will support the added weight of 4.90 adult badgers!

I'd scanned in all our books at once, so this doesn't represent how many books we added to our collection in 2024, but rather the whole lot. At any rate, it's common now that when I look at our bookshelves I find myself trying to imagine five adult badgers somehow sitting in them. Which is hard, given that I have no idea how big an adult badger is.

Big Numbers

Despite the badgers, it's not all that many books really. But even still I felt the need to choose some sort of ordered arrangement. Now, without worrying about the usefulness of any particular arrangement, I wonder how many options there really are? How many ways could I have arranged these books?

In my last post I talked about enumerating possibilities, but in all the examples I used I permitted every possible option. Nothing was excluded, even if two coin tosses or dice throws resulted in the same outcome I still counted them as a unique possibility. What I was talking about there were Cartesian Products. That is: given a set of options $A$ and a set of options $B$ the total range of possibilities of combing both is $A \times B$ , the Cartesian Product of the sets $A$ and $B$ .

$\begin{align*} A &= \{\text{a}, \text{b}, \text{c}\} \\ B &= \{1, 2, 3\} \\ A \times B &= \begin{Bmatrix} (\text{a}, 1) & (\text{a}, 2) & (\text{a}, 3) \\ (\text{b}, 1) & (\text{b}, 2) & (\text{b}, 3) \\ (\text{c}, 1) & (\text{c}, 2) & (\text{c}, 3) \end{Bmatrix} \end{align*}$

The total number of possibilities is the size of each input set multiplied together.

$\begin{align*} |A| &= 3 \\ |B| &= 3 \\ |A \times B| &= |A| \times |B| \\ &= 9 \end{align*}$

What if we imagine that arrangements are the same as combined sets of possibilities? That we can choose an element (any element) from the set $A$ 'number of elements in $A$ ' (a.k.a. $|A|$ ) times? We might then say that the total number of arrangements of the elements in $A$ , is $|A^{|A|}|$ .

$\mathit{\text{No. arrangements of } A} = \overbrace{|A| \times |A| \times ... \times |A|}^{|A|}$

More concretely.

$\begin{align*} A &= \{\text{a}, \text{b}, \text{c}\} \\ A^{|A|} &= A \times A \times A \\ &= \begin{Bmatrix} (\text{a}, \text{a}, \text{a}) & (\text{a}, \text{a}, \text{b}) & (\text{a}, \text{a}, \text{c}) \\ (\text{a}, \text{b}, \text{a}) & (\text{a}, \text{b}, \text{b}) & (\text{a}, \text{b}, \text{c}) \\ (\text{a}, \text{c}, \text{a}) & (\text{a}, \text{c}, \text{b}) & (\text{a}, \text{c}, \text{c}) \\ (\text{b}, \text{a}, \text{a}) & (\text{b}, \text{a}, \text{b}) & (\text{b}, \text{a}, \text{c}) \\ (\text{b}, \text{b}, \text{a}) & (\text{b}, \text{b}, \text{b}) & (\text{b}, \text{b}, \text{c}) \\ (\text{b}, \text{c}, \text{a}) & (\text{b}, \text{c}, \text{b}) & (\text{b}, \text{c}, \text{c}) \\ (\text{c}, \text{a}, \text{a}) & (\text{c}, \text{a}, \text{b}) & (\text{c}, \text{a}, \text{c}) \\ (\text{c}, \text{b}, \text{a}) & (\text{c}, \text{b}, \text{b}) & (\text{c}, \text{b}, \text{c}) \\ (\text{c}, \text{c}, \text{a}) & (\text{c}, \text{c}, \text{b}) & (\text{c}, \text{c}, \text{c}) \end{Bmatrix} \\ |A^{|A|}| &= |A| \times |A| \times |A| \\ &= 27 \end{align*}$

But we know that's not right. There are many things in there which aren't arrangements. Arranging a collection won't duplicate any of its elements, so things like $(\text{a}, \text{a}, \text{a})$ and $(\text{c}, \text{b}, \text{b})$ shouldn't be counted.

What we need is a method to count arrangements that accounts for elements being used only once. Because that's how we might lay out an arrangement of some collection. We'd select an item to start with, then select the next item from what's left, then the next from what's left, and so on until we've placed every item and run out of options to select from.

For example, given the following collection of items.

We could arrange them like so.

Step	Options	Selection	Result
1
2
3

At each step the set of options reduces by one. Once 'b' is put down, only 'a' and 'c' are left. On the next step once 'a' is put down only 'c' is left. So, in general, rather than this.

$\mathit{\text{No. arrangements of } A} = \overbrace{|A| \times |A| \times ... \times |A|}^{|A|}$

What we have instead is this.

$\mathit{\text{No. arrangements of } A} = |A| \times (|A| - 1) \times (|A| - 2) ... \times 1$

For the 'a', 'b', 'c' example then we'd have.

$\begin{align*} A &= \{\text{a}, \text{b}, \text{c}\} \\ \mathit{\text{No. arrangements of } A} &= |A| \times (|A| - 1) \times 1 \\ &= 3 \times 2 \times 1 \\ &= 6 \end{align*}$

And here they all are.

The result you get from multiplying all the numbers less than some number is called the factorial of that number. It's written like this: $n!$ , and can be defined like this:

$n! = n \times (n - 1) \times (n - 2) \times ... \times 2 \times 1$

Right now we've got 419 books which can be sorted by a code. So how many ways could I have arranged them? What's the factorial of 419?

$419 \times 418 \times 417 \times ... \times 3 \times 2 \times 1$

It's this extremely large number.

2,908,421,057,896,340,474,361,403,432,093,534,061,335,411,576,803,237,495,067,620,711,879,592,887,139,553,095,110,943,225,026,262,406,188,778,320,680,189,451,711,755,153,962,269,093,225,791,463,391,282,309,784,673,974,891,888,521,144,472,954,435,530,212,518,938,132,427,719,386,733,483,011,170,577,979,186,407,238,784,514,071,038,335,200,586,427,023,669,168,853,348,715,855,934,206,590,666,886,642,210,446,229,488,444,697,075,683,231,304,763,674,216,898,857,425,205,633,114,990,510,459,902,307,876,760,425,583,148,615,724,632,751,661,133,203,237,905,432,668,028,736,816,553,087,322,961,245,249,766,243,189,634,620,342,650,104,310,228,160,720,464,432,084,583,406,587,932,793,842,882,817,312,083,420,305,011,614,174,094,373,347,653,327,314,151,733,173,716,990,349,453,642,618,332,498,813,131,703,848,734,139,461,889,356,833,404,468,988,162,294,586,549,217,704,913,958,653,349,142,669,023,627,237,073,393,370,291,667,739,306,611,350,729,516,327,327,902,034,218,650,157,031,351,111,738,938,367,328,649,342,699,827,177,333,219,328,746,468,751,048,295,766,233,367,259,458,382,317,867,840,592,639,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000.

And here's the kind of question that led to my current cosy combinatorial corner: given a sorted list of all possible arrangements, what is the (deep breath) 2,844,113,050,416,543,052,622,601,377,724,177,730,794,635,580,558,165,101,289,087,907,211,046,617,413,162,202,727,992,682,826,505,610,602,434,122,752,894,055,956,478,001,246,987,975,654,867,880,927,440,389,088,878,653,260,211,539,374,415,866,208,792,187,943,989,161,515,680,780,226,637,325,041,170,571,521,944,136,857,756,357,610,408,690,480,987,642,359,289,070,142,617,861,100,311,817,255,304,659,710,866,175,534,883,866,241,789,715,155,834,223,460,969,127,343,122,506,553,271,818,293,567,727,306,529,166,052,023,268,045,186,141,937,347,041,818,966,122,038,640,793,796,799,685,516,455,929,549,507,013,186,833,430,265,450,466,044,505,744,178,793,393,254,499,311,497,130,361,821,989,719,697,224,981,301,053,842,088,290,183,796,995,774,643,108,623,128,572,424,596,295,207,864,719,379,479,419,093,894,525,035,709,405,252,305,572,483,849,439,595,150,582,362,925,993,365,553,189,173,069,073,069,239,396,792,666,051,139,882,098,736,318,227,488,496,884,365,414,402,236,391,348,245,541,096,370,960,228,964,587,593,160,349,264,902,263,727,302,169,421,863,892,154,866,370,306,041,614,127,727,830,967,236,995,581,667,801,392,480,447,679,422,359,386,205,525,674,979,615,059,536,733,032,646,807,296,164,119,104,440,573,184,560,777,118,243rd one? You know … for example.

Lexicographic order

How might arrangements be sorted? To answer that I'd need to define when one arrangement should come before another. Here's a scheme.

Step through pairs of elements from both arrangements. I.E. First look at the first element of arrangement $x$ and the first element from arrangement $y$ , then the second element from arrangement $x$ and the second element from arrangement $y$ , and so on.
For the first pair with mismatching elements, if the element from $x$ should come before the element from $y$ then the whole arrangement $x$ should come before the arrangement $y$ .

For example, here's two arrangements of the first three letters of the alphabet.

$\begin{align*} x &= (\text{b}, \text{c}, \text{a}) \\ y &= (\text{b}, \text{a}, \text{c}) \end{align*}$

Compare the first element of each ordering: $x_1 = \text{b}$ and $y_1 = \text{b}$ . They match, so move onto the next pair of elements.
Compare the second element of each ordering: $x_2 = \text{c}$ and $y_2 = \text{a}$ . They don't match and $\text{a}$ comes before $\text{c}$ , therefore $y$ comes before $x$ .

Sorting elements this way will result in them being in Lexicographic order, and it might seem like the way mostly everything is sorted in day to day life … because it is! This is how we sort words in the dictionary, hence the name.

Indexing

The example above sorts arrangements with reference to a pre-existing ordering of their elements: the alphabet. But it didn't have to. We could've used some other ordering, and ended up with a different result. If we lived in an alternate timeline where the Latin alphabet began $(\text{c}, \text{a}, \text{b})$ rather than $(\text{a}, \text{b}, \text{c})$ then the order of our example arrangements would be reversed and $x$ would come before $y$ .

The alphabet we have is one possible arrangement. One of 403,291,461,126,605,635,584,000,000 options in fact. To reduce confusion when talking about any alternative arrangements we can choose to ignore the specific elements of whatever collection we're arranging and instead refer only to their position in some select conventional ordering. The alphabet is the convention for letters, where 'a' is first, 'b' is second, 'c' is third, and so on. We can then discuss other arrangements by saying only where elements 1, 2, and 3 are. For example, the arrangement $(\text{b}, \text{a}, \text{c})$ becomes $(2, 1, 3)$ .

When we can use a sequence of numbers to refer to all the elements of a collection we're able to say that the collection is indexed by those numbers. The numbers point to elements in the collection.²

For our books I'll use Alphabetical by title as the conventional ordering. This then defines the Lexicographic order for the list of possible arrangements.

Lehmer codes

Right, here we go: Lehmer codes index all possible arrangements of collections of elements, and they do so in Lexicographic order.

To see how we can revisit my earlier example of laying out the first three letters of the alphabet.

Step	Options	Selection	Result
1
2
3

Lehmer codes end up recording the steps in this procedure as a sequence of choices of index. Like so:

Select the second option.
Select the first option from what's left.
Select the first option from what's left.

Now, the indexes in Lehmer codes all begin at 0, rather than 1, so for the arrangement: $(\text{b}, \text{a}, \text{c})$ , we have this sequence of choices: $(1, 0, 0)$ .

Here are all the choices someone could make when laying out the first three letters of the alphabet.

Step 1	Step 2	Step 3
0 from (a, b, c)	0 from (b, c)	0 from (c)
0 from (a, b, c)	1 from (b, c)	0 from (b)
1 from (a, b, c)	0 from (a, c)	0 from (c)
1 from (a, b, c)	1 from (a, c)	0 from (a)
2 from (a, b, c)	0 from (a, b)	0 from (b)
2 from (a, b, c)	1 from (a, b)	0 from (a)

Which can be condensed into this list of sequences.

$(0, 0, 0)$
$(0, 1, 0)$
$(1, 0, 0)$
$(1, 1, 0)$
$(2, 0, 0)$
$(2, 1, 0)$

And here's the nifty trick: this is counting, but it's counting in a number system quite unlike the one we use day to day.

At the first step there are $3!$ (a.k.a $3 \times 2 \times 1$ ) possible outcomes, then at the second step there are $2!$ . I could also say that at the second step there are $(3 \times 2 \times 1) / 3$ choices. It's the same thing, but by saying it that way I'm trying to emphasise that the first choice puts the rest of the sequence into one of three branches.

I could also try to emphasize that this way: when you choose 'b' you skip over the $2!$ possibilities you might've reached if you'd chosen 'a'. When you choose 'c' you skip over the $2!$ possibilities from choosing 'a', and also the $2!$ from 'b'. So when choosing 'c' you skip over $2 \times 2!$ possibilities.

When viewed this way it's is not completely unlike ordinary counting. If asked to find the 243rd something you could skip over the first 200 things, then you could skip over the next 40 things, finally arriving at the 3rd thing after that. By choosing 2 in the hundreds place this number goes down the branch where the range of possibilities is within $200 \leq n \lt 300$ .

What's different in the case of these sequences of index choices is that instead of hundreds, tens and ones we have $2!$ s, $1!$ s and $0!$ s.

What happens then if we multiply each index in the above list of sequences by its factorial place value? (Just as we do with ordinary numbers.)

$(0 \times 2!, 0 \times 1!, 0 \times 0!) = (0, 0, 0)$
$(0 \times 2!, 1 \times 1!, 0 \times 0!) = (0, 1, 0)$
$(1 \times 2!, 0 \times 1!, 0 \times 0!) = (2, 0, 0)$
$(1 \times 2!, 1 \times 1!, 0 \times 0!) = (2, 1, 0)$
$(2 \times 2!, 0 \times 1!, 0 \times 0!) = (4, 0, 0)$
$(2 \times 2!, 1 \times 1!, 0 \times 0!) = (4, 1, 0)$

Interesting, and now if we add the places together? (Again, just as we do with ordinary numbers.)

$0 + 0 + 0 = 0$
$0 + 1 + 0 = 1$
$2 + 0 + 0 = 2$
$2 + 1 + 0 = 3$
$4 + 0 + 0 = 4$
$4 + 1 + 0 = 5$

What we're looking at here is the Factorial number system, and how factorial numbers can be converted into ordinary, decimal numbers.

Now, factorial numbers are not just factorial numbers, they are also Lehmer codes. The number 2 is the decimal representation of the Lehmer code $(1, 0, 0)$ . Which is a sequence of choices made when laying out an arrangement of elements. It's possible to transform that sequence of choices into indexes into the original collection. If we do so with the sequence $(1, 0, 0)$ we get these indexes: $(1, 0, 2)$ . Following those indexes back to the letters they point to we get $(\text{b}, \text{a}, \text{c})$ .

This gives me everything I need to answer my silly question: What is the 2,844, … 243rd possible arrangement of our 5-ish badgers worth of books? I only have to take that number, subtract one, convert that from decimal to factorial, and then use those digits as instructions for laying out an arrangement. This all happens very quickly. It takes about two hundredths of a second to go from an enormous number to a list of books.

> BS.writeFile "sorted_books.json" (Aeson.encode (decodeLehmer books (toFactorial n)))
(0.02 secs, 17,994,448 bytes)

[
  {
    "author": "Tufte, Edward R.",
    "code": "001.4226",
    "title": "The Visual Display of Quantitative Information"
  },
  {
    "author": "Weizenbaum, Joseph",
    "code": "001.64",
    "title": "Computer Power and Human Reason"
  },
  {
    "author": "Gleick, James",
    "code": "003.857",
    "title": "Chaos: Making a New Science"
  },
  {
    "author": "Galloway, Alexander",
    "code": "004.09",
    "title": "Uncomputable: Play and Politics In the Long Digital Age"
    // ...

I think that's really something.

Having written this I can think of a couple of reasons I've found Lehmer codes so interesting.

Firstly, there's something indescribably cool about how they work. Embedded in the numbers of this very strange number system is the procedure for encoding and decoding them as arrangments of elements. The strange number system is perfectly suited to encoding and decoding arrangements, it's factorial! This, to me, feels simultaneously obvious and obscure and I love it.

Secondly, when I first came across Lehmer codes I was unaware of so many of the foundations upon which they're built, and it left me a bit awe struck. I was overwhelmed thinking about all the epistemic iterations³ it would take to arrive at something like this. All the large and small thoughts, and all the people who thought them.

At the end of the day I have no idea if Lehmer codes will ever prove to be practically useful to me. Regardless, I do know that I find them wonderful.

We were very sad to see Logical Unsanity close. The ABC did a nice feature at the time: Quirky tin shed bookshop 'born out of laziness' offers booklovers' sanctuary↩︎
… from indicare "to point out," …

Index - Etymology, Origin & Meaning ↩︎
This is a term I've just come across from reading Beyond Measure by James Vincent (2022). It was coined by Hasok Chang in his book Inventing Temperature (2004).↩︎

Crossing paths with Descartes

hi@bradparker.id.au (Brad Parker) — Sat, 31 May 2025 00:00:00 UTC

Lately I've been thinking about really fun solutions. You know, those solutions which use some insight you were completely unaware of and make it all look like magic. It's been more than once that I've found myself stumbling into something called combinatorics when trying to learn more about them. On the way to better understanding these combinatorial solutions I've been reflecting on things more foundational. Here, hang on, I'll try to explain what I mean.

Two-up

Two-up is a game played in Australia. Someone tosses two coins in the air, at the same time, and people place bets on the outcome. It is forbidden by law except for one day of the year, ANZAC day. If you want to understand why betting on a coin toss might be outlawed (and otherwise just enjoy a great if but harrowing read) I reckon Wake in Fright is worth your time.

How many possible states can two coins be in? The first coin can be heads or tails, and the second coin can be heads or tails.

	Second heads	Second tails
First heads	First heads, second heads	First heads, second tails
First tails	First tails, second heads	First tails, second tails

There are two coins, each with two sides they could land on.

$\mathit{\text{No. of sides coin 1}} \times \mathit{\text{No. of sides coin 2}} = \mathit{\text{No. of possibilities}}$

So.

$2 \times 2 = 4$

In Two-up two of these possibilities are counted as the same outcome. The order of the coins doesn't matter, which is handy, keeping track of them flying through the air, a few pints deep, would surely be diabolical. So one outcome ends up being twice as likely as the other two. Sounds simple, ends up pretty complicated for John Grant.

Hazard

Hazard is a dice game which was popular in 17th and 18th century England. Someone rolls a pair of dice and places bets on the outcome. The rules for assessing the success or failure of an outcome are fairly complicated. It's referenced in Chaucer's Canterbury Tales in a passage that may have given rise to the idiom "at sixes and sevens", which refers to being in a state of confusion or disarray. I've never read Canterbury tales, I suspect if I tried I'd end up at sixes and sevens.

How many possible states can two dice be in? The first dice can have any one of the numbers one to six face up, as can the second.

	1	2	3	4	5	6
1
2
3
4
5
6

There are two dice, each with six sides they could land on.

$\mathit{\text{No. of sides dice 1}} \times \mathit{\text{No. of sides dice 2}} = \mathit{\text{No. of possibilities}}$

So.

$6 \times 6 = 36$

As with Two-up, many of these possibilities are counted as the same outcome. All that matters is the sum of the numbers on each dice. So not only is 6 and 4 equal to 4 and 6, but also 3 and 7. But we needn't worry about all that.

DNA

One more game of chance.

Deoxyribonucleic acid is a pretty special molecule. Two chains of smaller molecular units (called nucleotides) each zipped together by the attractive force brought about by electron density being greater nearer to nitrogen or oxygen atoms than hydrogen atoms.

DNA nucleotides all have a deoxyribose component attached to one of four different sub-units (nitrogenous bases): adenine, cytosine, guanine and thymine. It's from these we get the letters we use to denote the genetic code: A, C, G and T.

DNA serves multiple purposes. One is as a sort of template that informs protein construction. Proteins are also pretty special molecules, built out of one or more chains (polypeptides) of other molecular units (amino acids).

In 1961 Francis Crick, Sydney Brenner, Leslie Barnett and R. J. Watts-Tobin figured out that this process of protein construction requires that the nucleotides that make up the protein-templating parts of DNA come in groups of three. They called these groups codons.

How many different codons could there be?

There are three 'slots' in each codon, which could each be filled by one of four nucleotide types.

$\mathit{\text{No. of nucleotides}} \times \mathit{\text{No. of nucleotides}} \times \mathit{\text{No. of nucleotides}} = \mathit{\text{No. of possible codons}}$

So.

$4 \times 4 \times 4 = 64$

In 1954 Watson and Crick intuited a list of the 20 amino acids believed to be constructed according to DNA's template. They were inspired to do so in response to a tidy, but ultimately incorrect, geometric explanation of the process put forward by a theoretical physicist, George Gamow. Later, in 1956, they came up with their own tidy, but ultimately incorrect, combinatorial explanation.¹

This seems like fiendishly complex stuff, there's sure to be many red herrings on the path to the right explanation. I guess that's just how the cookie crumbles.

Quartets

The string quartet has been around as a musical genre and an ensemble instrumentation since the 1750s. In 1825 Beethoven wrote some particularly spicy quartets, numbers 12 to 16, the last major compositions he completed. I love them, though not everyone does. A contemporary of Beethoven, and composer himself, Louis Spohr, described them as "indecipherable, uncorrected horrors."

Three of the quartets, 15, 13, and 14, centre thematically around particular groupings of four notes, groupings united by how far away their constituents are from each other. Which has me wondering what indecipherable, uncorrected horrors might lurk in the collection of all possible voicings a string quartet could play?

First, how many are there? I'm honestly not sure, the ranges of these instruments are a bit flexible, it can depend on the player, whether harmonics are included, the construction of a particular instance of an instrument. I'll use these ranges so we can at least get a ballpark figure.

Violin: G3 - A7, 51 notes
Viola: C3 - E6, 41 notes
Cello: C2 - A5, 46 notes

The first and second violins can each play any one of 51 notes, the viola one of 41, and the cello one of 46. So.

$51 \times 51 \times 41 \times 46 = \text{4,905,486}$

That is a lot.

In space

It's a vast space of possibilities, and Beethoven's four note groupings are in there somewhere, but where? Where, for example, is Cello E4, Viola F4, Second Violin G♯4, First Violin A4? An arrangement of notes played by specific instruments which from now on I will refer to as The Voicing.

It might seem like a nonsense question, but at least it can be answered, and one answer is this: we can think of each instrument as a dimension in space and plot The Voicing within it. Instead of the familiar three dimensions we think of moving about in day to day life, left-right, up-down and forward-backward, we have four dimensions: the cello, the viola, the second violin and the first violin. Because each instrument is constrained in the range of notes it can play we're really talking about some four dimensional shape. We can imagine plotting coordinates within this shape by measuring how far along each dimension a point extends. Starting at 0, being the lowest note that instrument can play, and counting each subsequent note, ending at the highest. I appreciate that counting from 0 might seem odd, but I've spent too much time with computers and it does make some of the following a little simpler.

So, where is The Voicing within that shape? It's at the position: Cello = 28, Viola = 17, Second Violin = 13 and First Violin = 14. And where is that exactly? Well, I can't show you. Unlike the examples of coin tosses, dice rolls or DNA codons, I can't create a visualisation of this space of possibilities. I've never seen any four dimensional things, and I lack the imagination to even try to picture them. What I can do though is change the four dimensional space of possibilities into a different shape and show you that instead.

To demonstrate, here's a two dimensional space of possibilities. Perhaps imagine it's a duet of tiny, five tone, glockenspiels.

Because this shape doesn't go on forever and I'm not planning on breaking any marbles into pieces two dimensions can be flattened into one by placing each row end to end. A rectangle can become a line.

With this scheme the second yellow marble moves from column 1 row 2 to column 11 row 0. Each marble is being counted, left to right along each row.

In general, columns are counted row number times (in this case 2), then column number is added.

$\mathit{\text{No. of columns}} \times \mathit{\text{Row number}} + \mathit{\text{Column number}}$

With the addition of another imaginary glockenspiel the same process can be demonstrated in three dimensions, this time labeled $x$ , $y$ , and $z$ .

This three dimensional shape can become a two dimensional shape by unstacking each layer and placing them end to end.

The yellow cube that begins at position $x = 1, y = 2, z = 1$ ends up at position $x = 1, y = 0, z = 11$ .

Because the $y$ dimension will now always be 0 we can ignore it which allows us to say: position $(1, 2, 1)$ has moved to $(1, 11)$ . More generally we could say: any position $(x, y, z)$ has moved to position $(x, |Z| \times y + z)$ . Where $|Z|$ means: the total number of possible values of $z$ . The $z$ dimension has now absorbed the $y$ dimension.

As with turning two dimensions into one, this counts positions. However here they're not all counted in one go. There's a count per $x$ value.

Unstacking the four dimensional shape containing all the string quartet voicings by combining the two violins into one dimension yields a three dimensional rectangular prism of these measurements: Cello = 46, Viola = 41, Violins = 2,601.

Allotting one cubic millimetre to each voicing yields something very roughly the size and shape of one of these standard cuts of timber. (If you look closely you might see a magpie for scale.)

The profile is wrong, it should be nearly square, and it's too short by about 200 millimeters. But it still helps me to look at it and think about what it tells me about the space of possibilities we've been talking about.

Firstly though, what it doesn't tell me is much about The Voicing. It can be found at Cello = 28, Viola = 17, Violins = 677. Which is about where I've placed this marble, except it'd be embedded within the timber, roughly in the centre of the cross-section.

I just don't find it all that enlightening to see. What I do find enlightening is imagining what happens to this piece of timber if we continue this process of unstacking dimensions.

If a three dimensional set of possibilities can be unstacked into a two dimensional set, then we can imagine slicing this piece of timber into one millimeter thin layers and laying them out side by side. If the timber were the right length to begin with we'd have a roughly 2.6 by 1.8 metre rectangular sheet. Which doesn't seem all that surprising to me.

Now, if a two dimensional set of possibilities can be unstacked into a one dimensional set, then we can imagine slicing this sheet into one millimetre by one millimetre pieces of dowel and laying them out end to end. If we did so we'd have a nearly five kilometre long line.

It doesn't change the numbers, but it does change my perspective. It reminds me of how unintuitive it is that each tiny cell in our bodies contains chromosomes that, if unwound, would be two metres in length.

In time

Laying everything out in a line like this is, in a sense, providing a way to visit every possibility in a space. If it were to be stacked back up again we'd have a path which snakes (in this case a very orderly, if but zig-zaggy, snake) through the whole four dimensional shape. Unstacking the whole thing into a line has laid every voicing out in a sequence.

Having this sequence enables us to turn any four note voicing into a single, unique, number between 0 and 4,905,485. For example The Voicing, which, in the space of all string quartet voicings, has the coordinates Cello = 28, Viola = 17, Second Violin = 13, First Violin = 14, becomes 1,371,932.

By referring to the scheme for unstacking shapes explored above it's possible to describe a method for finding the number ( $n$ ) of any voicing made up of:

A note played by the first violin ( $v_1$ ), from within the set of notes it can play ( $V$ )
A note played by the second violin ( $v_2$ ), from within the set of notes it can play ( $V$ )
A note played by the viola ( $w$ ), from within the set of notes it can play ( $W$ )²
A note played by the cello ( $c$ ), from within the set of notes it can play ( $C$ )

Each dimension can be unstacked just as the rows of the five by five grid of marbles were laid out end to end.

$\begin{align*} & (v_1, v_2, w, c) \\ & (v_1 \times |V| + v_2, w, c) \\ & ((v_1 \times |V| + v_2) \times |W| + w, c) \\ n =& ((v_1 \times |V| + v_2) \times |W| + w) \times |C| + c \end{align*}$

Where putting ' $|$ ' either side of the name for a set of things means "the total number of things in the set." For example $|C|$ means: the total number of notes a cello can play.

This allows for counting through all the voicings and it bears a resemblance to how we usually count things. A number such as 1,825 is usually thought of as "5 ones, plus 2 tens, plus 8 hundreds, plus 1 thousand." However it could also be stated as follows.

$\begin{align*} \text{1,825} &= ((1 \times 10 + 8) \times 10 + 2) \times 10 + 5 \\ &= ((10 + 8) \times 10 + 2) \times 10 + 5 \\ &= (18 \times 10 + 2) \times 10 + 5 \\ &= (180 + 2) \times 10 + 5 \\ &= 182 \times 10 + 5 \\ &= \text{1,820} + 5 \\ &= \text{1,825} \end{align*}$

Which to me looks like successively pushing the ones place to the left, by multiplying by ten, to make room for the next ones value.

This suggests there should be a method for finding The Voicing's number which more closely matches how we talk about everyday numbers, and indeed there is, though it might not look the same at first flush.

$\begin{align*} n &= v_1 \times (|V| \times |W| \times |C|) + v_2 \times (|W| \times |C|) + w \times |C| + c \\ &= 14 \times (51 \times 41 \times 46) + 13 \times (41 \times 46) + 17 \times 46 + 28 \\ &= \text{1,371,932} \end{align*}$

It's possible to express the number 1,825 equivalently.

$\begin{align*} \text{1,825} &= 1 \times (10 \times 10 \times 10) + 8 \times (10 \times 10) + 2 \times 10 + 5 \\ &= \text{1,825} \end{align*}$

Which suggests that these two expressions are equivalent in general.

$\begin{align*} n &= ((v_1 \times |V| + v_2) \times |W| + w) \times |C| + c \\ &= (v_1 \times |V| + v_2) \times (|W| \times |C|) + v \times |C| + c \\ &= v_1 \times (|V| \times |W| \times |C|) + v_2 \times (|W| \times |C|) + v \times |C| + c \end{align*}$

What has been arrived at here looks to me like a sort of number system³. One that counts through all the voicings a string quartet can play, just as we might count anything else. What makes this a little more interesting is that it doesn't just count, it encodes. Unlike when counting many things in day to day life, counting these voicings using the method described allows for recovering the original four parts from the single number assigned to them when counted.

To perform this recovery, to decode what was encoded, we need to undo each multiplication and addition which converted the note positions into their corresponding number. To undo one step of multiplication and addition we can use what's called Euclidean division.

If you ever did long division in school, where you figured out how many times some number (the "divisor", $d$ ) fit into some other number ( $n$ ) with some remainder ( $r$ ) left over, you've done Euclidean division. It's division where you end up with a whole number (the "quotient" $q$ ), and possibly some remainder; some amount smaller than the number you were dividing by.

Let's say we begin with some number $n$ which we know to be made by multiplying some number $q$ by some number $d$ and then adding some number $r$ (which we know is less than $d$ ) to that.

$n = q \times d + r$

This is an abstract form of the numbers we're aiming to decode.

$\begin{align*} n &= ((v_1 \times |V| + v_2) \times |W| + w) \times |C| + c \\ q &= ((v_1 \times |V| + v_2) \times |W| + w) \\ d &= |C| \\ r &= c \\ n &= q \times d + r \end{align*}$

Which can be shuffled around to produce $q$ .

$q = \frac{n - r}{d}$

And also $r$ .

$r = n - q \times d$

There are several methods to figure out $q$ , one is to perform something called floored division which is regular division but the result is rounded down to the nearest whole number. It's written like this.

$q = \lfloor \frac{n}{d} \rfloor$

Floored division discards the remainder, but it can be recovered with one of the above shufflings.

$r = n - q \times d$

This provides us with a procedure for decoding a number step by step, which goes something like this.

Begin with the number we want to decode.

$\begin{align*} n &= ((v_1 \times |V| + v_2) \times |W| + w) \times |C| + c \\ &= ((v_1 \times 51 + v_2) \times 41 + w) \times 46 + c \\ &= \text{1,371,932} \end{align*}$
Then, find the value for $q_c$ , the first quotient, by using floored division with $|C|$ as the denominator.

$\begin{align*} q_c &= \lfloor \frac{n}{|C|} \rfloor \\ &= \lfloor \frac{\text{1,371,932}}{46} \rfloor \\ &= \text{29,824} \\ \end{align*}$

Which can be used to find $c$ , the note the cello plays, by using the shuffling which gave us $r$ above.

$\begin{align*} c &= n - |C| \times q_c \\ &= \text{1,371,932} - 46 \times \text{29,824} \\ &= 28 \end{align*}$

The first layer of addition and multiplication has now been peeled off.

$\begin{align*} \text{1,371,932} &= c + |C| \times q_c \\ &= 28 + 46 \times \text{29,824} \end{align*}$
The process is then repeated beginning from $q_c$ instead of $n$ to find $q_w$ , the second quotient.

$\begin{align*} q_w &= \lfloor \frac{q_c}{|W|} \rfloor \\ &= \lfloor \frac{\text{29,824}}{41} \rfloor \\ &= 727 \\ \end{align*}$

Which can be used to find $w$ , the note the viola plays.

$\begin{align*} w &= q_c - |W| \times q_w \\ &= \text{29,824} - 41 \times 727 \\ &= 17 \end{align*}$
We repeat the process once more to find $q_{v_2}$ in order to find $v_2$ , the note the second violin plays.

$\begin{align*} q_{v_2} &= \lfloor \frac{q_w}{|V|} \rfloor \\ &= \lfloor \frac{727}{51} \rfloor \\ &= 14 \\ v_2 &= q_w - |V| \times q_{v_2} \\ &= 727 - 51 \times 14 \\ &= 13 \\ \end{align*}$
By finding $q_{v_2}$ we've already found the note the first violin plays.

$\begin{align*} v_1 &= q_{v_2} \\ &= 14 \end{align*}$

With that we're done.

$\begin{align*} c &= 28 \\ w &= 17 \\ v_2 &= 13 \\ v_1 &= 14 \end{align*}$

With this ability to decode ordinary numbers into string quartet voicings it's possible to enumerate all the possible voicings just by counting from 0 to 4,905,485. Doing so sounds like this.

Note without JavaScript you'll get about a six minute sample.

At this speed, ten notes per second, it would take you more than a day and a half of uninterrupted listening to reach The Voicing. I like it. I think it sounds very meditative. Very swoopey.

Turning voicings into numbers has also afforded us the opportunity to be disorderly, rather than orderly, snakes. We can use a non-repeating random number generator⁴ to erratically jump around, but still visit each and every voicing no more than once. I'm not sure if there's any telling when you'll hit The Voicing, but it's in there. To hear them all, to make sure you've heard the one, would take over five days and sixteen hours.

Note without JavaScript you'll get about a six minute sample.

If I listen for long enough I can talk myself into hearing bass lines and melodies. I think it's sort of fun. Bleep, bloop.

And now here we are at the end, having found some sort of path through an overwhelming space of possibilities. Not only that, along the way we've been able to visit an Australian literary classic, confusing dice games from Early modern Britain, attempts to explain the bewildering complexities of DNA, and some really banging Beethoven. I hope you enjoyed walking it as much as I enjoyed finding it.

Thank you for your time.

Deciphering the Genetic Code: The Most Beautiful False Theory in Biochemistry – Part 1 ↩︎
I'm sorry, there's already a 'v' and 'w' is just the next letter in the alphabet. If it helps pronounce it "wiola" in your head. Or out loud. Whatever you're up for.↩︎
A mixed radix number system.↩︎
In this case a Lehmer random number generator.↩︎

The danger of using models as metaphors

hi@bradparker.id.au (Brad Parker) — Thu, 21 Mar 2024 00:00:00 UTC

I read Elinor Ostrom's book Governing the commons on a whim after seeing it mentioned in a discussion about open source software communities. I believe this isn't an uncommon way for software folk to become aware of this book. Of the many fascinating insights Ostrom presents there are a few that have really stuck with me. My personally sticky insights aren't really related to the core argument, it's all well out of my wheel house after all. My ears most pricked up when Ostrom wrote about the social responsibilities of scientists and the social impact of social research.

Towards the start of the book in a subsection of the first chapter entitled The metaphorical use of models Ostrom writes.

When models are used as metaphors, an author usually points to the similarity between one or two variables in a natural setting and one or two variables in a model. If calling attention to similarities is all that is intended by the metaphor, it serves the usual purpose of rapidly conveying information in graphic form. These three models have frequently been used metaphorically, however, for another purpose. The similarity between the many individuals jointly using a resource in a natural setting and the many individuals jointly producing a suboptimal result in the model has been used to convey a sense that further similarities are present.

Later she provides an example to illustrate how bad things can get when models become metaphors and are then used to create policy.

Relying on metaphors as the foundation for policy advice can lead to results substantially different from those presumed to be likely. Nationalizing the ownership of forests in Third World countries, for example, has been advocated on the grounds that local villagers cannot manage forests as to sustain their productivity and their value in reducing soil erosion. In countries where small villages had owned and regulated their local communal forests for generations, nationalization meant expropriation.

The three models being referred to in the first quotation are prior attempts to explain the dynamics of common resource governance, but we needn't go into any more detail than that. What I seem to have taken away from this is a broader point about the danger of attempting to fit insights onto a domain over which they were never intended to apply. It's possible I'm zooming out too far, for example one of the model-cum-metaphors Ostrom is referring to above is the prisoner's dilemma which has a precise definition, but this starts to feel close to an impulse in my field. We software developers love a structuring metaphor, such as architecture or cellular biology¹.

How do we know when we've inadvertently turned a model into a metaphor and misapplied it? When we've latched onto those couple of similar variables and missed all the dissimilar ones? Ostrom's suggestion is fairly simple to state: if your data contradicts the model you're trying to apply, either your data is wrong, the model is wrong, or the model doesn't fit your domain. This could perhaps be summarised, flippantly, as "do science." Indeed her argument against the inevitability of the tragedy of the commons is that there are ample examples of self-organizing communities who share common resources without falling prey to the effect.

I'm not exactly sure where all this leads me. Wanting to apply the scientific method to find the boundaries of the myriad structuring metaphors we use to think and communicate sure sounds exhausting; for all involved. But on the other hand we run the risk of accidentally limiting ourselves, or causing real harm, by labouring under a seemingly well justified but ill-fitting model. I genuinely have no idea what I'm supposed to do with these thoughts, but at least now I've shared my unease.

And then there is the idea of simulation itself, where the whole idea of “data” and “state” starts to get eclipsed by “competent objects” that can cooperate, much more like biological cells and human societies can.

From Alan Kay answering the question 'What thought process would lead one to invent object-oriented programming?' on Quora ↩︎

Using Nix to deploy a Haskell web app to Heroku

hi@bradparker.id.au (Brad Parker) — Sun, 16 Feb 2020 00:00:00 UTC

Let's say we've written a web app in Haskell.

$ curl http://localhost:8000/hello/World
Hello, World!

$ curl http://localhost:8000/hello/Haskell
Hello, Haskell!

It sure would be nice if we could share it with other people. Let's deploy this wonderful thing to the internet using Heroku.

Haskell and Nix

Thanks to the Nix using Haskell community packaging up Haskell libraries and executables is very convenient.

First, we'll make a directory for our project.

~ $ mkdir haskell-on-heroku
~ $ cd $_
~/haskell-on-heroku $

Then we can ask Cabal to get us started with a scaffold.

~/haskell-on-heroku $ nix run \
> nixpkgs.cabal-install \
> --command cabal init \
> --minimal \
> --cabal-version=2.4

Finally, we add a default.nix which makes use of cabal2nix.

{ haskellPackages ? (import <nixpkgs> {}).haskellPackages }:
  haskellPackages
    .callCabal2nix "haskell-on-heroku" ./. {}

And it's now ready to try out.

~/haskell-on-heroku $ nix run \
> --file default.nix \
> env --command cabal new-run
# ...
Hello, Haskell!

An amazing web application

Our web server uses the nifty Haskell library Servant server so we'll need to add it and Warp as dependancies to haskell-on-heroku.cabal.

 executable haskell-on-heroku
   main-is:             Main.hs
   build-depends:       base >=4.12 && <4.13
+                     , servant-server
+                     , warp
   default-language:    Haskell2010

We'll probably also need to put the implementation of our app in Main.hs.

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeApplications #-}
{-# LANGUAGE TypeOperators #-}
{-# OPTIONS_GHC -Wall #-}

module Main where

import Control.Applicative ((<|>))
import Data.Proxy (Proxy (Proxy))
import Network.Wai.Handler.Warp (run)
import Servant
  ( (:>),
    Capture,
    Get,
    Handler,
    PlainText,
    serve,
  )
import System.Environment (getEnv)

type Greeter =
  "hello"
    :> Capture "name" String
    :> Get '[PlainText] String

greet :: String -> Handler String
greet name =
  pure $ "Hello, " <> name <> "!"

getPort :: IO Int
getPort =
  read <$> getEnv "PORT" <|> pure 8000

main :: IO ()
main = do
  port <- getPort
  run port $ serve (Proxy @Greeter) greet

A deployable image

The Docker-using members of the Nix community have made building docker-compatible images from Nix derivations very convenient.

To try it out put the following in a file somewhere, say ~/hello.nix.

let
  nixpkgs = import <nixpkgs> {};
in
  nixpkgs.dockerTools.buildImage {
    name = "hello-docker-nix";
    tag = "latest";
    contents = [ nixpkgs.hello ];
  }

Then build it.

~ $ nix-build hello.nix

Load the result.

~ $ docker load < result

And finally run it.

~ $ docker run hello-docker-nix:latest hello
Hello, world!

We can put our package in a docker-compatible image in a similar fashion. Put the following in a file called release.nix in your haskell-on-heroku project directory.

{ nixpkgs ? import <nixpkgs> {} }:
let
  inherit (nixpkgs)
    callPackage
    dockerTools;

  package = callPackage ./. {};
in
  dockerTools.buildImage {
    name = "haskell-on-heroku";
    tag = "latest";
    contents = [
      package
    ];
  }

Using the callPackage helper means we're less likely to accidentally end up with multiple versions of the Nix packages set. If that should happen, our app would still run fine, we just might end up with a bigger than necessary image.

Let's try to build it and see how big it is.

~/haskell-on-heroku $ nix-build release.nix
# ... build output
~/haskell-on-heroku $ docker load < result
3f5a871dd9ee: Loading layer   38.1MB/38.1MB
Loaded image: haskell-on-heroku:latest

It's about 38.1 megabytes. That's not bad.

Now, I'm going to save us some trouble and add busybox to our image and also make sure it has a Cmd configured.

diff --git a/release.nix b/release.nix
index 8fa2527..e14a534 100644
--- a/release.nix
+++ b/release.nix
@@ -3,7 +3,8 @@
 let
   inherit (nixpkgs)
     callPackage
-    dockerTools;
+    dockerTools
+    busybox;

   package = callPackage ./. {};
 in
@@ -12,5 +13,9 @@ in
     tag = "latest";
     contents = [
       package
+      busybox
     ];
+    config = {
+      Cmd = ["/bin/${package.pname}"];
+    };
   }

We need to specify a command so Heroku knows how to run the container it'll create from our image. We need busybox because Heroku will attempt to run our image's command with bash -c. Images built with Nix's dockerTools are so minimal they don't have bash or even /bin/sh.

Shipping it

We now have everything we need. The last steps are to create a Heroku application, push our image to the Heroku registry and release it.

First, assuming you already have a Heroku account, log into the Heroku command line.

~/haskell-on-heroku $ nix run nixpkgs.heroku \
> --command heroku login

Then also login into the container registry.

~/haskell-on-heroku $ nix run nixpkgs.heroku \
> --command heroku container:login

Next create an app.

~/haskell-on-heroku $ nix run nixpkgs.heroku \
> --command heroku create

Build our image.

~/haskell-on-heroku $ nix-build release.nix

Load it.

~/haskell-on-heroku $ docker load < result

Tag it with our Heroku application's name and the process type we'd like to run it as.

~/haskell-on-heroku $ docker tag \
> haskell-on-heroku \
> registry.heroku.com/infinite-anchorage-09330/web

Push it to the registry.

~/haskell-on-heroku $ docker push \
> registry.heroku.com/infinite-anchorage-09330/web

And, lastly, release it.

~/haskell-on-heroku $ nix run nixpkgs.heroku \
> heroku container:release -a infinite-anchorage-09330 web

Hey presto, our amazing application will now be accessible by the whole world. How cool.

That does seem like a lot to remember, however, so we'd best pop as much of it as we can in a script. A script we might call ./deploy.

#!/usr/bin/env bash

set -exu

app_name=$1
result=$(nix-build --no-out-link release.nix)

docker load < $result
docker tag haskell-on-heroku registry.heroku.com/$app_name/web
docker push registry.heroku.com/$app_name/web

nix run nixpkgs.heroku --command \
  heroku container:release -a $app_name web

With that building and deployment is a little more manageable.

~/haskell-on-heroku $ ./deploy infinite-anchorage-09330

I've prepared a repository of everything we've gone through here as a reference. With any luck it may help someone get their awesome Haskell application onto the mad expanse that is the internet.

Home sweet home

hi@bradparker.id.au (Brad Parker) — Wed, 12 Feb 2020 00:00:00 UTC

The operations practice of defining infrastructure as code has been popular for a while, and it’s for good reason. Lot's of strange and unexpected things can happen to our very important web-servers, so being able to quickly build another one in a (close to) identical state can be very handy. Sadly, strange and unexpected things don't just happen to web-servers, they can happen our development machines too. And, heck, they're also pretty important.

Like infrastructure, can we define some of the state of our laptops using code? I'm not sure, but I think so, at least I'll explain what I've tried.

Bootstrapping scripts

First, I used a lightly modified fork of Ryan Bates' dot files which bootstraps itself using a lovely rake task.

This went pretty well. I didn't have to bootstrap new machines too often, I didn't really have many preferences regarding command line tools, and I only had so many technology stacks to be setup for.

The trouble came when I wanted to change things. It wasn't always clear to me that the changes I made would be applied properly when I ran the script on a fresh machine. It always seemed that I missed something and had to do some manual work in addition to the rake task.

The only part that I found to be very predicable were the files that were symlinked into my home directory. So after a while I cut the repository down to nothing but those.

Staying fresh

A colleague introduced a few us at Everydayhero to Fresh. It's great and with it I just started to feel that my development machine had become easily recoverable. I could run fresh on a minimally setup Macbook as well as on one I'd been working away on for months and the results were, more or less, the same.

Even though I still didn't do all that much with Fresh that couldn't be achieved with symlinking I did begin to see the potential.

Despite this I was still looking for something more complete. Fresh managed a lot, but it didn't manage everything. The plurality of thing managers I was using became apparent when I finally documented how I'd been setting everything up.

Home manager

I'd been using Nix as a Homebrew replacement for a while before finding Home manager. It's given me the ability to consolidate almost everything I need to setup and update my development machines into one command.

Apart from simplifying the process of setting up a new machine, Home manager has also allowed to me to manage more stuff more predictably.

Utilities

There are a few non-default command line utilities I find myself always installing. Before Home manager if I wanted to setup a new machine I'd usually be moving lists of currently installed dependencies around.

brad@old $ brew list > deps.txt

brad@new $ xargs brew install < deps.txt

I'd moved to Nix before becoming aware of Homebrew Bundle's Brewfile, which looks like a great solution. Just not quite as holistic as I've found Nix / Home manager to be and with it my list of utilities is easy to manage.

Vim

Before Home manager I'd used a few of the popular Vim plugin managers (Pathogen, Vundle and Plug). They all worked well for me, but with my current setup I get a good plugin manager for Vim without an extra tool or installation step.

Tmux

I've only ever used two Tmux plugins. Initially I installed them with TPM but now I install them with Home manager (Tmux sensible gets included by default).

Bash

Sometimes it's nice to string utilities together into useful aliases, or try to bring a little more consistency to working across both macOS and Linux. Sadly, those niceties can get out of hand, so I've found it useful to factor them out into bite-sized chunks.

Git

I get a lot of use from Git bash completions and a few handy aliases.

I get all of that, and a few other things, installed and configured with one tool. What's more:

It's repeatable, if my environment bootstraps once I'm confident it'll do so again.
It's idempotent, I can keep running home-manager switch as often as I like without issues.
It works, more or less, the same on both my work machines running macOS and my home laptop running PureOS, a Debian fork. Because it's so repeatable I'm able to set up a build pipeline which ensures that every update I make will install happily on both platforms. I accept that this is perhaps taking things too far.

Now that I've said that, and now that I'm pretty comfortable with the approach, I expect I'll push it a little further. Why not?

Getting close to the conceptual metal

hi@bradparker.id.au (Brad Parker) — Mon, 27 Jan 2020 00:00:00 UTC

A few years ago I watched the Structure and interpretation of computer programs video lectures, kindly posted publicly by MIT OpenCourseWare. There's so much interesting material in them but one part really struck me, in 5B: Computational Objects Gerald Jay Sussman defines a pair "in terms of nothing but air, hot air".

(define (cons x y)
  (λ (m) (m x y)))

(define (car x)
  (x (λ (a b) a)))

(define (cdr x)
  (x (λ (a b) b)))

Previous lectures had established pairs as a sort of fundamental thing. After all, pairs could be used to make lists and in Lisp once you've got pairs and lists you've got a lot. Sussman, however, was revealing that there was a level below that, there was an even more fundamental thing: functions.

Later, through learning Haskell, I became familiar with the concept of algebraic data types. It was astonishing to me that you could make so much out of so little. You seemingly only need types that are "pair-like" (products) and types that are "either-like" (sums).

data Product a b
  = P a b

data Sum a b
  = A a
  | B b

newtype Maybe a
  = Maybe (Sum () a)

newtype List a
  = List (Maybe (Product a (List a)))

newtype NonEmpty a
  = NonEmpty (Product a (List a))

When Sussman chalked up that pair made of nothing but "hot air" he attributed the encoding to Alonzo Church. Appreciating that many other types can be made from pair-like things and either-like things, I wondered if there existed a Church encoding for Either. After all, with Either and Pair it sure seemed like I'd be able to construct anything else I needed. I found Church encodings for lots of other interesting things, like booleans and natural numbers, but nothing quite like Either.

Eventually I'd happen across Scott encoding, named for Dana Scott to whom it is attributed. It's super interesting.

Scott encoding

With Scott encoding we have a consistent method for representing any algebraic datatype in the untyped lambda calculus. It goes something like this: say we have a datatype D, with N constructors, (d₁ … d_N), such that constructor d_i has arity A_i. The Scott encoding for constructor d_i of D would be:

d_i := λx₁ … x_{A_i}. λc₁ … c_N. c_i x₁ … x_{A_i}

If you've never seen this sort of notation before don't worry we'll be moving to mostly Haskell soon enough. Until then we could have a couple of swings at stating this in words.

Each constructor, d₁ through d_N, is responsible for accepting the arguments needed to call its chosen continuation, any one of c₁ through c_N.
There is one continuation per constructor (c_i is the continuation for d_i), and each continuation needs to be passed all the arguments its constructor was passed (d_i was passed x₁ … x_{A_i} and so c_i will passed them as well).

In this way values are held in the closure of each constructor. These internal values are used when the consumer of a value of type D passes in a continuation which will accept them as arguments.

To develop an intuition for how this actually works we can try converting some well-known Haskell data types to this encoding.

Pair

To Scott encode any one constructor for a given datatype we need to know how many other constructors it has, and how many arguments each of them accept. Haskell's pair type looks like syntax, however by asking GHCi for some information about it we can see that its much like any other user-defined datatype.

$ ghci
> :info (,)
data (,) a b = (,) a b  -- Defined in `GHC.Tuple'

It has one constructor which happens to share the name of its type.

> :type (,)
(,) :: a -> b -> (a, b)

For our purposes we'll rename the (,) type to Pair and the (,) constructor to pair. As Pair's only constructor, pair need only accept one continuation c. The arguments that pair accepts, such that it can call c with them, are the resulting Pair's first and second elements.

pair := λx₁, x₂. λc. c x₁ x₂

But what is Pair? How might we write the type of a function that accepts a Pair? First we might rewrite pair in Haskell, initially as a fairly direct translation.

pair = \x1 x2 -> \c -> c x1 x2

Then, without the first explicit lambda.

pair x1 x2 = \c -> c x1 x2

That can be loaded into GHCi or into a Repl.it Haskell session in order to find its inferred type.

$ ghci
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from /home/brad/.ghci
> pair x1 x2 = \c -> c x1 x2
> :type pair
pair :: t1 -> t2 -> (t1 -> t2 -> t3) -> t3

You might be wondering why these lambdas are arranged the way they are. Why not use any of the following, equivalent, arrangements.

pair = \x1 -> \x2 -> \c -> c x1 x2
pair = \x1 -> \x2 c -> c x1 x2
pair = \x1 x2 c -> c x1 x2

The answer is that I'm sneakily trying to reveal what the Pair component of pair's type is, and, as it ends up, what the datatype component of any Scott encoded constructor's type happens to be.

type Pair t1 t2 t3
  = (t1 -> t2 -> t3) -> t3

The equivalent Haskell type, (,) a b has one less type variable. It's interesting to think about what it would mean for Pair to ditch t3 somehow.

type Pair t1 t2
  = (t1 -> t2 -> _) -> _

One way of looking at this is that it's up to the continuation (c) what it does with x₁ and x₂, the constructor (pair) doesn't get a say. This means that whatever pair returns must work for anything its consumer may want to produce. In Haskell types we might say that Pair must work for all possible types its provided continuation could return.

type Pair t1 t2
  = forall t3. (t1 -> t2 -> t3) -> t3

pair :: Pair a b
pair a b = \c -> c a b

Is this equivalent to (,)? How might we tell? One way might be to write a function which converts (,)s into Pairs and one which converts Pairs into (,)s. Then, given those two functions, we should be able to observe that their composition doesn't do anything. Or put another way: for these two functions:

from :: (a, b) -> Pair a b
from = _

to :: Pair a b -> (a, b)
to = _

Their composition in either direction is equivalent to an identity function.

from ∘ to = id_Pair
to ∘ from = id_(,)

If these two functions can be written, and they have this property, it means that Pair and (,) are isomorphic.

So then, is it possible to write from?

from :: (a, b) -> Pair a b
from (a, b) = _

Which is sort of asking: given an a and a b, is it possible to construct a Pair a b?

from :: (a, b) -> Pair a b
from (a, b) = pair a b

How about to?

to :: Pair a b -> (a, b)
to p = _

Recall that p is a function waiting to be passed a continuation. That continuation will be passed both an a and a b.

to :: Pair a b -> (a, b)
to p = p (\a b -> _)

So now, given an a and a b can we construct an (a, b)?

to :: Pair a b -> (a, b)
to p = p (\a b -> (a, b))

With that done: is to ∘ from an identity function? It does seem so.

> (to . from) (1, 2)
(1,2)
> (to . from) (True, "Nifty")
(True,"Nifty")

Without a Show or Eq instance for (->) it's hard to observe the same for from ∘ to. We can at least observe what to ∘ from ∘ to does.

> (to . from . to) (pair 1 2)
(1,2)
> (to . from . to) (pair True "Very nifty")
(True,"Very nifty")

Well, I'm convinced. Are you? For further tangible evidence you could try re-implementing everything in Data.Tuple in terms of Pair. It's a pretty fun exercise and helped me to become more comfortable with these lambda-only pairs.

Either

In Haskell Either is defined like this:

data Either a b
  = Left a
  | Right b

For the purposes of Scott encoding we can say that Either has two constructors which each accept one argument. This means that whatever each constructor returns will accept two continuations which themselves each accept one argument. Each constructor will accept an argument to pass to their chosen continuation.

The first constructor left, then, will accept one argument to pass to the first continuation.

left := λx. λc₁, c₂. c₁ x

And the second will accept one to pass to the second continuation.

right := λx. λc₁, c₂. c₂ x

Each continuation can accept an argument of a different type (say c₁ takes an α and c₂ takes a β) but both should return something of the same type (say γ). The reason for this is how values of the Either type are used, this may become evident later. Using a polymorphic lambda calculus rather than an untyped lambda calculus we can write out how those types line up.

left := Λα, β, γ. λx ^α. λc₁ ^{α → γ}, c₂ ^{β → γ}. c₁ x
right := Λα, β, γ. λx ^β. λc₁ ^{α → γ}, c₂ ^{β → γ}. c₂ x

For me at least, it's a little easier to see in Haskell.

left :: a -> (a -> c) -> (b -> c) -> c
left x = \c1 c2 -> c1 x

right :: b -> (a -> c) -> (b -> c) -> c
right x = \c1 c2 -> c2 x

As we did with Pair, we might ask: where's Either in all of that? And, as with Pair, I have sneakily positioned certain lambdas to try and point it out. However, in this case another method might be to put left and right side by side and view their shared return type.

left  :: a -> ((a -> c) -> (b -> c) -> c)
right :: b -> ((a -> c) -> (b -> c) -> c)

Which might yield the following.

type Either a b c
  = (a -> c) -> (b -> c) -> c

As with Pair we can make sure that the choice of c is left solely up to the consumer of an Either.

type Either a b
  = forall c. (a -> c) -> (b -> c) -> c

left :: a -> Either a b
left x = \c1 c2 -> c1 x

right :: b -> Either a b
right x = \c1 c2 -> c2 x

Can we write to and from for converting between our Either and Haskell's?

from :: Prelude.Either a b -> Either a b
from = _

to :: Either a b -> Prelude.Either a b
to = _

For from we can be given a Left containing an a or a Right containing a b.

from :: Prelude.Either a b -> Either a b
from e =
  case e of
    Left a -> _
    Right b -> _

So we have two questions to answer. Given an a can I construct an Either a b?

from :: Prelude.Either a b -> Either a b
from e =
  case e of
    Left a -> left a
    Right b -> _

Given a b can I construct an Either a b?

from :: Prelude.Either a b -> Either a b
from e =
  case e of
    Left a -> left a
    Right b -> right b

For to it would be nice if we could use a case expression as we did with from and in a sense we can.

to :: Either a b -> Prelude.Either a b
to e =
  e
    (\a -> _)
    (\b -> _)

This was less clear when looking at Pair and (,) but Scott-encoded datatypes are almost ready-made case expressions. Each continuation is like a constructor-only pattern-match.

As with to we again have two questions to answer, and they're very similar. First, given an a can I construct a Prelude.Either a b?

to :: Either a b -> Prelude.Either a b
to e =
  e
    (\a -> Left a)
    (\b -> _)

Next, given a b can I construct a Prelude.Either a b?

to :: Either a b -> Prelude.Either a b
to e =
  e
    (\a -> Left a)
    (\b -> Right b)

Are Prelude.Either and Either equivalent?

> to (from (Left "Hrm?"))
Left "Hrm?"
> to (from (to (left "Ahhhhh")))
Left "Ahhhhh"
> to (from (Right 2))
Right 2
> to (from (to (right 14)))
Right 14

Looks likely. Re-implementing everything in Data.Either would sure be convincing.

From hot air to solid ground

This is all well and good but I'm still left wondering what could actually be built with this. Knowing, intellectually, that Haskell's Either and the lambdas-only Either are equivalent isn't as satisfying as using Either and Pair to write real(ish) programs.

We could use Either and Pair to write a little REPL which evaluates Lisp-like expressions.

$ runhaskell HotAir.hs
Enter an expression to evaluate. E.G.
(+ (* (+ 12 7) 100) 58)
> (+ 1 2)
3
>

Yeah, let's do that.

To make things less onerous this little language could be limited to only adding and multiplying natural numbers. No define, no λ. Nothing other than positive integer literals (0, 1, 2 …), addition (+) and multiplication (*).

> (define (sqr x) (* x x))
Error: Unexpected char: 'd'
>

Before we begin: this is going to move quite fast. Don't worry if you miss some of the details of the implementation. It's more important to be generally aware of what the program does and to keep in mind that we're building it as a bit of a stress test for our lambda-only datatypes. If you'd like to learn more about how to build programs in Haskell I'd reccomend all the material produced by Typeclasses and the Data61 Functional Programming Course.

Now, the first thing we should do is redefine Pair and Either using newtype, rather than type (GHC is pretty great, but sometimes we need to cut it some slack).

{-# LANGUAGE BlockArguments #-}
{-# LANGUAGE RankNTypes #-}
{-# LANGUAGE NoImplicitPrelude #-}
{-# OPTIONS_GHC -Wall -Werror #-}

module HotAir where

newtype Pair a b = Pair
  { runPair ::
      forall c.
      (a -> b -> c) ->
      c
  }

pair :: a -> b -> Pair a b
pair x1 x2 = Pair \c -> c x1 x2

newtype Either a b = Either
  { runEither ::
      forall c.
      (a -> c) ->
      (b -> c) ->
      c
  }

left :: a -> Either a b
left x = Either \c1 _ -> c1 x

right :: b -> Either a b
right x = Either \_ c2 -> c2 x

Next, we might define a monadic parser.

import Data.String (String)

newtype Parser a = Parser
  { parse ::
      String ->
      Either String (Pair a String)
  }

A satisfy function for creating a Parser which compares the next character using given Char -> Bool predicate would be handy.

{-# LANGUAGE LambdaCase #-}

import Data.Bool (Bool)
import Data.Char (Char)
import Data.List ((++))

satisfy :: (Char -> Bool) -> Parser Char
satisfy p = Parser \case
  [] -> left "Unexpected EOF"
  (c : cs) ->
    if p c
      then right (pair c cs)
      else left ("Unexpected char: " ++ show c)

We can use it to define a char function which matches a specific Char.

import Data.Eq ((==))

char :: Char -> Parser Char
char = satisfy . (==)

Functor, Applicative and Alternative instances for Parser will make composing and manipulating them nicer.

import Control.Applicative
  ( Alternative (empty, (<|>)),
    Applicative (pure, (<*>)),
  )
import Data.Function (const)
import Data.Functor (Functor (fmap))

instance Functor Parser where
  fmap :: (a -> b) -> Parser a -> Parser b
  fmap f pa =
    Parser \str ->
      runEither (parse pa str)
        left
        (\p -> right (runPair p (pair . f)))

instance Applicative Parser where
  pure :: a -> Parser a
  pure a = Parser (right . pair a)

  (<*>) :: Parser (a -> b) -> Parser a -> Parser b
  pf <*> pa =
    Parser \str ->
      runEither (parse f str)
        left
        (\p -> runPair p (\f -> parse (fmap f pa)))

instance Alternative Parser where
  empty :: Parser a
  empty = Parser (const (left "Empty"))

  (<|>) :: Parser a -> Parser a -> Parser a
  Parser l <|> Parser r =
    Parser \str ->
      runEither (l str)
        (const (r str))
        right

A convenient way to ignore unused input will be useful, too.

fst :: Pair a b -> a
fst = (`runPair` const)

execParser :: Parser a -> String -> Either String a
execParser p str =
  runEither (parse p str)
    left
    (right . fst)

That should be enough to write an eval function for our little language.

import Control.Applicative (some, (*>), (<*))
import Data.Char (isDigit)
import Data.Functor ((<$), (<$>))
import GHC.Num ((*), (+))
import Numeric.Natural (Natural)

eval :: String -> Either String Natural
eval = execParser expression
  where
    expression :: Parser Natural
    expression = natural <|> application
    natural :: Parser Natural
    natural = read <$> some (satisfy isDigit)
    application :: Parser Natural
    application =
      char '(' *> operator <* space
        <*> expression <* space
        <*> expression <* char ')'
    operator :: Parser (Natural -> Natural -> Natural)
    operator = (+) <$ char '+' <|> (*) <$ char '*'
    space :: Parser Char
    space = char ' '

Finally, a main function that lets a user input expressions and view the result of evaluating them.

module HotAir (main) where

import Control.Monad (forever)
import System.IO
  ( IO,
    getLine,
    print,
    putStr,
    putStrLn,
  )

main :: IO ()
main = do
  putStrLn "Enter an expression to evaluate. E.G."
  putStrLn "(+ (* (+ 12 7) 100) 58)"
  forever $ do
    putStr "> "
    line <- getLine
    runEither (eval line)
      (putStrLn . ("Error: " ++))
      print

And here it is, a little REPL made of mostly hot air.

{-# LANGUAGE BlockArguments #-}
{-# LANGUAGE InstanceSigs #-}
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE RankNTypes #-}
{-# LANGUAGE NoImplicitPrelude #-}
{-# OPTIONS_GHC -Wall -Werror #-}

module HotAir (main) where

import Control.Applicative
  ( Alternative (empty, (<|>)),
    Applicative (pure, (<*>)),
    some,
    (*>),
    (<*),
  )
import Control.Monad (forever)
import Data.Bool (Bool)
import Data.Char (Char, isDigit)
import Data.Eq ((==))
import Data.Function (const, ($), (.))
import Data.Functor (Functor (fmap), (<$), (<$>))
import Data.List ((++))
import Data.String (String)
import GHC.Num ((*), (+))
import Numeric.Natural (Natural)
import System.IO
  ( IO,
    getLine,
    print,
    putStr,
    putStrLn,
  )
import Text.Read (read)
import Text.Show (Show (show))

newtype Pair a b = Pair
  { runPair ::
      forall c.
      (a -> b -> c) ->
      c
  }

pair :: a -> b -> Pair a b
pair x1 x2 = Pair \c -> c x1 x2

fst :: Pair a b -> a
fst = (`runPair` const)

newtype Either a b = Either
  { runEither ::
      forall c.
      (a -> c) ->
      (b -> c) ->
      c
  }

left :: a -> Either a b
left x = Either \c1 _ -> c1 x

right :: b -> Either a b
right x = Either \_ c2 -> c2 x

newtype Parser a = Parser
  { parse ::
      String ->
      Either String (Pair a String)
  }

satisfy :: (Char -> Bool) -> Parser Char
satisfy p = Parser \case
  [] -> left "Unexpected EOF"
  (c : cs) ->
    if p c
      then right (pair c cs)
      else left ("Unexpected char: " ++ show c)

char :: Char -> Parser Char
char = satisfy . (==)

instance Functor Parser where
  fmap :: (a -> b) -> Parser a -> Parser b
  fmap f pa =
    Parser \str ->
      runEither (parse pa str)
        left
        (\p -> right (runPair p (pair . f)))

instance Applicative Parser where
  pure :: a -> Parser a
  pure a = Parser (right . pair a)

  (<*>) :: Parser (a -> b) -> Parser a -> Parser b
  pf <*> pa =
    Parser \str ->
      runEither (parse pf str)
        left
        (\p -> runPair p (\f -> parse (fmap f pa)))

instance Alternative Parser where
  empty :: Parser a
  empty = Parser (const (left "Empty"))

  (<|>) :: Parser a -> Parser a -> Parser a
  Parser l <|> Parser r =
    Parser \str ->
      runEither (l str)
        (const (r str))
        right

execParser :: Parser a -> String -> Either String a
execParser p str =
  runEither (parse p str)
    left
    (right . fst)

eval :: String -> Either String Natural
eval = execParser expression
  where
    expression :: Parser Natural
    expression = natural <|> application
    natural :: Parser Natural
    natural = read <$> some (satisfy isDigit)
    application :: Parser Natural
    application =
      char '(' *> operator <* space
        <*> expression <* space
        <*> expression <* char ')'
    operator :: Parser (Natural -> Natural -> Natural)
    operator = (+) <$ char '+' <|> (*) <$ char '*'
    space :: Parser Char
    space = char ' '

main :: IO ()
main = do
  putStrLn "Enter an expression to evaluate. E.G."
  putStrLn "(+ (* (+ 12 7) 100) 58)"
  forever $ do
    putStr "> "
    line <- getLine
    runEither (eval line)
      (putStrLn . ("Error: " ++))
      print

You can save that code into a file and execute it with runhaskell if you want to see it in action.

$ runhaskell HotAir.hs
Enter an expression to evaluate. E.G.
(+ (* (+ 12 7) 100) 58)
> (+ (* (+ 12 7) 100) 58)
1958
>

That was quite a whirlwind.

I know I really shouldn't be surprised that I could write my program with (more or less) only lambdas. Lambda calculus is "a universal model of computation" after all. I also shouldn't be surprised that translating lambda calculus terms into Haskell is so natural as Haskell based on a lambda calculus. But in this post I really wanted to focus on the wonder I feel when being able to use abstract notions like lambda calculus to write runnable programs. My hope is that by writing this I might help someone find something they find wonderful, and encourage them to explore it.

One possible avenue, which I might go into in a future post, is this: the above program still makes use of some pre-defined Haskell datatypes like Bool, Natural, [a] and Char. Does it have to? Could we build those entirely out of lambdas too? How far can we go?

Huge thanks to Brad Bow for his clarifying suggestions.

Deploying a fully automated Nix-based static website

hi@bradparker.id.au (Brad Parker) — Thu, 21 Nov 2019 00:00:00 UTC

My website has changed a bit over time. It was initially a WordPress blog, contorted to do double-duty as a folio for my design work. More recently it's been a static site, first generated by Metalsmith and later Hakyll. About a month ago I started to feel that hosting my site on Github pages was a little disconnected. I know this sounds odd, but I wanted to have a more active role in its deployment and take more responsibility for how it's run. Silly, right?

Well, now here we are. If you're reading this at bradparker.com then you're getting the full experience. It's being served from a Digital Ocean droplet, running NixOs which has been configured to run a Warp web-server which is serving a static site built by Hakyll.

While researching how I might set all of this up I came across a great article on the Digital Ocean blog entitled Deploying a Fully-automated Git-based Static Website in Under 5 Minutes. The result seemed almost exactly what I wanted, a Github-pages-like experience but managed by me. The only thing that I felt it needed was a slightly more declarative approach.

First, a Droplet

An important attribute of this setup for me is reproducibility. I should be able to stand a new server up in functionally same state as the current one by running one command. I don't want to have to access to server directly and configure anything by running a sequence of steps. This includes things like the security best practices outlined in another DigitalOcean blog article referenced in the aforementioned.

To achieve this my server is running NixOs, a "completely declarative" operating system built on top of the Nix package manager. Nix is one of those very powerful and very mysterious things, it's hard to fully explain because it can do so much. I use it as much as I can get away with, it configures my work and personal machines and now the server which hosts my website.

Chris Martin has a great post which outlines how to get a DigitalOcean droplet running NixOs. It is possible to extend those instructions slightly and get something more automated. To do this I took some advice from the NixOs Infect README and made use of the fact that DigitalOcean images come with cloud-init installed. This makes it possible to run NixOs infect on the initial boot of a droplet without ever having to SSH into it. Using doctl you can create a droplet and have NixOs installed on it with one command.

#!/usr/bin/env bash

set -e

doctl compute droplet create \
  "bradparker.com" \
  --size s-1vcpu-3gb \
  --image ubuntu-16-04-x64 \
  --region sgp1 \
  --user-data "$(cat <<-EOF
    #cloud-config
    write_files:
    - path: /etc/nixos/host.nix
      permissions: '0644'
      content: |
        {pkgs, ...}:
        {
          # TODO: ... everything
        }
    runcmd:
      - curl https://raw.githubusercontent.com/elitak/nixos-infect/master/nixos-infect | PROVIDER=digitalocean NIXOS_IMPORT=./host.nix NIX_CHANNEL=nixos-19.09 bash 2>&1
    EOF
    )"

Now, the above command would not result in anything very useful. It's not running my website, and if I wanted to get the rest working I wouldn't even be able to SSH in to continue manually. I'm going to need a user who can SSH in and run commands as root if needs be.

{ config, pkgs, ... }:
{
  services.openssh = {
    passwordAuthentication = false;
    permitRootLogin = "no";
  };

  users.mutableUsers = false;

  users.users.brad = {
    isNormalUser = true;
    hashedPassword = "$1$cmckqoXC$eBya/upETQKFbInZPz5y8.";
    home = "/home/brad";
    description = "Brad Parker";
    extraGroups = [
      "wheel"
    ];
    openssh.authorizedKeys.keys = [
      "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD8pPHsHkSNeX+YTVfbmrMltnWs+6dejWClomo2jBQuSv93WAzChWA1lhh8rUo9yxlud4FSV2iJ+3qQ6lpxgt8Dd9rS+xa8SS1rga/nTG5eqJqfq1fDyKXF+TpbHbNO09QdWZjRTcvv1DNCd51FyWUsFojECjoY0KmyjxJCmVyKgdvWjW/9DHiP0a1MS1ILtREr873D5SiQlKKj8T1AaVs7tToZNtZWxE2U4L4ibWYiWoJnVMi85t0nGj2NNBsVJsYhAb5fh5Uj5La9R3nR7RrkCLCHOKzuZu6VkWIQcObFCgb5G80DDt9Vp8uiNDCMwMLZA2HPi4CpGjWEfgHbvYEbgRNZIQcJhid12HscjEChLN4uvdp+9TiQwFsTm3kSBgflERFOdX4qbJMy/XQz8FEgUb2E8/lo3P3DMUeYhNXNGO6jmBvknNBv6XXjhhOFBa9xmT1jnjakSpKBWvICIWqLFmi8MByldqTW7oJKNpDH4qI4ML0goPaZ9ketlGs8PUwr1d4bCR/Dm79VTuiiUNw+i0J4sQrs2ufkEtuBQfBmHywVhl/sDS/3A7vUZWZ+2Cb+9qssAGjEy424anWRdkFx/x+m+9Zreb2tkXDlofEd18fCNbqAZzVTYAoDlQUmen1IJOEhWPpC724p3wyv8Y2SN1pMunPQ7ThmuPEB/hOSEQ== hi@bradparker.com"
    ];
  };
}

The above file describes a Nix module which will create a user able to log into the machine using SSH. I've also made sure that only key pairs can be used for SSH authentication and it's not possible to log in directly as the root user. In order to run commands as root I first have to SSH in and then elevate my privileges using sudo. In order to do this I've given my user a password by providing the hashedPassword option. I generated a value for this by using openssl on my local machine.

$ openssl passwd -1
Password: <Type a password>
Verifying - Password: <Type it again>
$1$JO1.bEGj$69.Gdj1/utBcmwRJYGu741

The mutableUsers option means that users or groups created with useradd or groupadd won't be persisted when Nix rebuilds the system.

Now, I'm going to want to update an existing server after it's been created from time to time. The way I do this is by modifying a local host.nix file, rsync-ing it up onto the server, using ssh to move it to /etc/nixos/ as root and running nixos-rebuild switch.

#!/usr/bin/env bash

set -exo pipefail

rsync host.nix bradparker.com:host.nix
ssh -t bradparker.com \
  'sudo mv host.nix /etc/nixos/host.nix && sudo nixos-rebuild switch'

The whole server is administered using a local host.nix file and the above script.

Next, a static site

For the static site I wanted the "push to deploy" experience of Github pages. To achieve this firstly I built a Nix derivation which describes, in full, how to build my static site. That derivation depends on another one which describes, in full, how to build the builder for my site. As an aside, herein lies one of the killer features of Nix over similar tools: it composes. I'll only need to say that I want the site built and builder will be brought along for the ride. Once I had a derivation for my site I then defined a systemd service to build it every five minutes.

{ options, lib, config, pkgs, ... }:
let
  serverName = "bradparker.com";
  webRoot = "/var/www/${serverName}";

  serviceConfig = config.services."${serverName}";
  options = {
    enable = lib.mkEnableOption "${serverName} service";
  };
in
  {
    options.services.${serverName} = options;
    config = lib.mkIf serviceConfig.enable {
      systemd.services."source-${serverName}" = {
        description = ''
          https://${serverName} source
        '';
        serviceConfig = {
          Type = "oneshot";
        };
        startAt = "*:0/5";
        path = with pkgs; [ nix gnutar gzip curl jq ];
        script = ''
          set -ex

          rev=$(curl https://api.github.com/repos/bradparker/bradparker.com/git/ref/heads/source | jq -r .object.sha)
          result=$(nix-build https://github.com/bradparker/bradparker.com/archive/$rev.tar.gz -A bradparker-com.site)

          ln -sfT $result${webRoot} ${webRoot}
        '';
      };
    };
  }

I curl the Github API to get the revision at the HEAD of my "source" branch, and call nix-build directly on the archive tarball for that revision. The web root symlink is then updated to point at the result. Fetching the latest revision rather than just using the tarball URL for the "source" branch means that Nix can still cache aggressively.

To make it possible to update my host configuration and this service configuration separately I've put the service configuration in a separate file (module.nix) that I fetch and import in host.nix.

{ config, pkgs, ... }:
let
  bradparker-source = builtins.fetchTarball {
    url = https://github.com/bradparker/bradparker.com/archive/355cf315becbf39ad7d3d032a88fe913ea0c4565.tar.gz;
  };
in
{
  imports = ["${bradparker-source}/module.nix"];

  # See above for user config.

  services."bradparker.com".enable = true;
}

Finally, a web server

At this stage the sensible thing to have done would have been to use the excellent Nginx NixOs module. I'd have auto-renewing SSL certificates (ACME) and a speedy web server in six lines of code. But I didn't do that, I wanted to see what it'd be like to use the Haskell web-server Warp to serve my site, even including painless HTTPS.

I needed to write some Haskell for both the application to serve the static site and one to serve ACME challenges in order to get auto-renewing certificates. Being that I love writing Haskell this was OK with me.

With those written I then needed some systemd services to run them. One for the static site.

{ options, lib, config, pkgs, ... }:
let
  package = import ./.;
  server = package.bradparker-com.server;

  serverName = "bradparker.com";
  webRoot = "/var/www/${serverName}";

  serviceConfig = config.services."${serverName}";
  options = {
    enable = lib.mkEnableOption "${serverName} service";
  };
in
  {
    options.services.${serverName} = options;
    config = lib.mkIf serviceConfig.enable {
      # ... Source service definition

      systemd.services.${serverName} = {
        wantedBy = [ "multi-user.target" ];
        wants = [
          "acme-${serverName}.service"
          "acme-challenge-${serverName}.service"
        ];
        requires = ["source-${serverName}.service"];
        script = ''
          ${server}/bin/server \
            --port 443 \
            --directory /var/www/${serverName} \
            --https-cert-file /var/lib/acme/${serverName}/fullchain.pem \
            --https-key-file /var/lib/acme/${serverName}/key.pem
        '';
        description = ''
          https://${serverName}
        '';
        serviceConfig = {
          KillSignal="INT";
          Type = "simple";
          Restart = "on-abort";
          RestartSec = "10";
        };
      };
    };
  }

Note I've put all the Nix derivations into a set so I can import them as a whole.

And a service for the ACME challenges.

{ options, lib, config, pkgs, ... }:
let
  package = import ./.;
  acme = package.bradparker-com.acme;

  serverName = "bradparker.com";
  acmeWebRoot =  "/var/lib/acme/acme-challenge";

  serviceConfig = config.services."${serverName}";
  options = {
    enable = lib.mkEnableOption "${serverName} service";
  };
in
  {
    options.services.${serverName} = options;
    config = lib.mkIf serviceConfig.enable {
      # ... Source service definition
      # ... Static site server service definition

      systemd.services."acme-challenge-${serverName}" = {
        wantedBy = [ "multi-user.target" ];
        script = ''
          ${acme}/bin/acme \
            --port 80 \
            --directory ${acmeWebRoot}
        '';
        description = ''
          The acme challenge server
        '';
        serviceConfig = {
          KillSignal="INT";
          Type = "simple";
          Restart = "on-abort";
          RestartSec = "10";
        };
      };
    };
  }

To go with that I needed to enable and configure the ACME service itself.

{ options, lib, config, pkgs, ... }:
let
  serverName = "bradparker.com";
  acmeWebRoot =  "/var/lib/acme/acme-challenge";

  serviceConfig = config.services."${serverName}";
  options = {
    enable = lib.mkEnableOption "${serverName} service";
  };
in
  {
    options.services.${serverName} = options;
    config = lib.mkIf serviceConfig.enable {
      # ... Source service definition
      # ... Static site server service definition
      # ... ACME challenge server service definition

      security.acme.certs = {
        ${serverName} = {
          email = "hi@bradparker.com";
          webroot = "${acmeWebRoot}";
          extraDomains = { "bradparker.com.au" = null; };
          postRun = "systemctl restart ${serverName}.service";
        };
      };
    };
  }

Finally, I needed to open the default HTTP and HTTPS ports in NixOs' firewall. I did this in the host.nix file rather than the module file.

{ config, pkgs, ... }:
let
  # ... Fetching source
in
{
  # ... User config, importing the module

  networking.firewall.allowedTCPPorts = [ 80 443 ];

  # ... Enabling the service
}

Hey, presto

That's it. I'm pretty happy with the result thus far and will likely continue to tweak it. If you haven't yet checked out Nix or NixOs I encourage you to do so, I think it's great.

Servant's type-level domain specific language

hi@bradparker.id.au (Brad Parker) — Sat, 5 Oct 2019 00:00:00 UTC

This post is about some reasonably advanced type-level features of The Glasgow Haskell Compiler and as such I assume some knowledge of Haskell. Despite this I've made an attempt to link to further resources on Haskell features as I introduce them. My hope is that even if everything here doesn't make perfect sense then at least some part of it might still be helpful.

I was shown Servant by a friend of mine not long after I'd started to learn Haskell. It was still new to me but one of the things that had really grabbed me about Haskell was its transparency. Without being very familiar with the language I found that I could tease out how some function or data structure worked just by poking around. Now I've spent a little time building stuff with Servant I'd like to see what can be learned by having a closer look at some of its functions and data structures. It's an interesting library, I think this'll be fun.

Servant has you define your API as a type. You're not expected to define a wholly new type, but rather combine existing types provided by the framework. These add up to a domain specific language, at the type level, for describing web APIs. This was quite a mental shift for me, that the type comes first, and drives the implementation. It's just so great that Servant's DSL is expressive enough to describe almost any API you might want to implement.

Our aim here will be to understand how Servant can take so many varied API descriptions and guide us to a corresponding implementation.

The example type

The example we'll use is close to the one used in Servant's introductory tutorial. We're going to describe an API which has two endpoints: GET /users which returns a JSON-encoded list of users and GET /users/:username which returns the user, again JSON-encoded, for a corresponding username.

We'll make use of type synonyms to group and give logical names to our API's sub-components.

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}

import Servant
  ( (:<|>)
  , (:>)
  , Capture
  , Get
  , JSON
  )

type UsersIndex =
  Get '[JSON] [User]

type UsersShow =
  Capture "username" String
    :> Get '[JSON] User

type UsersAPI =
  "users"
    :> (UsersIndex :<|> UsersShow)

Commit

Note for each code example I'll try to show only the bits of the finished module which are relevant to what's being currently discussed. Below some code samples you'll see a "commit" annotation which will link to a matching diff in the example repository.

This first code example already contains a few type-level features that likely look interesting. Let's take a closer look.

Type literals

Capture "username" String
        ~~~~~~~~~~

GHC supports both numeric and string-like type level literals. Of the greatest interest to us are the string-like type literals.

The first thing to note about these are that unlike most types we encounter in Haskell type-level string literals are not of kind Type but rather Symbol.

Note I'm using the extension NoStarIsType to replace * with Type when talking about kinds.

Another important thing to keep in mind is that each unique value is a different type. Or put another way, the type "foo" is distinct from the type "bar". When talking about these types as a whole it's helpful to step up to the kind layer and refer to them as Symbols. For example: "foo" and "bar" are different types, but they're both Symbols.

When working with Servant, Symbols are used to easily define things like static route segments, as well as named route and query parameters. Symbols can be brought down from the type level to strings at the value level. This means that Servant is able to extract data from API types for use at run time. More on this later.

Data type promotion

Get '[JSON] [User]
    ~~~~~~~

We don't have to be content with only strings and numbers moving up to the type level, by enabling the DataKinds language extension many other data types will become available for use up there too. Servant makes use of type level lists in some parts of its API.

The "tick" (') prefix is used to disambiguate type-level lists and the type of value-level lists. This is required in situations where GHC might need a hint to tell what is meant by, say:

[Int]

Is that the type for a list of Int, or is that a type level list with only one element, Int?

Similar to Symbol types like '[Int] and '[Bool] are not of kind Type. Those particular examples are of kind [Type]. It's worth noting, however, that the contents of a type level list need not always be of kind Type.

 > :kind '[Maybe]
'[Maybe] :: [Type -> Type]

If we ask GHCi for the kind of '[] we see something interesting.

 > :kind '[]
'[] :: [k]

This shows us that type level lists are kind polymorphic. That k is a kind variable, as with type variables these are introduced by an implicit forall.

'[] :: forall k. '[k]

Just as the type for value-level lists is parameterized over some arbitrary other type.

 > :type []
[] :: [a]

 > :type [1, 2, 3]
[1, 2, 3] :: Num a => [a]

 > :type [(+ 1), (+ 2), (+ 3)]
[(+ 1), (+ 2), (+ 3)] :: Num a => [a -> a]

The kind for type-level lists is parameterized over some arbitrary other kind.

 > :kind '[]
'[] :: [k]

 > :kind '[(), Bool, Ordering]
'[(), Bool, Ordering] :: [Type]

 > :kind '[Either (), Either Bool, Either Ordering]
'[Either (), Either Bool, Either Ordering] :: [Type -> Type]

Servant uses type level lists for a couple of purposes. In this post we'll see how they can help us specify the set of content types a given API can accept and return.

Type operators

type UsersAPI =
  "users"
    :> (UsersIndex :<|> UsersShow)
    ~~             ~~~~

By enabling the TypeOperators extension we can write infix type constructors in much same the way that we can value-level infix functions.

As with value-level infix operators, type operators have a precedence relative to other operators and can associate to the left or right. The syntax for defining these properties for a given type operator is the same as for value-level operators.

(f . g) a = f (g a)

infixr 9 .

type (f ∘ g) a = f (g a)

infixr 9 ∘

Putting it to use

As UsersAPI is the type for an API which serves users we'll need to define what a User is.

{-# LANGUAGE DeriveGeneric #-}

import Data.Time (Day)
import GHC.Generics (Generic)

data User = User
  { name :: String
  , age :: Int
  , email :: String
  , username :: String
  , registrationDate :: Day
  } deriving (Generic)

Commit

We're deriving a Generic instance for use later, it just helps to get it out-of-the-way now.

At the end of the day a runnable Haskell program is a value of type IO (). So how do we get from UsersAPI to one of those? The servant-server package provides us a few functions for turning types like UsersAPI into WAI Applications. The simplest of these is serve, so that's what we'll go with.

serve
  :: HasServer api '[]
  => Proxy api
  -> ServerT api Handler
  -> Application

The warp package provides a run function which gets us from an Application to a IO ().

run :: Port -> Application -> IO ()

In order for serve to return an Application we'll need to give it two things. The first is a Proxy of our UsersAPI type. If you've seen types like Map a or Set a before you might think that a Proxy a is some sort of container. Proxy is sort of a container, but rather a than for carrying around values it's for carrying around types.

data Proxy (t :: k) = Proxy

See that the only data constructor, also called Proxy, is nullary. At the value level it's empty, but at the type level it contains some t of kind k. Values of Proxy are made easier to construct with the help of the TypeApplications language extension. Using TypeApplications we can make explicit the types which are inferred and applied to our expressions.

 > :t Proxy
Proxy :: Proxy t
 > :t Proxy @Int
Proxy @Int :: Proxy Int
 > :t Proxy @UsersAPI
Proxy @UsersAPI :: Proxy UsersAPI

The Proxy @UsersAPI argument is needed by serve for slightly obscure reasons, but for our purposes it's enough to say that it's a value which carries the UsersAPI type around.

Producing a value for the second argument will be the topic of this post. We can use a typed hole for now, giving us enough for a skeleton main.

{-# LANGUAGE TypeApplications #-}

import Data.Proxy (Proxy(Proxy))
import Network.Wai (Application)
import Network.Wai.Handler.Warp (run)
import Servant (serve)

usersApp :: Application
usersApp = serve (Proxy @UsersAPI) _usersServer

main :: IO ()
main = run 8080 usersApp

Commit

Trying to compile what we have so far will result in two errors. The first is complaining about a missing instance.

• No instance for (ToJSON User) arising from a use of ‘serve’
• In the expression: serve (Proxy @Users) _usersServer
  In an equation for ‘usersApp’:
      usersApp = serve (Proxy @Users) _usersServer

The second tells us the type of the _usersServer value we've yet to define.

• Found hole:
    _usersServer
      :: Handler [User] :<|> ([Char] -> Handler User)
  Or perhaps ‘_usersServer’ is mis-spelled, or not in scope
• In the second argument of ‘serve’, namely ‘_usersServer’
  In the expression: serve (Proxy @Users) _usersServer
  In an equation for ‘usersApp’:
      usersApp = serve (Proxy @Users) _usersServer

We'll come back to why we're being asked about ToJSON instances but for now we'll just give Servant what it wants.

import Data.Aeson (ToJSON)

instance ToJSON User

Commit

A Server's type

We will try to understand where that ToJSON requirement came from but first we're going to focus on the typed hole. I think asking GHCi what the type of serve is when partially applied with a Proxy UsersAPI is instructive here.

 > :t serve (Proxy @UsersAPI)
serve (Proxy @UsersAPI)
  :: (Handler [User] :<|> ([Char] -> Handler User))
     -> Application

This is interesting. In the type of serve where previously there was a ServerT api Handler there is now a Handler [User] :<|> ([Char] -> Handler User). Where did it come from?

Recall that serve has the following type.

serve
  :: forall api
   . HasServer api '[]
  => Proxy api
  -> ServerT api Handler
  -> Application

There's a type variable api, which is constrained to be types with a HasServer instance. There's then two arguments, both referring to that constrained api type. We've been able to produce a value for the first using Proxy @UserAPI, a value for the second will take a little more doing.

I mean, what is ServerT? Why does it disappear when UsersAPI is substituted for api? It can't be a type constructor, type constructors don't just disappear. What does GHCi have to say about it?

 > :info ServerT
class HasServer (api :: k)
                (context :: [Type]) where
  type family ServerT (api :: k) (m :: Type -> Type) :: Type
  ...
        -- Defined in ‘Servant.Server.Internal’

ServerT is a type family. Type families look quite like type constructors. Like type constructors they accept types as arguments, unlike type constructors they're able to return different types depending on those arguments. It ends up looking something like a function which pattern matches on types.

ServerT is part of the HasServer type class, therefore it will be defined for any type which has a HasServer instance. It accepts the poly kinded api type HasServer is parameterized over as its first argument and some type constructor m of kind Type -> Type as its second. It then returns some type of kind Type.

GHCi allows us to evaluate type families, to see what the resulting type is at different type arguments.

 > :kind! ServerT Users Handler
ServerT UsersAPI Handler :: Type
= Handler [User] :<|> ([Char] -> Handler User)

Here we can see how UsersAPI becomes Handler [User] :<|> ([Char] -> Handler User) when substituted for api in the type of serve.

But that's not super satisfying to me. I feel like we're skipping a few steps. I'd like to see if we can find out what those steps are.

One of the great advantages of referentially transparent languages like Haskell is that if we want to see how an expression is evaluated we can do the evaluating ourselves, manually. We can substitute values for function parameters and continue evaluating the resulting expressions until we're only left with values. We'll attempt to apply this strategy to see what happens when UsersAPI is applied to ServerT.

Stepping through

When UsersAPI is substituted for api in the type of serve the ServerT type family is evaluated with it as its first argument and the Handler type as its second.

ServerT UsersAPI Handler

As ServerT is part of the HasServer type class there is a ServerT implementation for each HasServer instance. For us to figure out which ServerT to use we'll need to know which HasServer instance to grab it from. This means that not only are we going to be evaluating calls to ServerT ourselves, but we're also going to have to resolve type class instances too. How do we find the right HasServer instance for UsersAPI?

We can start by asking GHCi to tell us everything is knows about UsersAPI.

 > :info UsersAPI
type UsersAPI = "users" :> (UsersIndex :<|> UsersShow)
        -- Defined at src/Main.hs:75:1

GHCi knows that UsersAPI is a type synonym and helpfully shows us its definition.

We still don't have a HasServer instance, so we'll ask GHCi what it knows about the type that UsersAPI is a synonym for. Sadly, we can't pass the whole type to :info, we can only ask about things like type families or type constructors when unapplied. So we'll need to start with the outermost type constructor, (:>), and go from there.

The instance we're interested in will only show up if we import a couple of modules first.

 > import GHC.TypeLits
 > import Servant (HasServer)

We can now expect :info to tell us about the HasServer instances relevant to Symbols and (:>).

 > :info (:>)
type role (:>) phantom phantom
data (:>) (path :: k) a
        -- Defined in ‘Servant.API.Sub’
infixr 4 :>
instance (KnownSymbol path, HasServer api context) =>
         HasServer (path :> api) context
  -- Defined in ‘Servant.Server.Internal’
instance forall k l (arr :: k -> l) api (context :: [Type]).
         (TypeError ...) =>
         HasServer (arr :> api) context
  -- Defined in ‘Servant.Server.Internal’

Note that we're being shown two HasServer instances for (:>), and it actually has many more than these, but for now we're only interested in one of them.

instance
  ( KnownSymbol path
  , HasServer api context
  ) =>
    HasServer (path :> api) context

How do I know that this is the relevant instance?

KnownSymbol is a special type class definable only for types of kind Symbol. It allows us to "read" type-level Symbols into value-level Strings.

Seeing it in that instance head tells me that path is a Symbol, it couldn't have a KnownSymbol instance if it were anything else. UsersAPI has the Symbol "users" to the left of (:>) so it's a good match.

GHCi is able to tell us about type family instances for types, however in our case the output doesn't emphasize that ServerT instances belong to HasServer instances. To find the relevant ServerT it helps to look at HasServer's documentation and the linked source for the instance in question.

Now we know which ServerT will be applied.

type instance
  ServerT (path :> api) m =
    ServerT api m

We can take that, substitute "users" for path, UsersIndex :<|> UsersShow for api and Handler for m.

ServerT (UsersIndex :<|> UsersShow) Handler

Notice that ServerT is being called again in this substituted body. It recurses, having now peeled off the "users" :> part of the type. We have a new ServerT to find, this time for (:<|>).

We can again ask GHCi using :info to tell us what it knows about (:<|>). Fortunately there's only one HasServer instance and therefore only one ServerT.

type instance
  ServerT (a :<|> b) m =
    ServerT a m :<|> ServerT b m

Substituting UsersIndex for a, UsersShow for b and Handler for m gets us the following.

ServerT UsersIndex Handler :<|> ServerT UsersShow Handler

We're faced with two recursive calls to ServerT in this substituted body. For the next step we'll need to choose which branch to evaluate first. Let's begin on the left.

ServerT UsersIndex Handler

Which is the relevant ServerT for UsersIndex? Let's find out.

First UsersIndex is a synonym.

type UsersIndex =
  Get '[JSON] [User]

Asking GHCi what Get is will reveal that it too is a synonym.

 > :info Get
type Get =
  Servant.API.Verbs.Verb 'Network.HTTP.Types.Method.GET 200
  :: [Type] -> Type -> Type
        -- Defined in ‘Servant.API.Verbs’

Fortunately Verb is the end of the line.

 > :info Servant.API.Verbs.Verb
type role Servant.API.Verbs.Verb phantom phantom phantom phantom
data Servant.API.Verbs.Verb (method :: k1)
                            (statusCode :: Nat)
                            (contentTypes :: [Type])
                            a
        -- Defined in ‘Servant.API.Verbs’
instance [safe] forall k1 (method :: k1) (statusCode :: Nat) (contentTypes :: [Type]) a.
                Generic (Servant.API.Verbs.Verb method statusCode contentTypes a)
  -- Defined in ‘Servant.API.Verbs’
instance [overlappable] forall k1 (ctypes :: [Type]) a (method :: k1) (status :: Nat) (context :: [Type]).
                        (Servant.API.ContentTypes.AllCTRender ctypes a,
                         Servant.API.Verbs.ReflectMethod method, KnownNat status) =>
                        HasServer (Servant.API.Verbs.Verb method status ctypes a) context
  -- Defined in ‘Servant.Server.Internal’

There's our HasServer instance, and so we're able to find the corresponding ServerT.

type instance
  ServerT (Verb method status ctypes a) m =
    m a

Verbs are very general, we can make this look a bit simpler by inlining all the arguments that have been applied in the UsersShow type.

UsersShow
  =
Get '[JSON] [User]
  =
Verb GET 200 '[JSON] [User]

We now see that we can substitute [User] for a. Because we're tracing the evaluation of ServerT as it's used in serve we'll always be substituting Handler for m. Making those two substitutions in the body of the type family instance above will result in the following.

Handler [User]

That's the left branch of (:<|>) done.

Handler [User] :<|> ServerT UsersIndex Handler

On to the right.

ServerT UsersShow Handler

What was UsersShow a synonym for?

type UsersShow =
  Capture "username" String
    :> Get '[JSON] User

It's outermost type constructor is (:>), which we've seen before. Despite this we haven't yet seen the ServerT that we'll need to evaluate this next step.

Remember that the instance we last saw required that the first argument to (:>) be of kind Symbol. The first argument to (:>) in UsersShow, however, isn't. It's a bad match, we'll have to find another instance.

Let's try asking about Capture.

 > :info Capture
type Capture = Servant.API.Capture.Capture' '[] :: Symbol -> Type -> Type
        -- Defined in ‘Servant.API.Capture’

We're told it's a synonym for Capture'.

 > import Servant.API.Capture (Capture')
 > :info Capture'
type role Capture' phantom phantom phantom
data Capture' (mods :: [Type]) (sym :: Symbol) a
        -- Defined in ‘Servant.API.Capture’
instance (KnownSymbol capture,
          Web.Internal.HttpApiData.FromHttpApiData a,
          HasServer api context) =>
         HasServer (Capture' mods capture a :> api) context
  -- Defined in ‘Servant.Server.Internal’

Capture' has only one HasServer instance, and it's defined only for Capture's which appear on the left-hand side of (:>). This looks like a good match.

The ServerT for this instance is, I think, the most interesting we've seen.

type instance
  ServerT (Capture' mods capture a :> api) m =
    a -> ServerT api m

Substituting '[] for mods, "username" for capture, String for a, Get '[JSON] User for api and Handler for m gives us a function.

String -> ServerT (Get '[JSON] User) Handler

This is magical. A Capture is transformed into a function which accepts the path parameter it represents.

We're nearly finished evaluating, we have one more call to ServerT.

ServerT (Get '[JSON] User) Handler

Fortunately we already know the ServerT to use here.

type instance
  ServerT (Verb method status ctypes a) m =
    m a

So let's apply it.

Handler User

And we're finished evaluating ServerT UsersShow Hander.

String -> Handler User

Which means we're finished evaluating ServerT UsersIndex Handler :<|> ServerT UsersShow Handler.

Which means we're finished evaluating ServerT UsersAPI Handler.

Handler [User] :<|> (String -> Handler User)

Here's all of those steps together.

ServerT UsersAPI Handler
ServerT ("users" :> (UsersIndex :<|> UsersShow)) Handler
ServerT (UsersIndex :<|> UsersShow) Handler
ServerT UsersIndex Handler :<|> ServerT UsersShow Handler
ServerT (Get '[JSON] [User]) Handler :<|> ServerT UsersShow Handler
ServerT (Verb GET 200 '[JSON] [User]) Handler :<|> ServerT UsersShow Handler
Handler [User] :<|> ServerT UsersShow Handler
Handler [User] :<|> ServerT (Capture "username" String :> Get '[JSON] User) Handler
Handler [User] :<|> (String -> ServerT (Get '[JSON] User)) Handler
Handler [User] :<|> (String -> ServerT (Verb GET 200 '[JSON] User)) Handler
Handler [User] :<|> (String -> Handler User)

So this is how Servant goes about transforming the UsersAPI type into the type for a server. We first declared a reasonable looking shape for our API as a type and now Servant is letting us know how we can implement a server for it.

Before we do that, however, we had another type error we were going to look into.

Content types

Why did we need to define a ToJSON instance for User? Where did that constraint come from?

We mention JSON twice in the type of UsersAPI, in both instances it's as an argument to the Get type constructor. Recall from above that Get is an alias for Verb, recall also that the only constraint on serve is that the provided api type has a HasServer instance. Is there anything interesting about the HasServer instance for Verb?

instance
  forall
    k1
    (ctypes :: [Type])
    a
    (method :: k1)
    (status :: Nat)
    (context :: [Type]).
  ( AllCTRender ctypes a
  , ReflectMethod method
  , KnownNat status
  ) =>
    HasServer (Verb method status ctypes a) context

The HasServer instance for Verb constrains the a type parameter to be an instance of a type class called AllCTRender. If we go looking for instances of this type class we're faced with something that might seem a little strange.

instance
  (TypeError ...) =>
    AllCTRender '[] ()

instance
  ( Accept ct
  , AllMime cts
  , AllMimeRender (ct : cts) a
  ) =>
    AllCTRender (ct : cts) a

It has two instances. One instance which will throw a custom type error when applied to an empty list and (), and one that doesn't refer to any concrete types, strange stuff.

Notice that type level lists can be pattern-matched and de-structured quite like value level lists, here in the constraints of this type class instance. This means that were able to iterate over type level lists in much the same way that we do for those at the value level. The iteration splits off into three more type classes: Accept, AllMime and AllMimeRender.

AllMimeRender has two instances, and with these this really starts to look like value level list iteration.

instance
  ( MimeRender ctyp a
  ) =>
    AllMimeRender '[ctyp] a

instance
  ( MimeRender ctyp a
  , AllMimeRender (ctyp' : ctyps) a
  ) =>
    AllMimeRender (ctyp : ctyp' : ctyps) a

We have a base case where there's only one element in the list. The other instance asserts that the head of the list has an instance of MimeRender and the (non-empty) tail has one for AllMimeRender, that is: it recurses.

Our API only speaks JSON, represented by its content-types list being '[JSON], so the first instance of AllMimeRender is used. This means that there needs to be an instance of MimeRender for JSON. By looking for that instance we see what we've been looking for.

instance ToJSON a => MimeRender JSON a

Here's how it all goes.

In order for there to be an instance of HasServer for Verb GET 200 '[JSON] User there has to be an instance of AllCTRender for '[JSON] and User
In order for there to be an instance of AllCTRender for '[JSON] and User there has to be an instance of AllMimeRender for '[JSON] and User
In order for there to be an instance of AllMimeRender for '[JSON] and User there has to be an instance of MimeRender for JSON and User
In order for there to be an instance of MimeRender for JSON and User there has to be an instance of Aeson's ToJSON for User

So that's why when we tried to apply UsersAPI to serve we needed to define a ToJSON instance for User.

Implementing a server for our type

Now we know how the typed hole _usersServer ends up with the type it does.

Handler [User] :<|> (String -> Handler User)

Let's go about creating a value of this type. We might start from the outside, in much the same order as we traced the evaluation of ServerT. This means first figuring out how to construct a value of type a :<|> b. Using GHCi we're able to view the definition of this type.

data (:<|>) a b = a :<|> b

It's pretty much a pair.

data (,) a b = (a, b)

It has only one data constructor, which happens to share the type's name. There's only one thing we can do, provide that constructor two values. Now, we don't actually have those values yet, so let's use typed holes for now.

usersServer :: Server UsersAPI
usersServer = _usersIndex :<|> _usersShow

Commit

Applying this to serve will let us know if we're on the right track.

  usersApp :: Application
- usersApp = serve (Proxy @UsersAPI) _usersServer
+ usersApp = serve (Proxy @UsersAPI) usersServer

Trying to compile should give us something like the following.

src/Main.hs:86:15: error:
    • Found hole: _usersIndex :: Handler [User]
      Or perhaps ‘_usersIndex’ is mis-spelled, or not in scope
    • In the first argument of ‘(:<|>)’, namely ‘_usersIndex’
      In the expression: _usersIndex :<|> _usersShow
      In an equation for ‘usersServer’:
          usersServer = _usersIndex :<|> _usersShow
    • Relevant bindings include
        usersServer :: Server UsersAPI (bound at src/Main.hs:86:1)
   |
86 | usersServer = _usersIndex :<|> _usersShow
   |               ^^^^^^^^^^^

src/Main.hs:86:32: error:
    • Found hole: _usersShow :: [Char] -> Handler User
      Or perhaps ‘_usersShow’ is mis-spelled, or not in scope
    • In the second argument of ‘(:<|>)’, namely ‘_usersShow’
      In the expression: _usersIndex :<|> _usersShow
      In an equation for ‘usersServer’:
          usersServer = _usersIndex :<|> _usersShow
    • Relevant bindings include
        usersServer :: Server UsersAPI (bound at src/Main.hs:86:1)
   |
86 | usersServer = _usersIndex :<|> _usersShow
   |                                ^^^^^^^^^^

Amongst that we're being told that we have two values we need to conjure up.

usersIndex :: Handler [User]
usersIndex = _

usersShow :: String -> Handler User
usersShow _uname = _

Commit

We'll start with usersIndex, which is a value of type Handler [User].

For the sake of this example our collection of users will be some static, sample data. I might do another post on my experience of using Beam in a Servant application, but for now let's keep it simple.

users :: [User]
users =
  [ User
    { name             = "Isaac Newton"
    , age              = 372
    , email            = "isaac@newton.co.uk"
    , username         = "isaac"
    , registrationDate = fromGregorian 1683 3 1
    }
  , User
    { name             = "Albert Einstein"
    , age              = 136
    , email            = "ae@mc2.org"
    , username         = "albert"
    , registrationDate = fromGregorian 1905 12 1
    }
  ]

Commit

Now that we have a value of [User] we need a function which can wrap it in a Handler. Looking at the documentation for Handler we see that it has an Applicative instance which will provide us just what we need.

 > :type pure @Handler
pure @Handler :: a -> Handler a

So there we have it.

usersIndex :: Handler [User]
usersIndex = pure users

Commit

For UsersShow we'll have a little more work to do. We're supplied the user name of the user we'd like returned, we should use it to look for that user in users. The function we'll need for finding elements of lists is find.

find :: Foldable t => (a -> Bool) -> t a -> Maybe a

Or more specifically.

 > :t find @[] @User
find @[] @User :: (User -> Bool) -> [User] -> Maybe User

We'll need a function User -> Bool and in our case the Bool should indicate whether a provided String matches a given Users username.

matchesUsername :: String -> User -> Bool
matchesUsername uname = (uname ==) . username

We're nearly there.

 > :t \uname -> find @[] (matchesUsername uname)
\uname -> find @[] (matchesUsername uname)
  :: String -> [User] -> Maybe User

 > :t \uname -> find @[] (matchesUsername uname) users
\uname -> find @[] (matchesUsername uname) users
  :: String -> Maybe User

The final step is to turn a Maybe User into a Handler User. For the Just case we have a User and are able to use pure @Handler to wrap it in a Handler, but for the Nothing case, what should we do?

usersShow :: String -> Handler User
usersShow uname =
  case find (matchesUsername uname) users of
    Nothing   -> _
    Just user -> pure user

Commit

Ordinarily when you ask a web server for a resource it can't find you get a 404 response back. How can we produce a value of Handler User that results in a 404?

By looking at the docs for Handler again we can see that it has a MonadError instance, this suggests that we can use throwError when we need to return something but can't return a User.

By looking at Handler's instance for MonadError we see that it's defined for a ServantErr type. So in our case throwError has the following type.

 > :t throwError @ServantErr @Handler
throwError @ServantErr @Handler :: ServantErr -> Handler a

Now it helps to look at the documentation for ServantErr. There we see quite a few values of the type, including a quite relevant-looking err404.

usersShow :: String -> Handler User
usersShow uname =
  case find (matchesUsername uname) users of
    Nothing   -> throwError err404
    Just user -> pure user

Commit

We've now defined everything we need and should have a runnable server.

Start it up.

$ runhaskell src/Main.hs

And try it out.

$ curl -sD /dev/stderr http://localhost:8080/users | jq .
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Sat, 21 Sep 2019 05:26:33 GMT
Server: Warp/3.2.28
Content-Type: application/json;charset=utf-8

[
  {
    "email": "isaac@newton.co.uk",
    "registrationDate": "1683-03-01",
    "age": 372,
    "username": "isaac",
    "name": "Isaac Newton"
  },
  {
    "email": "ae@mc2.org",
    "registrationDate": "1905-12-01",
    "age": 136,
    "username": "albert",
    "name": "Albert Einstein"
  }
]

$ curl -sD /dev/stderr http://localhost:8080/users/albert | jq .
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Sat, 21 Sep 2019 05:25:09 GMT
Server: Warp/3.2.28
Content-Type: application/json;charset=utf-8

{
  "email": "ae@mc2.org",
  "registrationDate": "1905-12-01",
  "age": 136,
  "username": "albert",
  "name": "Albert Einstein"
}

$ curl -sD /dev/stderr http://localhost:8080/users/unknown | jq .
HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Date: Sat, 21 Sep 2019 05:26:00 GMT
Server: Warp/3.2.28

What's it all for?

My hope when planning this post was that I'd become a little more familiar with the type level programming features of GHC Haskell. I wasn't sure which features or to what extent. Having finished I'd say that I've started to understand this topic. At the very least I've spent a bit of time becoming more familiar with a library that makes great use of GHC's type level features.

The DSL provided by Servant allows us to construct types which specify an API contract. With it we were able to specify static route segments and named route parameters using Symbols. We could associate those routes with HTTP verbs which could accept and return many content-types using type level lists. Combining these components was made easy with infix type constructors.

The way Servant has us think "specification first" is very appealing to me, and the more time I spend with Haskell the more this method of designing and implementing software just feels right. The type level feels much more declarative: you don't talk as much about what you want to happen, instead you talk more about what things you would like to exist. Then it's up to you and the compiler to figure out how that might be possible.

There's another, more practical, benefit to this in Servant's case however. We've seen that specifications can be turned into the types of servers which implement them, however we didn't explore how we can also use them to automatically produce clients, documentation, and even property tests.

I imagine investigating those packages would be good for even more type level learnings.

Let's learn about lenses

hi@bradparker.id.au (Brad Parker) — Tue, 21 Aug 2018 00:00:00 UTC

Before we begin: in order to understand a lot of the following you'll first need some familiarity with Haskell syntax and be somewhat comfortable with what a Functor is.

OK, now firstly: why learn about lenses? Every time I've seen lenses in action it's looked like wizardry. I mean, just look at these examples of using lens-aeson:

someJson =
  [json| {
    packages: [
      { version: 1 },
      { version: 5 }
    ]
  }|]

versions =
  key "packages"
    . _Array
    . traverse
    . key "version"
    . _Number

print $ someJson ^.. versions
-- [1.0,5.0]

BS.putStrLn $ someJson & versions %~ (+ 2)
-- {
--   "packages": [
--     { "version": 3 },
--     { "version": 7 }
--   ]
-- }

What? Reaching into some JSON, transforming it and just putting it back together, there's no way it can be so easy? Right?

Let's try to understand what's going on here. We should probably start with something a little simpler than the JSON example above, how about tuples?

OK, imagine we have these two magic functions. _1 for doing stuff to the first element of a tuple and _2 for doing stuff to the second.

We can compose them _1 . _2, which lets us do stuff to the second element of a tuple which is itself the first element of a tuple.

((a, b), c)
--   ^
--   This thing, here.

What kind of stuff?

We can "view" (^.) the value:

> ((2, True), 'b') ^. _1 . _2
True

We can "modify" (%~) the value:

> ((2, True), 'b') & _1 . _2 %~ not
((2,False),'b')

We can traverse over a list of these tuples and view the value of each as a list (^.. with traverse):

> [((2, True), 'b'), ((3, False), 'b')] ^.. traverse . _1 . _2
[True,False]

While we traverse (this time over any Traversable thing), instead of viewing we can modify:

> [((2, True), 'b'), ((3, False), 'b')] & traverse . _1 . _2 %~ not
[((2,False),'b'),((3,True),'b')]

That looks pretty useful to me. What do you reckon?

To understand how this all works we should probably move step by step. Let's try this:

First we're going to define slightly simplified versions of _1 and _2, as one and two.
Then we'll move on to define similarly simplified versions of ^. (as view) and %~ (as modify).
Finally we'll adapt _1 and _2 to be a little closer to their definitions in the lens library.

We might seek to explore ^.. and how it interacts with traverse in another post. It's super cool, but we'll likely need to get there via some other concepts.

Lenses `_1` (one) and `_2` (two)

What are these two magic functions? How do they work?

We'll start by describing "functional references" al la Twan van Laarhoven, a precursor to what we now call lenses. For the sake of simplicity we're still going to refer to these as "lenses".

In the above article a lens is defined as something with this type:

Functor f => (a -> f a) -> s -> f s

Let's make an alias for use later.

type Lens' s a =
  forall f. Functor f =>
    (a -> f a) -> s -> f s

The goal of this section is going to be implementing functions that satisfy this type. In the next section we'll seek to understand why this type is the way it is.

Imagine that s stands for "structure", this will become clearer a little later.

Something bothered me when I first saw this type, it looks like it's missing two functions: s -> a, a -> s. If we had an s -> a we could turn the s we get as a second argument into the a we need to supply to the a -> f a we receive as the first argument. If we had an a -> s we could use fmap to transform the f a to an f s ready to return it.

We could write a function which accepts an s -> a and an a -> s in order to build a lens, something like this:

makeLens
  :: (s -> a)
  -> (a -> s)
  -> Lens' s a
makeLens sToA aToS aToFa s =
  fmap aToS (aToFa (sToA s))

It ends up those functions are precisely what we need to provide to make a lens. The first function is the "getter", the second the "setter".

What might a getter for the first element of a tuple look like?

getOne :: (a, c) -> a
getOne (a, _) = a

Looks about right. How about the setter?

setOne :: (a, c) -> a -> (a, c)
setOne (_, c) a = (a, c)

That should work.

one :: Lens' (a, c) a
one aToFa ac =
  makeLens getOne (setOne ac) aToFa ac

We don't really need this makeLens function, so let's get rid of it.

one :: Lens' (a, c) a
one aToFa (a, c) =
  fmap (\a' -> (a', c)) (aToFa a)

We now do the getting by pattern matching and the setting with an anonymous function: \a' -> (a', c). We don't need separate functions necessarily for getting and setting, which might explain why they're not mentioned in the type.

Let's compare the type of one to the lens type above.

       Functor f => (a -> f a) -> s      -> f s
one :: Functor f => (a -> f a) -> (a, c) -> f (a, c)

So s in the lens type is (a, c) in the type for one. The getter function s -> a is generic enough to say that "a is some part of s", without being specific about what s is at all.

OK, now two is going to be pretty much the same, except that we'll focus on the second element of the tuple.

two :: Lens' (c, a) a
two aToFa (c, a) =
  fmap (\a' -> (c, a')) (aToFa a)

If we compare it with the lens type:

       Functor f => (a -> f a) -> s      -> f s
two :: Functor f => (a -> f a) -> (c, a) -> f (c, a)

All together now:

       Functor f => (a -> f a) -> s      -> f s
one :: Functor f => (a -> f a) -> (a, c) -> f (a, c)
two :: Functor f => (a -> f a) -> (c, a) -> f (c, a)

I reckon we've got a reasonable idea now how the functions one and two are lenses. That is to say: we can see how the types line up.

The next step is to understand how we can use each of these functions as both a getter and a setter.

Lens interpreters `(^.)` (view) and `(%~)` (modify)

We've made the types line up, but to what end? What is this crazy type up to anyway? The genius here lies largely in the Functor f constraint.

To start with we'll pick a very boring Functor to stand in for f: Identity.

newtype Identity a = Identity { runIdentity :: a }

instance Functor Identity where
  fmap :: (a -> b) -> Identity a -> Identity b
  fmap f (Identity a) = Identity (f a)

See how boring this thing is? It's just a container for a single value, all fmap can do is unpack it to get at the a inside, apply f to a and pack it back up again.

Note that our lenses expect a a -> f a, which in this case will be a -> Identity a. We have a function already which can fulfil that type, Identity's only data constructor:

Identity :: a -> Identity a

So we can pass the Identity data constructor as the first argument to one and see what we get.

> :t one Identity
one Identity :: (a, c) -> Identity (a, c)

Hmm, and if we pass it some value?

> one Identity (1, 'a')
Identity (1,'a')

Makes sense, and I guess all we can do now is unpack it using runIdentity :: Identity a -> a:

> runIdentity (one Identity (1, 'a'))
(1,'a')

Wait, this looks familiar.

> familiar = runIdentity . one Identity
> familiar (1, 'a')
(1,'a')
> id (1, 'a')
(1,'a')

Right, right. Cool. So absolutely nothing happens but in a fairly involved way.

We can make something happen though.

> one (Identity . (+ 2)) (1, 'a')
Identity (3,'a')

We can compose Identity :: a -> Identity a with some a -> a. We can write a pretty general function for this:

> modifyOne f tuple = runIdentity (one (Identity . f) tuple)
> modifyOne (+ 2) (1, 'a')
(3,'a')

... an even more general function:

> modify lens modification value = runIdentity (lens (Identity . modification) value)
> modify one (+ 2) (1, 'a')
(3,'a')

Wow, what's the type of modify then?

> :t modify
modify :: ((a -> Identity a) -> s -> Identity s) -> (a -> a) -> s -> s

Modify takes a lens:

> :t modify one
modify one :: (a -> a) -> (a, c) -> (a, c)

An a -> a, which will be composed with Identity :: a -> Identity a, still giving us the a -> Identity a we need:

> :t modify one (+ 2)
modify one (+ 2) :: Num a => (a, c) -> (a, c)

And some value the lens can act on:

> :t modify one (+ 2) (1, 'a')
modify one (+ 2) (1, 'a') :: Num a => (a, Char)

If we make things less generic for a second it might be a bit clearer:

one :: (a -> Identity a) -> (a, c) -> Identity (a, c)
one aToFa (a, c) = fmap (\a' -> (a', c)) (aToFa a)
--  ^^^^^          ^^^^^^^^^^^^^^^^^^^^^
--  |              All this does is put the
--  |              modified a back in the tuple:
--  |              Identity a -> Identity (a, c)
--  |
--  |
--  This is the Identity data constructor composed
--  with some function a -> a, the modification

This is all pretty interesting but still doesn't answer why we have the Functor f constraint, Identity here just makes getting to the value kinda obscure.

Let's put another weird Functor in place of f: Const.

newtype Const c a = Const { getConst :: c }

instance Functor (Const c) where
  fmap :: (a -> b) -> Const c a -> Const c b
  fmap _ (Const c) = Const c

This was pretty mind-bending for me at first. What possible use could this thing have? It looks like it's a Functor where fmap doesn't do anything, the function argument is totally ignored.

It's worth having another look at the signature for fmap in that last code block. The function argument isn't applied, but the type does change. We've still gone from f a to f b after we've fmaped.

Let's look at these fmaps side by side.

fmap :: (a -> b) -> f        a -> f        b
fmap :: (a -> b) -> Identity a -> Identity b
fmap :: (a -> b) -> Const c  a -> Const c  b

What happens when we use Const :: a -> Const a a as our a -> f a in one?

> one Const (1, 'a')
Const 1

Huh? Weird. Wonder what the type is.

> :t one Const (1, 'a')
one Const (1, 'a') :: Num a => Const a (a, Char)

Right, so we do have a Const a (a, c), which fits the f (a, c) returned by one. The really interesting thing is that the "setter" part of one didn't actually get applied. Const's fmap ignores that function argument.

one :: Functor f => (a -> f a) -> (a, c) -> f (a, c)
one aToFa (a, c) = fmap (\a' -> (a', c)) (aToFa a)
--                      ^^^^^^^^^^^^^^^^
--                      This is never run

Once more, if we make things less generic for a second it might be a bit clearer:

one :: (a -> Const a a) -> (a, c) -> Const a (a, c)
one aToFa (a, c) = fmap (\a' -> (a', c)) (aToFa a)
--  ^^^^^          ^^^^^^^^^^^^^^^^^^^^^
--  |              All this does is
--  |              change the type:
--  |              Const a a -> Const a (a, c)
--  |
--  |
--  This is the Const data constructor.

The Functor constraint is satisfied, so we're able to use Identity and Const interchangeably. When we use Identity the setter function is run and so the updated value is popped back into our tuple, when we use Const the setter is skipped.

Once again there's little more we can do than unwrap the Functor:

> getConst (one Const (1, 'a'))
1

Also once again, we can write a very generic version of this:

> view lens value = getConst (lens Const value)
> :t view
view :: ((a -> Const a a) -> s -> Const a s) -> s -> a
> view one (1, 'a')
1

Polymorphism?

So far modify is somewhat limited, we can only provide it functions that accept and return values of the same type (a -> a). What might it take to adapt one, two and modify to work with functions that accept one type and return another?

The answer is: nothing. All we need to do is change some type signatures and variable names. The actual functions are already good to go.

Our current lens type becomes the same as the lens library type by adding two more type variables: b and t. When we allow the value yanked out of s to have it's type changed from a to b, putting it back must be allowed to change the type of s, and so we return f t.

Functor f => (a -> f a) -> s -> f s
Functor f => (a -> f b) -> s -> f t

We can define another handy alias.

type Lens s t a b =
  forall f. Functor f =>
    (a -> f b) -> s -> f t

one becomes _1 with only some type and value variable renaming:

one :: Lens' (a, c) a
one aToFa (a, c) =
  fmap (\a' -> (a', c)) (aToFa a)

_1 :: Lens (a, c) (b, c) a b
_1 aToFb (a, c) =
  fmap (\b -> (b, c)) (aToFb a)

As with two to _2:

two :: Lens' (c, a) a
two aToFa (c, a) =
  fmap (\a' -> (c, a')) (aToFa a)

_2 :: Lens (c, a) (c, b) a b
_2 aToFb (c, a) =
  fmap (\b -> (c, b)) (aToFb a)

modify becomes %~ thusly

modify :: Lens' s a -> (a -> a) -> s -> s
modify lens modification value =
  runIdentity $
    lens (Identity . modification) value

(%~) :: Lens s t a b -> (a -> b) -> s -> t
(%~) lens modification value =
  runIdentity $
    lens (Identity . modification) value

infixr 7 %~

view gets much the same treatment, we also flip the args to make it match ^.:

view :: Lens' s a -> s -> a
view lens value = getConst (lens Const value)

(^.) :: s -> Lens s t a b -> a
(^.) value lens = getConst (lens Const value)

infixr 7 ^.

And hey presto we have polymorphic lenses, quite like those in Control.Lens!

Some more fun examples to see what we've done in action:

> ('a', 4) & _1 %~ (: "bc")
("abc",4)
> ('a', 4) & _2 %~ show
('a',"4")
> ('a', 4) ^. _1
'a'
> ('a', 4) ^. _2
4
-- Let's get crazy
> ('a', 4) & _1 %~ (: "bc") & _2 %~ (show . subtract 4)
("abc","0")

Conclusion

This type didn't make a heap of sense to me when I first saw it:

type Lens s t a b =
  forall f. Functor f =>
    (a -> f b) -> s -> f t

After having gone through all of the implementations above, I reckon I get it now. This is not to say I understand all the weird and wonderful things in lens, but I feel like I'm on the way. With any luck this may have helped you too.

You already know what Monads are

hi@bradparker.id.au (Brad Parker) — Fri, 6 Jan 2017 00:00:00 UTC

Assuming you're a javascript dev who's had to deal with anything aysncronous.

Firstly, we're going to talk about promises, so:

What do we talk about when we talk about Promises?

It's very likely you've done something like this:

doSomethingAsync(function (resultOfAsyncThing) {
  console.log(resultOfAsyncThing)
})

There's an interesting concept highlighted in that code above.

What if we tried to do the following:

let resultOutsideCallback

doSomethingAsync(function (resultOfAsyncThing) {
  resultOutsideCallback = resultOfAsyncThing
})

How will we get to use resultOutsideCallback?

let resultOutsideCallback

doSomethingAsync(function (resultOfAsyncThing) {
  resultOutsideCallback = resultOfAsyncThing
})

while(true) {
  if (resultOutsideCallback) {
    console.log(resultOutsideCallback)
    break
  }
}

That's... not awesome. Everything we try to execute after that while loop is blocked until the async function's callback is called. But if we want to work with that value "outside" of the callback, that's kind of how we'd have to do it.

So, it appears that the resultOfAsyncThing value is best left "inside" the callback. This is something that just kind of happens when working with callbacks. A callback creates an (almost) inescapable context.

someAsyncAction(function (a) {
  someOtherAsyncAction(function (b) {
    console.log(a + b)
  })
})

Promises are a way of turning this concept of "the eventual value of an async action" into an object we can interact with. The object is a container for this future value that grants you access to it through the then method. The then method is a way to get into the context in which the eventual value exists.

someAsyncAction().then(function (resultOfAsyncAction) {
  console.log(resultOfAsyncAction)
})

Now, the example above doesn't really offer anything over using callbacks. But there is an aspect of promises that makes them very handy:

someAsyncAction().then(function (value) {
  return value.toUppercase()
}).then(function (uppercaseValue) {
  return uppercaseValue.split('').reverse().join()
}).then(function (uppercaseAndReversedValue) {
  console.log(uppercaseAndReversed)
})

Each time we call then we get a new promise back. The new promise now "contains" the value returned by the function we passed in. So the above could be re-written as:

const uppercased = someAsyncAction().then(function (value) {
  return value.toUppercase()
})

const uppercasedAndReversed = uppercased.then(function (uppercaseValue) {
  return uppercaseValue.split('').reverse().join()
})

uppercasedAndReversed.then(function (uppercasedAndReversedValue) {
  console.log(uppercasedAndReversedValue)
})

It looks a lot like performing a chain of transformations on an array to me:

[1, 2, 3].map(function (value) {
  return value * 2
}).map(function (value) {
  return value + 2
})

Ok, so what does this have to do with Monads?

Here we should highlight that Array#map is a lot like Promise#then. Where then is something you call on a container for a future value, map is something you call on a container for many values. They both kind of work the same way, you pass a function which will return a new value which goes back into the container.

const a = Promise.resolve(1)
// Promise { 1 }
const b = a.then(function (a) { return a + 2 })
// Promise { 3 }

const as = [ 1, 1, 1 ]
// [ 1, 1, 1 ]
const bs = as.map(function (a) { return a + 2 })
// [ 3, 3, 3 ]

This concept of a container for a value which can have that value transformed by passing a function to the container has a name: Functor. That's a bit of a mouthful so let's pull it apart. Array is a functor because it has a map method which allows you to operate on and transform its contents. Promise is a functor because it has a then method which allows you to operate on and transform its eventual value.

The functions passed to Array#map and Promise#then both return the new contents of their respective containers. So you pass in a fuction which returns a "normal" value and when it's called that value will be wrapped up in a container for us.

Promise.resolve(1).then(function (a) {
  return a + 2
//       ^^^^^
//       normal, un-wrapped value
})

This is not always what you want. Say, for example, that you need to perform two async actions in series, one will pass data to the next.

readAFile().then(function (content) {
  return writeAFile(content)
}).then(function (report) {
  console.log(report)
})

See that? Interesting, right? You can return a new Promise from the function passed to then, when you do you get a new promise back, but it has the same contents as the one you returned.

const a = Promise.resolve(1)
// Promise { 1 }

const b = Promise.resolve(2)
// Promise { 2 }

const c = a.then(function () {
  return b
})
// Promise { 2 }

c === b
// false

So you either return a "normal" value and it'll get wrapped up in a Promise for you, or you can return a Promise and its content will be the content of the new promise returned.

This second version of the function passed to then, where it accepts a normal value but returns a Promise has an array equivelant: flatMap.

flatMap([1, 2, 3], function (elem) {
  return [elem, '-']
})
// [ 1, '-', 2, '-', 3, '-' ]

The function passed to flatMap takes a normal value but returns an Array. We don't get an array of arrays back because something happens to that returned value so that it results in a flattened array.

This extension to the idea of a functor, whereby the function passed to it returns a container of the same type has a name: Monad.

With the value-extracting and container-flattening parts of these operations abstracted away composing functions which return containers is made simpler. We can chain together async operations easily because the plumbing for “where did this value come from?” and “where should this new Promise go?” happens inside the then method. That’s what we get through good, composable abstractions.

This was intended as something of a practical introduction to the concept of a Monad, there is more to it than I’ve outlined here. If you’d like to dive deeper have a look at http://learnyouahaskell.com/a-fistful-of-monads. You may want to start from the begining of the book to gain some familiarity with Haskell.

[1] For more info on how promises perform their flattening: http://www.mattgreer.org/articles/promises-in-wicked-detail/

[2] A simple implementation of flatMap might look like: https://gist.github.com/bradparker/34c7bd8d627809333adcc3e02cd7f755

Petrichor Ensemble — Gig Poster

hi@bradparker.id.au (Brad Parker) — Fri, 8 Feb 2013 00:00:00 UTC

Brisbane Bicycle Film Festival 2012 — Poster

hi@bradparker.id.au (Brad Parker) — Wed, 3 Oct 2012 00:00:00 UTC

Design futures — Typography

hi@bradparker.id.au (Brad Parker) — Wed, 1 Jun 2011 00:00:00 UTC

Brad Parker

Books, Badgers and Big Numbers

Books

Badgers

Big Numbers

Lexicographic order

Indexing

Lehmer codes

Crossing paths with Descartes

Two-up

Hazard

DNA

Quartets

In space

In time

The danger of using models as metaphors

Using Nix to deploy a Haskell web app to Heroku

Haskell and Nix

An amazing web application

A deployable image

Shipping it

Home sweet home

Bootstrapping scripts

Staying fresh

Home manager

Utilities

Vim

Tmux

Bash

Git

Getting close to the conceptual metal

Scott encoding

Pair

Either

From hot air to solid ground

Deploying a fully automated Nix-based static website

First, a Droplet

Next, a static site

Finally, a web server

Hey, presto

Servant's type-level domain specific language

The example type

Type literals

Data type promotion

Type operators

Putting it to use

A Server's type

Stepping through

Content types

Implementing a server for our type

What's it all for?

Let's learn about lenses

Lenses _1 (one) and _2 (two)

Lens interpreters (^.) (view) and (%~) (modify)

Polymorphism?

Conclusion

You already know what Monads are

What do we talk about when we talk about Promises?

Ok, so what does this have to do with Monads?

Petrichor Ensemble — Gig Poster

Brisbane Bicycle Film Festival 2012 — Poster

Design futures — Typography

Lenses `_1` (one) and `_2` (two)

Lens interpreters `(^.)` (view) and `(%~)` (modify)