December 4th, 2017

Creating your own tiny static publishing platform

I’ve been using static publishing platforms for a while now. The output is enduring and easily archived, and reliable and robust. As an author, there’s also a lot of truth to the unreasonable effectiveness of GitHub browsability however much I disagree with the philosophy therein of committing build products in with the sources. I’ve used Jekyll; Hexo, which is what I use to write this blog; I’ve used Movable Type long ago.

However, all these systems are more complex than I’d like, and prone to bit-rot, far far faster than the content they generate. Runtimes change. Dependencies rot as maintainers move on and no longer can account for those runtime changes. Development moves on to new major versions or being built with a newer fad in software design. Hexo has treated me better than most, but it is large, and the configuration rather arbitrary in places. Plugins have to be written specifically for Hexo, so there’s a balkanized ecosystem that doesn’t flourish as well as other parts do. All these static publishing tools tend to have things in common. Builds have to happen as quickly as they can, and usually this is a bit too slowly. The author will want to preview their work in context, so serving up the rendered pages is important. Live rebuilds by file monitoring reduce friction in the workflow for some people, though I personally don’t care much for it, preferring to run a build when I’m ready.

It turns out that building derived things from a list of inputs with dependencies is a thing that computers have been told to do for a long time. Nearly all compiled software is built this way. We have tools like make(1) and a host of other, more complex and less general tools for various programming languages. I’ve always wondered why we didn’t use those to build sites as well. People have, it turns out, but make(1) in particular is a bit messier for the task than one would hope. There are other tools, and I settled on building with one called tup

This weekend I built a small static publishing platform, and you can too. I wanted to build a site using Tufte CSS, and the minimalism of the presentation is a great fit for a super tiny static publishing platform.

A site like this needs to output:

Each post as an HTML file
An index page listing posts
Its CSS and any assets needed to render

This really isn’t a huge list.

First, let’s reach for a tool that can take a list of files and build all the derived things. make(1) is annoying here, because you have to tell it what to build, and it backtracks and figures out how to make it. We don’t actually have that information easily encoded, but we will have a list of sources, and can make a list of what to do with them. If you’re writing, you probably have a reason for it, right? Or an asset, it’s going to get used, why else would it be there? Starting at the source makes a lot more sense, and as it turns out, it makes incremental builds a lot faster. Enter our first player: tup.

$ brew cask install osxfuse
$ brew install tup

I’m not sure why tup now depends on FUSE, but that’s a task for another day.

Let’s start a directory for our project.

$ mkdir my-static-site
$ npm init
$ mkdir posts

Make a sample markdown file in the posts directory.

Next we create a Tupfile to describe how we’re going to build this site. Then we can just type tup to build the site, or tup monitor on Linux for that live building mode. First, let’s handle each post as HTML. We can use an off the shelf markdown renderer at first.

$ npm install marked

Here’s a Tupfile

: foreach posts/*.md |> marked %f -o %o |> public/%B.html

This means that for each post in the posts directory, we’ll make an equivalent HTML file.

Let’s take a look at some of these rendered files. We’ll need to serve this directory by HTTP if we want to see it as we will on the web.

$ tup
$ npx serve public/

We can now open the site preview at the URL it spits out (usually http://localhost:5000)

Just a directory full of HTML, and ‘full’ is just our one test post, but we should be able to navigate to one. We have a static site, if a lousy one! That HTML is pretty spartan, so let’s add some assets.

Copy the et-book directory of fonts from the Tufte CSS package into the root of the project, and the tufte.css file.

Let’s add a few rules to publish those as part of the site, too. Added to the Tupfile:

: foreach et-book/et-book-bold-line-figures/* |> cp %f %o |> public/%f
: foreach et-book/et-book-display-italic-old-style-figures/* |> cp %f %o |> public/%f
: foreach et-book/et-book-roman-line-figures/* |> cp %f %o |> public/%f
: foreach et-book/et-book-roman-old-style-figures/* |> cp %f %o |> public/%f
: foreach et-book/et-book-semi-bold-old-style-figures/* |> cp %f %o |> public/%f
: foreach *.css |> cp -r %f %o |> public/%b

Run tup again.

The assets got copied in. Now we have to actually put them in the HTML. That’s going to mean templates.

ejs is simple enough and behaves tidily and doesn’t have a lot of dependencies, so let’s use that for output templates.

$ npm install ejs

We’re going to have to create a script to render our markdown and template the file.

Let’s call this render.js:

const marked = require('marked')
const ejs = require('ejs')
const { promisify } = require('util')
const { readFile, writeFile } = require('fs')
const readFileAsync = promisify(readFile)
const writeFileAsync = promisify(writeFile)
const path = require('path')

main.apply(null, process.argv.slice(2)).catch(err => {
    console.warn(err)
    process.exit(1)
})

async function main(layoutFile, templateFile, postFile, outputFile) {
    const layoutP = readFileAsync(layoutFile, 'utf-8')
    const templateP = readFileAsync(templateFile, 'utf-8')
    const contentP = readFileAsync(postFile, 'utf-8')

    const content = marked(await contentP)
    const layout = ejs.compile(await layoutP)
    const template = ejs.compile(await templateP)

    const dest = path.basename(postFile).replace(/\.md$/, '.html')

    const body = template({ content, require })
    const rendered = layout({ content: body })

    await writeFileAsync(outputFile, rendered)
}

It expects two templates: a layout (the skeleton and boilerplate of the page) and a template (the post template). Let’s create those now.

layout.ejs:

<!doctype html>
<html>
	<head>
		<meta charset='utf-8'>
		<link rel='stylesheet' href='tufte.css'>
	</head>

	<body>
		<%- content %>
	</body>
</html>

and post.ejs:

<section>
	<%- content %>
</section>

And in the Tupfile, let’s replace the marked render with our own. Additionally, let’s tell tup that the HTML depends on the templates, so if those change, we update all the HTML.

: foreach posts/*.md | layout.ejs post.ejs |> node render layout.ejs post.ejs %f %o |> public/%B.html

Let’s run tup again and see the output. Much prettier, right?

Now about that index! The index needs to know the post’s title, and really, posts don’t even have titles yet. Let’s add some to our test post as YAML front matter. Add this at the top of the markdown file.

----
title: My Post
date: 2017-12-04 01:51:43
----

Every post gets a title and the date.

Let’s change our renderer to put the title on the page so we don’t have to reduplicate it.

Install front-matter

$ npm install front-matter

And update render.js

const marked = require('marked')
const ejs = require('ejs')
const { promisify } = require('util')
const { readFile, writeFile } = require('fs')
const readFileAsync = promisify(readFile)
const writeFileAsync = promisify(writeFile)
const frontMatter = require('front-matter')
const path = require('path')

main.apply(null, process.argv.slice(2)).catch(err => {
    console.warn(err)
    process.exit(1)
})

async function main(layoutFile, templateFile, postFile, outputFile) {
    const layoutP = readFileAsync(layoutFile, 'utf-8')
    const templateP = readFileAsync(templateFile, 'utf-8')
    const contentP = readFileAsync(postFile, 'utf-8')

    const post = frontMatter(await contentP)
    const content = marked(post.body)
    const layout = ejs.compile(await layoutP)
    const template = ejs.compile(await templateP)

    const dest = path.basename(postFile).replace(/\.md$/, '.html')

    const body = template(Object.assign({ }, post.attributes, { content }))
    const rendered = layout(Object.assign({ }, post.attributes, { content: body }))

    await writeFileAsync(outputFile, rendered)
}

And to post.ejs, the title.

<h1><%= title %></h1>

And in layout.ejs, let’s add a title tag too.

<title><%= title %> — My Blog</title>

Run tup again and let’s check our work.

Now a little harder part. Let’s make the index page.

We’ll need a script to generate it, index.js:

const ejs = require('ejs')
const fm = require('front-matter')
const path = require('path')
const { promisify } = require('util')
const { readFile, writeFile } = require('fs')
const readFileAsync = promisify(readFile)
const writeFileAsync = promisify(writeFile)

main.apply(null, process.argv.slice(2)).catch(err => {
    console.warn(err)
    process.exit(1)
})

async function main(outputFile, layoutFile, templateFile, ...metadataFiles) {
    const tP = readFileAsync(templateFile, 'utf-8')
    const lP = readFileAsync(layoutFile, 'utf-8')

    const metadata = await Promise.all(
        metadataFiles.map(
            f => readFileAsync(f, 'utf-8')
                .then(fm)
                .then(e => Object.assign(e.attributes, { dest: path.basename(f).replace(/\.md$/, '.html')} ))))

    metadata.sort((a, b) => {
        a = new Date(a.date)
        b = new Date(b.date)
        return a>b ? -1 : a<b ? 1 : 0
    })

    const layout = ejs.compile(await lP)
    const template = ejs.compile(await tP)

    const rendered = layout({
        title: 'Posts',
        content: template({ metadata })
    })

    await writeFileAsync(outputFile, rendered)
}

And an index.ejs:

<h1>My Blog</h1>
<section>
    <% metadata.forEach(entry => { %>
        <p>
            <a href='<%= entry.dest %>'><%= entry.title %></a>
        </p>
    <% }) %>
</section>

And in our Tupfile:

: templates/layout.ejs templates/index.ejs posts/*.md |> node index %o %f |> public/index.html

Run tup once more and we should have a bare-bones site.

Let’s add one more thing before we go, some dates to the posts.

To the template calls in both render.js and index.js, let’s add the require function, so that templates can require their own stuff.Where there’s template({ metadata }), let’s change that to template({ metadata, require })

Then, let’s install fast-strftime.

$ npm install strftime

An expanded index.ejs:

<% const strftime = require('fast-strftime') %>
<h1>My Blog</h1>
<section>
    <% metadata.forEach(entry => { %>
        <p>
            <a href='<%= entry.dest %>'><%= entry.title %></a> <%= date ? strftime('%Y-%m-%d', date) : '' %>
        </p>
    <% }) %>
</section>

And the page template, post.ejs:

<% const strftime = require('fast-strftime') %>

<h1><%= title %></h1>

<% if (date) { %>
    <p>posted <%= strftime('%Y-%m-%d', date) %></p>
<% } %>
<section>
	<%- content %>
</section>

Run tup once more, and you’ve got a static site, being generated by some simple code.