joeldueckdotcom

I’m thinking of making my own Racket #lang that would serve the same purpose as Pollen — a programming environment for publishing web pages and books.

Why would I do this? I have enjoyed using Pollen very much. But after seven years of improving my Racket skills, my “Pollen projects” have been making less and less actual use of Pollen’s facilities and features. I’ve started to wish for a DSL that has only the pieces I need: something smaller, faster, and with fewer moving parts.

This would be something designed only for my own use, perhaps only for a single project. I’d make it publicly available, but not as a package on the package server.

Differences from Pollen

New Features

Integrating Markdown

Markdown itself is too limited, but it would be nice to be able to use it as a starting point and add tag functions where desired. Layering two independent syntaxes on top of each other is very tricky, however.

The commonmark package may be ideal for this: it implements the CommonMark standard, is fast, and parses to an AST rather than to HTML.

Here’s how it would work (I have a proof of concept coded already):

  1. The source file is parsed with a Pollen-like reader first. After all statements are evaluated, any non-string values are serialized and URI-encoded into the attribute of an HTML placeholder tag, which is inserted as a string.
  2. All the strings are concatenated and parsed by commonmark, producing a document struct. (Any HTML-like content in the document will be contained in an html struct.)
  3. The document is then converted to an AST in X-expression form. During this process, any HTML placeholder tags are replaced by the URI-decoded, deserialized value of the tag‘s attribute.
  4. This final AST becomes the doc provided by the source document.

Handling Markdown nested inside tag functions

There would be a wrinkle when a tag function might itself contain text that should be parsed as markdown. For example, there might be **bold text** in part of the caption of a figure.

In order for markdown to be allowed within tag functions, each tag function would need to run its string elements through their own separate markdown parsing pass. This would have the effect of sealing off markdown inside tag functions from markdown outside tag functions. For example, markdown-based footnote references or link references inside a tag function would not be able to reference anything outside it. On the other hand, a tag function would be able to do fancy things like contain its own footnotes.

Name ideas

Example syntax

#lang interpunct example/markup

•(meta title "Prepare to be amazed" date "2020-05-07")

This is a paragraph. **Bold text**, etc. — you know the markdown drill.

The `example/markup` above is an optional module path that will be `require`d into
the current document for additional tag functions.

> Famous quotation.
>
> •attrib{Surly Buster, [_Fight to Win_](https://surly.guy/fight-to-win/) (2008)}

The above is an example of a markdown blockquote containing a tag function which
in turn contains markdown.

•note[#:date "2020-05-07" #:by "A Reader" #:bylink "foo@msn.com"]{
This is a note added to the document[^1].

•poem[#:title "Institutions"]{
‘Ləh’
}

[^1]: It can contain its own footnotes and link references.
}

Prior Art

How to Create a Pollen Markup Alternative in 61 Lines by Sage Gerard. Sage’s take was more about a text markup format that you could send through eval rather than creating a proper #lang where the sources would behave like first-class modules. Still a very useful experiment.