joeldueckdotcom

I’m thinking of making my own Racket #lang that would serve the same purpose as Pollen — a programming environment for publishing web pages and books.

Why would I do this? I have enjoyed using Pollen very much. But after seven years of improving my Racket skills, my “Pollen projects” have been making less and less actual use of Pollen’s facilities and features. I’ve started to wish for a DSL that has only the pieces I need: something smaller, faster, and with fewer moving parts.

This would be something designed only for my own use, perhaps only for a single project. I’d make it publicly available, but not as a package on the package server.

Differences from Pollen

• All markup/tag functions would be defined in and provided by the language. There would be no equivalent of a pollen.rkt file and no setup submodule in which to provide additional runtime options.
  • Although perhaps an optional module path could be included on the #lang line to be auto-required into the document...
• The location of the “project root” would just be the current directory.
• There would be no preprocessor or “pagetree” flavors of the language.
• The tag functions would produce a generic AST (in the form of an X-expression) rather than going directly to HTML in X-expression form. The AST would be independent of the output format, so tag functions would not need to check for the current output format and the source document always compiles to the same result.
• The language would provide “renderers” for transforming the AST to HTML and LaTeX.
• The language would not provide any command-line tools. Sources would be compiled with raco make and rendered with, e.g., racket -tm render.rkt.
• The language would not provide a project server for local preview/testing. Any simple local web server will work to serve static files, and fswatch could be used to run make whenever a .rkt file changes. Alternatively, I could write a lightweight web server that runs make on every requested file before serving it.

New Features

Integrating Markdown

Markdown itself is too limited, but it would be nice to be able to use it as a starting point and add tag functions where desired. Layering two independent syntaxes on top of each other is very tricky, however.

The commonmark package may be ideal for this: it implements the CommonMark standard, is fast, and parses to an AST rather than to HTML.

Here’s how it would work (I have a proof of concept coded already):

1. The source file is parsed with a Pollen-like reader first. After all statements are evaluated, any non-string values are serialized and URI-encoded into the attribute of an HTML placeholder tag, which is inserted as a string.
2. All the strings are concatenated and parsed by commonmark, producing a document struct. (Any HTML-like content in the document will be contained in an html struct.)
3. The document is then converted to an AST in X-expression form. During this process, any HTML placeholder tags are replaced by the URI-decoded, deserialized value of the tag‘s attribute.
4. This final AST becomes the doc provided by the source document.

Handling Markdown nested inside tag functions

There would be a wrinkle when a tag function might itself contain text that should be parsed as markdown. For example, there might be **bold text** in part of the caption of a figure.

In order for markdown to be allowed within tag functions, each tag function would need to run its string elements through their own separate markdown parsing pass. This would have the effect of sealing off markdown inside tag functions from markdown outside tag functions. For example, markdown-based footnote references or link references inside a tag function would not be able to reference anything outside it. On the other hand, a tag function would be able to do fancy things like contain its own footnotes.

Name ideas

• #lang interpunct
• #lang yarn
• #lang dispatch
• #lang microtype

Example syntax

#lang interpunct example/markup

•(meta title "Prepare to be amazed" date "2020-05-07")

This is a paragraph. **Bold text**, etc. — you know the markdown drill.

The `example/markup` above is an optional module path that will be `require`d into
the current document for additional tag functions.

> Famous quotation.
>
> •attrib{Surly Buster, [_Fight to Win_](https://surly.guy/fight-to-win/) (2008)}

The above is an example of a markdown blockquote containing a tag function which
in turn contains markdown.

•note[#:date "2020-05-07" #:by "A Reader" #:bylink "foo@msn.com"]{
This is a note added to the document[^1].

•poem[#:title "Institutions"]{
‘Ləh’
}

[^1]: It can contain its own footnotes and link references.
}

Prior Art

How to Create a Pollen Markup Alternative in 61 Lines by Sage Gerard. Sage’s take was more about a text markup format that you could send through eval rather than creating a proper #lang where the sources would behave like first-class modules. Still a very useful experiment.