I’m thinking of making my own Racket #lang
that would serve the same purpose as Pollen — a programming environment for publishing web pages and books.
Why would I do this? I have enjoyed using Pollen very much. But after seven years of improving my Racket skills, my “Pollen projects” have been making less and less actual use of Pollen’s facilities and features. I’ve started to wish for a DSL that has only the pieces I need: something smaller, faster, and with fewer moving parts.
This would be something designed only for my own use, perhaps only for a single project. I’d make it publicly available, but not as a package on the package server.
Differences from Pollen
- All markup/tag functions would be defined in and provided by the language. There would be no equivalent of a
pollen.rkt
file and nosetup
submodule in which to provide additional runtime options.- Although perhaps an optional module path could be included on the
#lang
line to be auto-required into the document...
- Although perhaps an optional module path could be included on the
- The location of the “project root” would just be the current directory.
- There would be no preprocessor or “pagetree” flavors of the language.
- The tag functions would produce a generic AST (in the form of an X-expression) rather than going directly to HTML in X-expression form. The AST would be independent of the output format, so tag functions would not need to check for the current output format and the source document always compiles to the same result.
- The language would provide “renderers” for transforming the AST to HTML and LaTeX.
- The language would not provide any command-line tools. Sources would be compiled with
raco make
and rendered with, e.g.,racket -tm render.rkt
. - The language would not provide a project server for local preview/testing. Any simple local web server will work to serve static files, and
fswatch
could be used to runmake
whenever a.rkt
file changes. Alternatively, I could write a lightweight web server that runsmake
on every requested file before serving it.
New Features
Integrating Markdown
Markdown itself is too limited, but it would be nice to be able to use it as a starting point and add tag functions where desired. Layering two independent syntaxes on top of each other is very tricky, however.
The commonmark
package may be ideal for this: it implements the CommonMark standard, is fast, and parses to an AST rather than to HTML.
Here’s how it would work (I have a proof of concept coded already):
- The source file is parsed with a Pollen-like reader first. After all statements are evaluated, any non-string values are serialized and URI-encoded into the attribute of an HTML placeholder tag, which is inserted as a string.
- All the strings are concatenated and parsed by
commonmark
, producing adocument
struct. (Any HTML-like content in the document will be contained in anhtml
struct.) - The document is then converted to an AST in X-expression form. During this process, any HTML placeholder tags are replaced by the URI-decoded, deserialized value of the tag‘s attribute.
- This final AST becomes the
doc
provided by the source document.
Handling Markdown nested inside tag functions
There would be a wrinkle when a tag function might itself contain text that should be parsed as markdown. For example, there might be **bold text**
in part of the caption of a figure.
In order for markdown to be allowed within tag functions, each tag function would need to run its string elements through their own separate markdown parsing pass. This would have the effect of sealing off markdown inside tag functions from markdown outside tag functions. For example, markdown-based footnote references or link references inside a tag function would not be able to reference anything outside it. On the other hand, a tag function would be able to do fancy things like contain its own footnotes.
Name ideas
#lang interpunct
#lang yarn
#lang dispatch
#lang microtype
Example syntax
#lang interpunct example/markup
•(meta title "Prepare to be amazed" date "2020-05-07")
This is a paragraph. **Bold text**, etc. — you know the markdown drill.
The `example/markup` above is an optional module path that will be `require`d into
the current document for additional tag functions.
> Famous quotation.
>
> •attrib{Surly Buster, [_Fight to Win_](https://surly.guy/fight-to-win/) (2008)}
The above is an example of a markdown blockquote containing a tag function which
in turn contains markdown.
•note[#:date "2020-05-07" #:by "A Reader" #:bylink "foo@msn.com"]{
This is a note added to the document[^1].
•poem[#:title "Institutions"]{
‘Ləh’
}
[^1]: It can contain its own footnotes and link references.
}
Prior Art
How to Create a Pollen Markup Alternative in 61 Lines by Sage Gerard. Sage’s take was more about a text markup format that you could send through eval
rather than creating a proper #lang
where the sources would behave like first-class modules. Still a very useful experiment.