I recently released Consfigurator 1.0.0 and I’m now returning to my Common Lisp reading. Building Consfigurator involved the ad hoc development of a cross between a Haskell-style functional DSL and a Lisp-style macro DSL. I am hoping that it will be easier to retain lessons about building these DSLs more systematically, and making better use of macros, by finishing my studying of macrology books and papers only after having completed the ad hoc DSL. Here’s my current list:

  • Finishing off On Lisp and Let Over Lambda.

  • Richard C. Waters. 1993. “Macroexpand-All: an example of a simple lisp code walker.” In Newsletter ACM SIGPLAN Lisp Pointers 6 (1).

  • Naive vs. proper code-walking.

  • Michael Raskin. 2017. “Writing a best-effort portable code walker in Common Lisp.” In Proceedings of 10th European Lisp Symposium (ELS2017).

  • Cullpepper et. al. 2019. “From Macros to DSLs: The Evolution of Racket”. Summet of Advances in Programming Languages.

One thing that I would like to understand better is the place of code walking in macro programming. The Raskin paper explains that it is not possible to write a fully correct code walker in ANSI CL. Consfigurator currently uses Raskin’s best-effort portable code walker. Common Lisp: The Language 2 includes a few additional functions which didn’t make it into the ANSI standard that would make it possible to write a fully correct code walker, and most implementations of CL provide them under one name or another. So one possibility is to write a code walker in terms of ANSI CL + those few additional functions, and then use a portability layer to get access to those functions on different implementations (e.g. trivial-cltl2).

However, both On Lisp and Let Over Lambda, the two most substantive texts on CL macrology, both explicitly put code walking out-of-scope. I am led to wonder: does the Zen of Common Lisp-style macrology involve doing without code walking? One key idea with macros is to productively blur the distinction between designing languages and writing code in those languages. If your macros require code walking, have you perhaps ended up too far to the side of designing whole languages? Should you perhaps rework things so as not to require the code walking? Then it would matter less that those parts of CLtL2 didn’t make it into ANSI. Graham notes in ch. 17 of On Lisp that read macros are technically more powerful than defmacro because they can do everything that defmacro can and more. But it would be a similar sort of mistake to conclude that Lisp is about read macros rather than defmacro.

There might be some connection between arguments for and against avoiding code walking in macro programming and the maintainance of homoiconicity. One extant CL code walker, hu.dwim.walker, works by converting back and forth between conses and CLOS objects (Raskin’s best-effort code walker has a more minimal interface), and hygienic macro systems in Scheme similarly trade away homoiconicity for additional metadata (one Lisp programmer I know says this is an important sense in which Scheme could be considered not a Lisp). Perhaps arguments against involving much code walking in macro programming are equivalent to arguments against Racket’s idea of language-oriented programming. When Racket’s designers say that Racket’s macro system is “more powerful” than CL’s, they would be right in the sense that the system can do all that defmacro can do and more, but wrong if indeed the activity of macro programming is more powerful when kept further away from language design. Anyway, these are some hypotheses I am hoping to develop some more concrete ideas about in my reading.