I’ve been blogging since 2005, but posts from 2005–10 have not been imported to the current iteration of my website.

I recently released Consfigurator 1.0.0 and I’m now returning to my Common Lisp reading. Building Consfigurator involved the ad hoc development of a cross between a Haskell-style functional DSL and a Lisp-style macro DSL. I am hoping that it will be easier to retain lessons about building these DSLs more systematically, and making better use of macros, by finishing my studying of macrology books and papers only after having completed the ad hoc DSL. Here’s my current list:

  • Finishing off On Lisp and Let Over Lambda.

  • Richard C. Waters. 1993. “Macroexpand-All: an example of a simple lisp code walker.” In Newsletter ACM SIGPLAN Lisp Pointers 6 (1).

  • Naive vs. proper code-walking.

  • Michael Raskin. 2017. “Writing a best-effort portable code walker in Common Lisp.” In Proceedings of 10th European Lisp Symposium (ELS2017).

  • Cullpepper et. al. 2019. “From Macros to DSLs: The Evolution of Racket”. Summet of Advances in Programming Languages.

One thing that I would like to understand better is the place of code walking in macro programming. The Raskin paper explains that it is not possible to write a fully correct code walker in ANSI CL. Consfigurator currently uses Raskin’s best-effort portable code walker. Common Lisp: The Language 2 includes a few additional functions which didn’t make it into the ANSI standard that would make it possible to write a fully correct code walker, and most implementations of CL provide them under one name or another. So one possibility is to write a code walker in terms of ANSI CL + those few additional functions, and then use a portability layer to get access to those functions on different implementations (e.g. trivial-cltl2).

However, both On Lisp and Let Over Lambda, the two most substantive texts on CL macrology, both explicitly put code walking out-of-scope. I am led to wonder: does the Zen of Common Lisp-style macrology involve doing without code walking? One key idea with macros is to productively blur the distinction between designing languages and writing code in those languages. If your macros require code walking, have you perhaps ended up too far to the side of designing whole languages? Should you perhaps rework things so as not to require the code walking? Then it would matter less that those parts of CLtL2 didn’t make it into ANSI. Graham notes in ch. 17 of On Lisp that read macros are technically more powerful than defmacro because they can do everything that defmacro can and more. But it would be a similar sort of mistake to conclude that Lisp is about read macros rather than defmacro.

There might be some connection between arguments for and against avoiding code walking in macro programming and the maintainance of homoiconicity. One extant CL code walker, hu.dwim.walker, works by converting back and forth between conses and CLOS objects (Raskin’s best-effort code walker has a more minimal interface), and hygienic macro systems in Scheme similarly trade away homoiconicity for additional metadata (one Lisp programmer I know says this is an important sense in which Scheme could be considered not a Lisp). Perhaps arguments against involving much code walking in macro programming are equivalent to arguments against Racket’s idea of language-oriented programming. When Racket’s designers say that Racket’s macro system is “more powerful” than CL’s, they would be right in the sense that the system can do all that defmacro can do and more, but wrong if indeed the activity of macro programming is more powerful when kept further away from language design. Anyway, these are some hypotheses I am hoping to develop some more concrete ideas about in my reading.

Posted Sun 08 May 2022 20:41:17 UTC

Consfigurator has long has combinators OS:TYPECASE and OS:ETYPECASE to conditionalise on a host’s operating system. For example:

  (debian-stable (apt:installed-backport "notmuch"))
  (debian-unstable (apt:installed "notmuch")

You can’t distinguish between stable releases of Debian like this, however, because while that information is known, it’s not represented at the level of types. You can manually conditionalise on Debian suite using something like this:

(defpropspec notmuch-installed :posix ()
  (switch ((os:debian-suite (get-hostattrs-car :os)) :test #'string=)
    ("bullseye" '(apt:installed-backport "notmuch"))
    (t          '(apt:installed "notmuch"))))

but that means stepping outside of Consfigurator’s DSL, which has various disadvantages, such as a reduction in readability. So today I’ve added some new combinators, so that you can say

  ("bullseye" (apt:installed-backport "notmuch"))
  (t          (apt:installed "notmuch")))

For my own use I came up with this additional simple wrapper:

(defmacro for-bullseye (atomic)
     ("bullseye" ,atomic)
     ;; Check the property is actually unapplicable.
     ,@(and (get (car atomic) 'punapply) `((t (unapplied ,atomic))))))

So now I can say

(for-bullseye (apt:pinned '("elpa-org-roam") '(os:debian-unstable) 900))

which is a succinct expression of the following: “on bullseye, pin elpa-org-roam to sid with priority 900, drop the pin when we upgrade the machine to bookworm, and don’t do anything at all if the machine is still on buster”.

As a consequence of my doing Debian development but running Debian stable everywhere, I accumulate a number of tweaks like this one over the course of each Debian stable release. In the past I’ve gone through and deleted them all when it’s time to upgrade to the next release, but then I’ve had to add properties to undo changes made for the last stable release, and write comments saying why those are there and when they can be safely removed, which is tedious and verbose. This new combinator is cleaner.

Posted Tue 03 May 2022 23:13:37 UTC Tags:

I am pleased to announce Consfigurator 1.0.0.

Reaching version 1.0.0 signifies that we will try to avoid API breaks. You should be able to use Consfigurator to manage production systems.

You can find the source at https://git.spwhitton.name/consfigurator for browsing online or git cloning.

Releases are made by publishing signed git tags to that repository. The tag for this release is named ‘v1.0.0’, and is signed by me.

On Debian/etc. systems, apt-get install cl-consfigurator


Consfigurator is a system for declarative configuration management using Common Lisp. You can use it to configure hosts as root, deploy services as unprivileged users, build and deploy containers, install operating systems, produce disc images, and more. Some key advantages:

  • Apply configuration by transparently starting up another Lisp image on the machine to be configured, so that you can use the full power of Common Lisp to inspect and control the host.

  • Also define properties of hosts in a more restricted language, that of :POSIX properties, to configure machines, containers and user accounts where you can’t install Lisp. These properties can be applied using just an SSH or serial connection, but they can also be applied by remote Lisp images, enabling code reuse.

  • Flexibly chain and nest methods of connecting to hosts. For example, you could have Consfigurator SSH to a host, sudo to root, start up Lisp, use the setns(2) system call to enter a Linux container, and then deploy a service. Secrets, and other prerequisite data, are properly passed along.

  • Combine declarative semantics for defining hosts and services with a multiparadigmatic general-purpose programming language that won’t get in your way.

Declarative configuration management systems like Consfigurator and Propellor share a number of goals with projects like the GNU Guix System and NixOS. However, tools like Consfigurator and Propellor try to layer the power of declarative and reproducible configuration semantics on top of traditional, battle-tested UNIX system administration infrastructure like distro package managers, package archives and daemon configuration mechanisms, rather than seeking to replace any of those. Let’s get as much as we can out of all that existing distro policy-compliant work!

Posted Sat 30 Apr 2022 19:50:47 UTC

I like to look at the Emacs subreddit and something I’ve noticed recently is people asking “should I start by writing my own Emacs config, or should I use this or that prepackaged one?” There is also this new config generator published by Philip Kaludercic. I find implicit in these the idea that one’s init.el is a singular product. To start using Emacs, newcomers seem to think, you need to couple it with a completed init.el, and so there is the question of writing your own or using one someone else has written. I think that an appropriate analogy is certain shell scripts. If you want to burn backups to DVDs you might download someone’s DVD burning shell script which tries to make that easy, or you might write your own. In both cases, you are likely to want to tweak the script after you’ve started using it, but there is nevertheless a discrete point at which you go from having part of a script and not being able to burn DVDs, to having a completed script and now being able to burn DVDs. Similarly, the idea that you can’t start using Emacs until you couple it with an init.el is like thinking that there is a process of producing or downloading an init.el, and only after that can you begin using Emacs.

This thinking makes sense if you’re developing one of the large Emacs configuration frameworks like Spacemacs or Doom Emacs. The people behind those projects are seeking to build something quite different from Emacs, using Emacs as a base, and for many people using that new, quite different thing is preferable to using Emacs. Then indeed, until you’ve finished developing your configuration framework’s init.el to a degree that you’re ready to release version 0.1 of your framework, you haven’t got something that’s ready to use. Like the shell script, there’s a discrete point after which you have a product, and there’s lots of labour that must precede it. (I think it’s pretty cool that Emacs is flexible enough to be something on its own and also a base for these projects.)

However, this temporal structure does not make sense to me when it comes to using just Emacs. I find the idea that one’s init.el is a singular product strange. In particular, my init.el is not something which transforms Emacs into something else, like the init.el that’s part of Doom Emacs does. It’s just a long collection of incrementally developed, largely unrelated hacks and tweaks. You could insist that it transforms default Emacs into Sean’s Emacs, but I think that misleadingly implies that there’s an overarching design and cohesion to my init.el, which just isn’t there – and it would be weird if it was there, because then I would be doing something more like the developers behind Doom Emacs. So if you’re not going to use one of the large configuration frameworks, then there is no task of “writing your own init.el” that stands before you. You just start using Emacs, and as part of that you’re going to write functions and rebind keys, etc., and your init.el is the file in which those changes are collected. The choice is not between writing your own init.el or downloading a prepackaged one. It’s between using Emacs, or using another product that has been built out of Emacs. The latter necessarily involves a completed init.el, but that’s an implementation detail.

I am very happy Kaludercic’s configuration generator has been made available, but I would be inclined to rename it. A new user of Emacs is likely to be overwhelmed with unintuitive defaults that have stuck around for mostly historical reasons. There are a lot of them, so it is a lot to ask of new users that they just identify the defaults that don’t suit them and add lines to their init.el to change those. When too many things are unintuitive, it’s hard to know where to start. Kaludercic’s configuration generator is a way to walk newcomers through the most significant defaults in a way that something structured like a reference manual would struggle to do. The result is some Lisp code, but I would prefer not to refer to that result as an Emacs configuration. It’s a series of configuration snippets that you can add to your Emacs configuration to help deal with the newcomer’s problem of too many unintuitive defaults.

I’m not sure it’s important to actually rename Kaludercic’s tool to something which says it’s a generator of configuration snippets rather than a generator of configurations. But I would like to challenge the idea that to start using Emacs you first need to couple it with a completed init.el. If you’re going to use Emacs, rather than Spacemacs or Doom Emacs, you can just start using it. If you find yourself butting up against a lot of unintuitive defaults, then you can use a walkthrough tool like Kaludercic’s to figure out what you need to add to your init.el to deal with those. But that is better understood as just more of the tweaking and customisation that Emacs users are always getting up to, not some prerequisite labour.

Posted Tue 26 Apr 2022 06:01:25 UTC Tags:

Here are a few new features I’ve added to GNU ELPA and upstream GNU Emacs recently. Text is adapted from the in-tree documentation I wrote for the new features. Thanks to everyone who offered feedback on my patches.

New feature to easily bypass Eshell’s own pipelining

Prefixing |, < or > with an asterisk, i.e. *|, *< or *>, will cause the whole command to be passed to the operating system shell. This is particularly useful to bypass Eshell’s own pipelining support for pipelines which will move a lot of data.

This has long been an obstacle when it comes to using Eshell as one’s main shell. The new syntax is easy to use and covers a lot of different use cases.

New Eshell module to help supplying absolute file names to remote commands

After enabling the new eshell-elecslash module, typing a forward slash as the first character of a command line argument will automatically insert the Tramp prefix. The automatic insertion applies only when default-directory is remote and the command is a Lisp function. This frees you from having to keep track of whether commands are Lisp function or external when supplying absolute file name arguments.

This is another attempt to solve an Eshell papercut. Suppose you execute

 cd /ssh:root@@example.com:
 find /etc -name "*gnu*"

and in reviewing the output of the command, you identify a file /etc/gnugnu that should be moved somewhere else. So you type

mv /etc/gnugnu /tmp

But since mv refers to the local Lisp function eshell/mv, not a remote shell command (unlike find(1)), to say this is to request that the local file /etc/gnugnu be moved into the local /tmp directory. After you enable eshell-elecslash, to then when you type the above mv invocation you will get the following input, which is what you intended:

mv /ssh:root@example.com:/etc/gnugnu /ssh:root@example.com:/tmp

imenu is now bound to M-g i globally

This is a useful command but everyone has to come up with their own binding for it. No longer.

New macro-writing macros, cl-with-gensyms and cl-once-only

These two macros are quite interesting. In the history of Common Lisp-style macros, these are the only two macro-writing macros that have emerged as essential tools for intermediate and advanced macrology. Most any other macro-writing macros are either project- or programmer-specific. In his book on Lisp macros Doug Hoyte proposes an alternative to defmacro, defmacro!, which is just the same as defmacro except that it builds in facilities equivalent to cl-with-gensyms and cl-once-only. I’ve long wanted to have these macros available in core Emacs Lisp, too, and now they are.

New package on GNU ELPA: transient-cycles

Many commands can be conceptualised as selecting an item from an ordered list or ring. Sometimes after running such a command, you find that the item selected is not the one you would have preferred, but the preferred item is nearby in the list. If the command has been augmented with transient cycling, then it finishes by setting a transient map with keys to move backwards and forwards in the list of items, so you can select a nearby item instead of the one the command selected. From the point of view of commands subsequent to the deactivation of the transient map, it is as though the first command actually selected the nearby item, not the one it really selected.

Protesilaos Stavrou helped me test the package and has written up some usage notes.

This is an idea I came up with in 2020, and refined in my init.el since then. This year I made it into a package.

Posted Thu 21 Apr 2022 20:43:28 UTC Tags:

Debian contributors end up with a number of Debian installations on their workstations for various purposes; there are perhaps five or six on my laptop right now. In addition to this, I now have three copies of GNU Emacs, too.


This is the version of Emacs in Debian stable. I have to keep it available for testing various of the Debian packages I maintain.


This is a build of Emacs 29.0.50 out of upstream git, with native Wayland support, and some custom Debian packaging to integrate it with Debian’s emacsen-common infrastructure: Emacs addon packages are pre-bytecompiled against this version of Emacs, and re-bytecompilation is fully automated. This is nice if you want to use development snapshots of Emacs but get your addon packages from Debian, as I do. These are, roughly, the extent of my changes on top of upstream:

diff --git a/.gitignore b/.gitignore
index 78557a5e87..9743ccca0c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,6 +23,11 @@
 # in Git 1.8.2 (March 2013).

+# Debian packaging
 # Personal customization.

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index de4cebccca..1596b5a1bf 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -311,6 +311,7 @@ package-directory-list
       (and (stringp f)
            (equal (file-name-nondirectory f) "site-lisp")
            (push (expand-file-name "elpa" f) result)))
+    (push "/usr/share/emacs-snapshot/site-lisp/elpa" result)
     (nreverse result))
   "List of additional directories containing Emacs Lisp packages.
 Each directory name should be absolute.
diff --git a/lisp/startup.el b/lisp/startup.el
index b79467339b..c54ab89b33 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1374,7 +1374,12 @@ command-line
     ;; be loaded from site-run-file and wants to test if -q was given
     ;; should check init-file-user instead, since that is already set.
     ;; See cus-edit.el for an example.
-    (if site-run-file
+    (when site-run-file
+        (let ((file "/etc/emacs/site-start.d/00debian.el"))
+          ;; When the Emacs build invokes Emacs, such as in the
+          ;; Makefile rule for ${unidir}/emoji-labels.el, 00debian.el
+          ;; might not exist.  Should be fine to just skip the load.
+          (when (file-readable-p file) (load file)))
         ;; Sites should not disable the startup screen.
         ;; Only individuals should disable the startup screen.
         (let ((inhibit-startup-screen inhibit-startup-screen))

If you’re using Wayland on Debian, or you just want or need a newer Emacs, you might like to install these snapshot builds yourself from http://silentflame.com/debian/, suites “bullseye-backports” and “unstable”.


The snapshot builds described in the previous section are what I use on machines other than my laptop. On my laptop, I have my Wayland compositor configured to use the emacs and emacsclient binaries in ~/src/emacs, if they exist. This is a build of a branch with work-in-progress patches I’m planning to send upstream, plus bug fixes, reversions of commits that introduced regressions which affect my setup, etc..

Why have this in addition to the emacs-snapshot.deb builds – if there are temporary fixup commits, why not apply them and build a .deb just for local use? Well, that is indeed what I was doing until recently, but if you’re developing patches for upstream Emacs it’s pretty inconvenient. Hyperlinks to function definitions in Help buffers take you somewhere read-only in /opt, for example, and rebuilds are slow because, by default, Debian package builds begin by cleaning the tree.

In addition to the diff above (minus the .gitignore change), and the temporary fixup commits, I also apply the following change on this branch. Having the above diff applied implies that a recent emacs-snapshot.deb is installed, to provide the byte-compiled addons. This informal dependency should work well enough so long as I update the snapshot .debs described in the previous section often enough.

diff --git a/lib-src/emacsclient.c b/lib-src/emacsclient.c
index 7769e015ed..3af4f7d0cf 100644
--- a/lib-src/emacsclient.c
+++ b/lib-src/emacsclient.c
@@ -1773,7 +1773,10 @@ start_daemon_and_retry_set_socket (void)
-      char emacs[] = "emacs";
+      const char *devel_build_rel = "/src/emacs/src/emacs";
+      char *emacs = xmalloc (strlen (egetenv ("HOME"))
+                + strlen (devel_build_rel) + 1);
+      strcpy ( stpcpy (emacs, egetenv ("HOME")), devel_build_rel);
       char daemon_option[] = "--daemon";
       char *d_argv[3];
       d_argv[0] = emacs;
@@ -1790,7 +1793,7 @@ start_daemon_and_retry_set_socket (void)
      d_argv[1] = daemon_arg;
 # endif
-      execvp ("emacs", d_argv);
+      execvp (emacs, d_argv);
       message (true, "%s: error starting emacs daemon\n", progname);
       exit (EXIT_FAILURE);

And I use something like this to have my Wayland compositor use these builds:


if [ -e "$HOME/src/emacs/admin/README.melete" \ # confirm we're on the branch
    -a -x "$HOME/src/emacs/lib-src/emacsclient" \
    -a -d "/usr/share/emacs-snapshot/site-lisp/elpa" ]; then
    EMACSCLIENT=$(command -v emacsclient)

$EMACSCLIENT -a "" -nc

There’s something similar in my shell configuration.

Posted Sun 26 Dec 2021 21:09:48 UTC

For some months now I’ve been working on some patches to Consfigurator to add support for Linux containers. My goal is to make Consfigurator capable of both performing the initial setup of a container and of entering the running container to apply configuration. For the case of unprivileged LXCs running as non-root, my work-in-progress branch can now do both of these things. As Consfigurator enters the container directly using system calls, it should be decently fast at configuring multiple containers on a host, and it will also be possible to have it do this in parallel. The initial setup for the container uses Consfigurator’s existing support for building root filesystems, and it should be easy to extend that to support arbitrary GNU/Linux distributions by teaching Consfigurator how to invoke bootstrapping tools other than debootstrap(8).

Here’s an example:

(defhost lxc1.silentflame.com ()
  (os:debian-stable "bullseye" :amd64)
  (apt:installed "systemd" "netcat")
  (apache:https-vhost ...))

(defhost lxctest.laptop.silentflame.com ()
  (os:debian-stable "bullseye" :amd64)
  (apt:proxy "")
  (apt:installed "linux-image-amd64" "lxc")

  (lxc:usernet-usable-by "spwhitton" "lxcbr0")
  (lxc:user-containers-autostart "spwhitton")
  (lxc:user-container-for '(:additional-lines
                            ("lxc.net.0.type = veth"
                             "lxc.net.0.flags = up"
                             "lxc.net.0.link = lxcbr0"

(defhost laptop.silentflame.com ()
  (libvirt:kvm-boots-chroot-for '(:always-deploys t)

This code is a simplified definition of my testing setup for this work. It defines three hosts: a container lxc1, a container host lxctest, and my laptop. When Consfigurator is asked to deploy the laptop, it will set up the root filesystem for lxctest and then boot it as a KVM virtual machine. Preparing that root filesystem will include setting up the root filesystem for lxc1, too, including shifting the ownership and ACLs to match the user namespace LXC will use when booting the container. Thus, once the deployment of the laptop is finished, it will be possible to boot the lxctest VM, connect to it as the user spwhitton, and start lxc1.

Consfigurator includes only minimal support for setting up container networking, as there are so many different ways in which you might want to do it. In my own consfig I’ve been developing properties to connect containers directly to my tinc VPN. A single tinc daemon runs on the container host, and other tinc daemons route a whole subnet, containing the addresses for each of the containers, to the container host’s tinc daemon. As the LXCs Consfigurator sets up run as non-root, some sort of setuid facility is required to configure this networking. Consfigurator’s ability to dump executable Lisp images is helping here. I define a function which runs as root to set up the networking:

(defun route-athenet-container-veth (host)
  (let ((user (getenv "USERV_USER"))
        (peer (getenv "USERV_U_PEER"))
        (ip (car (uiop:command-line-arguments))))
    (unless (string-prefix-p (format nil "veth~D_" (getenv "USERV_UID")) peer)
      (error "~A does not belong to requester." peer))
    (unless (member (cons user ip) (get-hostattrs 'veth-ips host) :test #'equal)
      (error "~A does not have permission to route ~A." user ip))
    (flet ((r (&rest args)
             ;; Explicitly passing nil means UIOP will not invoke a shell.
             (run-program args :force-shell nil)))
      (eswitch ((getenv "USERV_U_HOOK_TYPE") :test #'string=)
         (apply #'r
                "sysctl" "-w"
         (r "ip" "addr" "flush" "dev" peer "scope" "link")
         (r "ip" "-6" "addr" "add" "fe80::1/64" "dev" peer)
         (r "ip" "-6" "route" "add" (strcat ip "/128") "dev" peer)
         (r "ip" "-6" "route" "del" (strcat ip "/128") "dev" peer))))))

and then apply the following property to lxctest to dump an image which will call this function and then exit:

 `(route-athenet-container-veth ,(intern (string-upcase (get-hostname)))))

I’m using GNU userv to enable ordinary users to run this image as root, so there there’s a small script which converts LXC’s LXC_HOOK_* environment variables into appropriate command line arguments to userv(1) such that the function above is able to access that information from its environment (the USERV_U_* variables above). You could just as easily do this with sudo, by giving permission for the relevant LXC_HOOK_* environment variables to survive the switch to root.

What’s particularly nice about this is that there’s no need to write any code to keep a config file updated, specifying which users are allowed to route which IPs to their containers. ROUTE-ATHENET-CONTAINER-VETH receives a HOST value for the container host and can just look at the metadata established by properties for particular containers. Each time this metadata is updated and lxctest is deployed, a fresh image is dumped containing the updated metadata.

This work has provided opportunities to make various other improvements to Consfigurator, especially with regard to dumping and reinvoking images. Making SBCL capable of entering user namespaces required a change upstream, which made it into the recent SBCL 2.1.8 release. I’m very grateful to the SBCL developers for their engagement with my project. I’ve been able to add a workaround so that Consfigurator can still enter user namespaces when run on the version of SBCL included in Debian stable. I also discovered that deploying all of my laptop, lxctest and lxc1 at once generates enough output to fill up a pipe, thus revealing a deadlock in Consfigurator’s IPC, which it was good to become aware of and fix. That involved writing my first multi-threaded Lisp, as there are two pipes that need to be kept from filling up, and to my surprise it worked first time. Take that Haskell :)

Posted Sun 24 Oct 2021 16:17:51 UTC

Yesterday I upgraded a machine from Debian “buster” to “bullseye” without apt-listchanges installed, oops. Here’s a way to get new NEWS.Debian entries after the fact.

perl -MDpkg::Changelog::Debian -wE'$parser = Dpkg::Changelog::Debian->new;
    for (</usr/share/doc/*/NEWS.Debian*>) {
        $_->get_timepiece && $_->get_timepiece->year < 2020 ? last : say
            for $parser->@* }' 2>&1 | mail -s"News for $(hostname)" you@example.com
Posted Sun 15 Aug 2021 21:55:44 UTC

I realised this week that my recent efforts to improve how Consfigurator makes the fork(2) system call have also created a way to install executables to remote systems which will execute arbitrary Common Lisp code. Distributing precompiled programs using free software implementations of the Common Lisp standard tends to be more of a hassle than with a lot of other high level programming languages. Executables will often be hundreds of megabytes in size even if your codebase is just a few megabytes, because the whole interactive Common Lisp environment gets bundled along with your program’s code. Commercial Common Lisp implementations manage to do better, as I understand it, by knowing how to shake out unused code paths. Consfigurator’s new mechanism uploads only changed source code, which might only be kilobytes in size, and updates the executable on the remote system. So it should be useful for deploying Common Lisp-powered web services, and the like.

Here’s how it works. When you use Consfigurator you define an ASDF system – analagous to a Python package or Perl distribution – called your “consfig”. This defines HOST objects to represent the machines that you’ll use Consfigurator to manage, and any custom properties, functions those properties call, etc.. An ASDF system can depend upon other systems; for example, every consfig depends upon Consfigurator itself. When you execute Consfigurator deployments, Consfigurator uploads the source code of any ASDF systems that have changed since you last deployed this host, starts up Lisp on the remote machine, and loads up all the systems. Now the remote Lisp image is in a similarly clean state to when you’ve just started up Lisp on your laptop and loaded up the libraries you’re going to use. Only then are the actual deployment instructions are sent on stdin.

What I’ve done this week is insert an extra step for the remote Lisp image in between loading up all the ASDF systems and reading the deployment from stdin: the image calls fork(2) and establishes a pipe to communicate with the child process. The child process can be sent Lisp forms to evaluate, but for each Lisp form it receives it will actually fork again, and have its child process evaluate the form. Thus, going into the deployment, the original remote Lisp image has the capability to have arbitrary Lisp forms evaluated in a context in which all that has happened is that a statically defined set of ASDF systems has been loaded – the child processes never see the full deployment instructions sent on stdin. Further, the child process responsible for actually evaluating the Lisp form received from the first process first forks off another child process and sets up its own control pipe, such that it too has the capacbility to have arbitrary Lisp forms evaluated in a cleanly loaded context, no matter what else it might put in its memory in the meantime. (Things are set up such that the child processes responsible for actually evaluating the Lisp forms never see the Lisp forms received for evaluation by other child processes, either.)

So suppose now we have an ASDF system :com.silentflame.cool-web-service, and there is a function (start-server PORT) which we should call to start listening for connections. Then we can make our consfig depend upon that ASDF system, and do something like this:

CONSFIG> (deploy-these ((:ssh :user "root") :sbcl) server.example.org
           ;; Set up Apache to proxy requests to our service.
           (apache:https-vhost ...)
           ;; Now apply a property to dump the image.
           (image-dumped "/usr/local/bin/cool-web-service"
                         '(cool-web-service:start-server 1234)))

Consfigurator will: SSH to server.example.org; upload all the ASDF source for your consfig and its dependencies; compile and load that code into a remote SBCL process; call fork(2) and set up the control pipe; receive the applications of APACHE:HTTPS-VHOST and IMAGE-DUMPED shown above from your laptop, on stdin; apply the APACHE:HTTPS-VHOST property to ensure that Apache is proxying connections to port 1234; send a request into the control pipe to have the child process fork again and dump an executable which, when started, will evaluate the form (cool-web-service:start-server 1234). And that form will get evaluated in a pristine Lisp image, where the only meaningful things that have happened is that some ASDF systems have been loaded and a single fork(2) has taken place. You’d probably need to add some other properties to add some mechanism for actually invoking /usr/local/bin/cool-web-service and restarting it when the executable is updated.

(Background: The primary reason why Consfigurator’s remote Lisp images need to call fork(2) is that they need to do things like setuid from root to other accounts and enter chroots without getting stuck in those contexts. Previously we forked right before entering such contexts, but that meant that Consfigurator deployments could never be multithreaded, because it might later be necessary to fork, and you can’t usually do that once you’ve got more than one thread running. So now we fork before doing anything else, so that the parent can then go multithreaded if desired, but can still execute subdeployments in contexts like chroots by sending Lisp forms to evaluate in those contexts into the control pipe.)

Posted Wed 21 Jul 2021 20:30:09 UTC Tags:

Tonight I’m provisioning a new virtual machine at Hetzner and I wanted to share how Consfigurator is helping with that. Hetzner have a Debian “buster” image you can start with, as you’d expect, but it comes with things like cloud-init, preconfiguration to use Hetzner’s apt mirror which doesn’t serve source packages(!), and perhaps other things I haven’t discovered. It’s a fine place to begin, but I want all the configuration for this server to be explicit in my Consfigurator consfig, so it is good to start with pristine upstream Debian. I could boot one of Hetzner’s installation ISOs but that’s slow and manual. Consfigurator can replace the OS in the VM’s root filesystem and reboot for me, and we’re ready to go.

Here’s the configuration:

(defhost foo.silentflame.com (:deploy ((:ssh :user "root") :sbcl))
  (os:debian-stable "buster" :amd64)

  ;; Hetzner's Debian 10 image comes with a three-partition layout and boots
  ;; with traditional BIOS.
    :device-file "/dev/sda" :boots-with '(grub:grub :target "i386-pc")))

  (on-change (installer:cleanly-installed-once
              ;; This is a specification of the OS Hetzner's image has, so
              ;; Consfigurator knows how to install SBCL and debootstrap(8).
              ;; In this case it's the same Debian release as the replacement.
              '(os:debian-stable "buster" :amd64))

    ;; Clear out the old OS's EFI system partition contents, in case we can
    ;; switch to booting with EFI at some point (if we wanted we could specify
    ;; an additional x86_64-efi target above, and grub-install would get run
    ;; to repopulate /boot/efi, but I don't think Hetzner can boot from it yet).
    (file:directory-does-not-exist "/boot/efi/EFI")

    (apt:installed "linux-image-amd64")

       (mounted-ext4-filesystem :mount-point "/")
         :mount-options '("umask=0077") :mount-point "/boot/efi"))))
    (file:lacks-lines "/etc/fstab" "# UNCONFIGURED FSTAB FOR BASE SYSTEM")

    (file:is-copy-of "/etc/resolv.conf" "/old-os/etc/resolv.conf")
    (mount:unmounted-below-and-removed "/old-os"))

  (apt:mirror "http://ftp.de.debian.org/debian")
  (as "root" (ssh:authorized-keys +spwsshkey+))
  (timezone:configured "Etc/UTC")
  (swap:has-swap-file "2G")

  (network:static "enp1s0" "xxx.xxx.xxx.xxx" "xxx.xxx.1.1" ""))

and to use it you evaluate this at the REPL:

CONSFIG> (deploy ((:ssh :user "root" :hop "xxx.xxx.xxx.xxx") :sbcl) foo.silentflame.com)

Here the :HOP parameter specifies the IP address of the new machine, as DNS hasn’t been updated yet. Consfigurator installs SBCL and debootstrap(8), prepares a minimal system, replaces the contents of /, gets to work applying the other properties, and then reboots. This gets us a properly populated fstab:

UUID=...            /           ext4    relatime    0   1
PARTUUID=...        /boot/efi   vfat    umask=0077  0   2
/var/lib/swapfile   swap        swap    defaults    0   0

(slightly doctored for more readable alignment)

There’s ordering logic so that the swapfile will end up after whatever filesystem contains it; a UUID is used for ext4 filesystems, but for fat32 filesystems, to be safe, a PARTUUID is used.

The application of (INSTALLER:BOOTLOADERS-INSTALLED) handles calling both update-grub(8) and grub-install(8), relying on the metadata specified about /dev/sda. Next time we execute Consfigurator against the machine, it’ll ignore all the property applications attached to the application of (INSTALLER:CLEANLY-INSTALLED-ONCE) with ON-CHANGE, and just apply everything following that block.

There are a few things I don’t have good solutions for. When you boot Hetzner’s image the primary network interface is eth0, but then for a freshly debootstrapped Debian you get enp1s0, and I haven’t got a good way of knowing what it’ll be (if you know it’ll have the same name, you can use (NETWORK:PRESERVE-STATIC-ONCE) to create a file in /etc/network/interfaces.d based on the current default route and corresponding interface).

Another tricky thing is SSH host keys. It’s easy to use Consfigurator to add host keys to your laptop’s ~/.ssh/known_hosts, but in this case the host key changes back and forth from whatever the Hetzner image has and the newly generated key you get afterwards. One option might be to copy the old host keys out of /old-os before it gets deleted, like how /etc/resolv.conf is copied.

This work is based on Propellor’s equivalent functionality. I think my approach to handling /etc/fstab and bootloader installation is an improvement on what Joey does.

Posted Sat 10 Jul 2021 04:20:43 UTC Tags: