I’ve been blogging since 2005, but posts from 2005–10 have not been imported to the current iteration of my website.

Debian contributors end up with a number of Debian installations on their workstations for various purposes; there are perhaps five or six on my laptop right now. In addition to this, I now have three copies of GNU Emacs, too.

/usr/bin/emacs

This is the version of Emacs in Debian stable. I have to keep it available for testing various of the Debian packages I maintain.

/opt/emacs-snapshot/bin/emacs

This is a build of Emacs 29.0.50 out of upstream git, with native Wayland support, and some custom Debian packaging to integrate it with Debian’s emacsen-common infrastructure: Emacs addon packages are pre-bytecompiled against this version of Emacs, and re-bytecompilation is fully automated. This is nice if you want to use development snapshots of Emacs but get your addon packages from Debian, as I do. These are, roughly, the extent of my changes on top of upstream:

diff --git a/.gitignore b/.gitignore
index 78557a5e87..9743ccca0c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,6 +23,11 @@
 # in Git 1.8.2 (March 2013).


+# Debian packaging
+config.guess
+config.sub
+debian/emacs-snapshot/
+
 # Personal customization.
 .dir-locals-2.el

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index de4cebccca..1596b5a1bf 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -311,6 +311,7 @@ package-directory-list
       (and (stringp f)
            (equal (file-name-nondirectory f) "site-lisp")
            (push (expand-file-name "elpa" f) result)))
+    (push "/usr/share/emacs-snapshot/site-lisp/elpa" result)
     (nreverse result))
   "List of additional directories containing Emacs Lisp packages.
 Each directory name should be absolute.
diff --git a/lisp/startup.el b/lisp/startup.el
index b79467339b..c54ab89b33 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1374,7 +1374,12 @@ command-line
     ;; be loaded from site-run-file and wants to test if -q was given
     ;; should check init-file-user instead, since that is already set.
     ;; See cus-edit.el for an example.
-    (if site-run-file
+    (when site-run-file
+        (let ((file "/etc/emacs/site-start.d/00debian.el"))
+          ;; When the Emacs build invokes Emacs, such as in the
+          ;; Makefile rule for ${unidir}/emoji-labels.el, 00debian.el
+          ;; might not exist.  Should be fine to just skip the load.
+          (when (file-readable-p file) (load file)))
         ;; Sites should not disable the startup screen.
         ;; Only individuals should disable the startup screen.
         (let ((inhibit-startup-screen inhibit-startup-screen))

If you’re using Wayland on Debian, or you just want or need a newer Emacs, you might like to install these snapshot builds yourself from http://silentflame.com/debian/, suites “bullseye-backports” and “unstable”.

~spwhitton/src/emacs/{lib-src/emacsclient,src/emacs}

The snapshot builds described in the previous section are what I use on machines other than my laptop. On my laptop, I have my Wayland compositor configured to use the emacs and emacsclient binaries in ~/src/emacs, if they exist. This is a build of a branch with work-in-progress patches I’m planning to send upstream, plus bug fixes, reversions of commits that introduced regressions which affect my setup, etc..

Why have this in addition to the emacs-snapshot.deb builds – if there are temporary fixup commits, why not apply them and build a .deb just for local use? Well, that is indeed what I was doing until recently, but if you’re developing patches for upstream Emacs it’s pretty inconvenient. Hyperlinks to function definitions in Help buffers take you somewhere read-only in /opt, for example, and rebuilds are slow because, by default, Debian package builds begin by cleaning the tree.

In addition to the diff above (minus the .gitignore change), and the temporary fixup commits, I also apply the following change on this branch. Having the above diff applied implies that a recent emacs-snapshot.deb is installed, to provide the byte-compiled addons. This informal dependency should work well enough so long as I update the snapshot .debs described in the previous section often enough.

diff --git a/lib-src/emacsclient.c b/lib-src/emacsclient.c
index 7769e015ed..3af4f7d0cf 100644
--- a/lib-src/emacsclient.c
+++ b/lib-src/emacsclient.c
@@ -1773,7 +1773,10 @@ start_daemon_and_retry_set_socket (void)
     }
   else
     {
-      char emacs[] = "emacs";
+      const char *devel_build_rel = "/src/emacs/src/emacs";
+      char *emacs = xmalloc (strlen (egetenv ("HOME"))
+                + strlen (devel_build_rel) + 1);
+      strcpy ( stpcpy (emacs, egetenv ("HOME")), devel_build_rel);
       char daemon_option[] = "--daemon";
       char *d_argv[3];
       d_argv[0] = emacs;
@@ -1790,7 +1793,7 @@ start_daemon_and_retry_set_socket (void)
      d_argv[1] = daemon_arg;
    }
 # endif
-      execvp ("emacs", d_argv);
+      execvp (emacs, d_argv);
       message (true, "%s: error starting emacs daemon\n", progname);
       exit (EXIT_FAILURE);
     }

And I use something like this to have my Wayland compositor use these builds:

#!/bin/sh

if [ -e "$HOME/src/emacs/admin/README.melete" \ # confirm we're on the branch
    -a -x "$HOME/src/emacs/lib-src/emacsclient" \
    -a -d "/usr/share/emacs-snapshot/site-lisp/elpa" ]; then
    EMACSCLIENT="$HOME/src/emacs/lib-src/emacsclient"
else
    EMACSCLIENT=$(command -v emacsclient)
fi

$EMACSCLIENT -a "" -nc

There’s something similar in my shell configuration.

Posted Sun 26 Dec 2021 21:09:48 UTC

For some months now I’ve been working on some patches to Consfigurator to add support for Linux containers. My goal is to make Consfigurator capable of both performing the initial setup of a container and of entering the running container to apply configuration. For the case of unprivileged LXCs running as non-root, my work-in-progress branch can now do both of these things. As Consfigurator enters the container directly using system calls, it should be decently fast at configuring multiple containers on a host, and it will also be possible to have it do this in parallel. The initial setup for the container uses Consfigurator’s existing support for building root filesystems, and it should be easy to extend that to support arbitrary GNU/Linux distributions by teaching Consfigurator how to invoke bootstrapping tools other than debootstrap(8).

Here’s an example:

(defhost lxc1.silentflame.com ()
  (os:debian-stable "bullseye" :amd64)
  (basic-props)
  (apt:installed "systemd" "netcat")
  (apache:https-vhost ...))

(defhost lxctest.laptop.silentflame.com ()
  (os:debian-stable "bullseye" :amd64)
  (apt:proxy "http://192.168.122.1:3142")
  (basic-props)
  (apt:installed "linux-image-amd64" "lxc")

  (lxc:usernet-usable-by "spwhitton" "lxcbr0")
  (lxc:user-containers-autostart "spwhitton")
  (lxc:user-container-for '(:additional-lines
                            ("lxc.net.0.type = veth"
                             "lxc.net.0.flags = up"
                             "lxc.net.0.link = lxcbr0"
                             ...))
                          "spwhitton"
                          lxc1.silentflame.com))

(defhost laptop.silentflame.com ()
  ...
  (libvirt:kvm-boots-chroot-for '(:always-deploys t)
                                lxctest.laptop.silentflame.com))

This code is a simplified definition of my testing setup for this work. It defines three hosts: a container lxc1, a container host lxctest, and my laptop. When Consfigurator is asked to deploy the laptop, it will set up the root filesystem for lxctest and then boot it as a KVM virtual machine. Preparing that root filesystem will include setting up the root filesystem for lxc1, too, including shifting the ownership and ACLs to match the user namespace LXC will use when booting the container. Thus, once the deployment of the laptop is finished, it will be possible to boot the lxctest VM, connect to it as the user spwhitton, and start lxc1.

Consfigurator includes only minimal support for setting up container networking, as there are so many different ways in which you might want to do it. In my own consfig I’ve been developing properties to connect containers directly to my tinc VPN. A single tinc daemon runs on the container host, and other tinc daemons route a whole subnet, containing the addresses for each of the containers, to the container host’s tinc daemon. As the LXCs Consfigurator sets up run as non-root, some sort of setuid facility is required to configure this networking. Consfigurator’s ability to dump executable Lisp images is helping here. I define a function which runs as root to set up the networking:

(defun route-athenet-container-veth (host)
  (let ((user (getenv "USERV_USER"))
        (peer (getenv "USERV_U_PEER"))
        (ip (car (uiop:command-line-arguments))))
    (unless (string-prefix-p (format nil "veth~D_" (getenv "USERV_UID")) peer)
      (error "~A does not belong to requester." peer))
    (unless (member (cons user ip) (get-hostattrs 'veth-ips host) :test #'equal)
      (error "~A does not have permission to route ~A." user ip))
    (flet ((r (&rest args)
             ;; Explicitly passing nil means UIOP will not invoke a shell.
             (run-program args :force-shell nil)))
      (eswitch ((getenv "USERV_U_HOOK_TYPE") :test #'string=)
        ("up"
         (apply #'r
                "sysctl" "-w"
                "net.ipv6.conf.all.forwarding=1"
                ...)
         (r "ip" "addr" "flush" "dev" peer "scope" "link")
         (r "ip" "-6" "addr" "add" "fe80::1/64" "dev" peer)
         (r "ip" "-6" "route" "add" (strcat ip "/128") "dev" peer)
         ...)
        ("down"
         ...
         (r "ip" "-6" "route" "del" (strcat ip "/128") "dev" peer))))))

and then apply the following property to lxctest to dump an image which will call this function and then exit:

(image-dumped
 "/usr/lib/userv/route-athenet-container-veth"
 `(route-athenet-container-veth ,(intern (string-upcase (get-hostname)))))

I’m using GNU userv to enable ordinary users to run this image as root, so there there’s a small script which converts LXC’s LXC_HOOK_* environment variables into appropriate command line arguments to userv(1) such that the function above is able to access that information from its environment (the USERV_U_* variables above). You could just as easily do this with sudo, by giving permission for the relevant LXC_HOOK_* environment variables to survive the switch to root.

What’s particularly nice about this is that there’s no need to write any code to keep a config file updated, specifying which users are allowed to route which IPs to their containers. ROUTE-ATHENET-CONTAINER-VETH receives a HOST value for the container host and can just look at the metadata established by properties for particular containers. Each time this metadata is updated and lxctest is deployed, a fresh image is dumped containing the updated metadata.

This work has provided opportunities to make various other improvements to Consfigurator, especially with regard to dumping and reinvoking images. Making SBCL capable of entering user namespaces required a change upstream, which made it into the recent SBCL 2.1.8 release. I’m very grateful to the SBCL developers for their engagement with my project. I’ve been able to add a workaround so that Consfigurator can still enter user namespaces when run on the version of SBCL included in Debian stable. I also discovered that deploying all of my laptop, lxctest and lxc1 at once generates enough output to fill up a pipe, thus revealing a deadlock in Consfigurator’s IPC, which it was good to become aware of and fix. That involved writing my first multi-threaded Lisp, as there are two pipes that need to be kept from filling up, and to my surprise it worked first time. Take that Haskell :)

Posted Sun 24 Oct 2021 16:17:51 UTC

Yesterday I upgraded a machine from Debian “buster” to “bullseye” without apt-listchanges installed, oops. Here’s a way to get new NEWS.Debian entries after the fact.

perl -MDpkg::Changelog::Debian -wE'$parser = Dpkg::Changelog::Debian->new;
    for (</usr/share/doc/*/NEWS.Debian*>) {
        $parser->load($_);
        $_->get_timepiece && $_->get_timepiece->year < 2020 ? last : say
            for $parser->@* }' 2>&1 | mail -s"News for $(hostname)" you@example.com
Posted Sun 15 Aug 2021 21:55:44 UTC

I realised this week that my recent efforts to improve how Consfigurator makes the fork(2) system call have also created a way to install executables to remote systems which will execute arbitrary Common Lisp code. Distributing precompiled programs using free software implementations of the Common Lisp standard tends to be more of a hassle than with a lot of other high level programming languages. Executables will often be hundreds of megabytes in size even if your codebase is just a few megabytes, because the whole interactive Common Lisp environment gets bundled along with your program’s code. Commercial Common Lisp implementations manage to do better, as I understand it, by knowing how to shake out unused code paths. Consfigurator’s new mechanism uploads only changed source code, which might only be kilobytes in size, and updates the executable on the remote system. So it should be useful for deploying Common Lisp-powered web services, and the like.

Here’s how it works. When you use Consfigurator you define an ASDF system – analagous to a Python package or Perl distribution – called your “consfig”. This defines HOST objects to represent the machines that you’ll use Consfigurator to manage, and any custom properties, functions those properties call, etc.. An ASDF system can depend upon other systems; for example, every consfig depends upon Consfigurator itself. When you execute Consfigurator deployments, Consfigurator uploads the source code of any ASDF systems that have changed since you last deployed this host, starts up Lisp on the remote machine, and loads up all the systems. Now the remote Lisp image is in a similarly clean state to when you’ve just started up Lisp on your laptop and loaded up the libraries you’re going to use. Only then are the actual deployment instructions are sent on stdin.

What I’ve done this week is insert an extra step for the remote Lisp image in between loading up all the ASDF systems and reading the deployment from stdin: the image calls fork(2) and establishes a pipe to communicate with the child process. The child process can be sent Lisp forms to evaluate, but for each Lisp form it receives it will actually fork again, and have its child process evaluate the form. Thus, going into the deployment, the original remote Lisp image has the capability to have arbitrary Lisp forms evaluated in a context in which all that has happened is that a statically defined set of ASDF systems has been loaded – the child processes never see the full deployment instructions sent on stdin. Further, the child process responsible for actually evaluating the Lisp form received from the first process first forks off another child process and sets up its own control pipe, such that it too has the capacbility to have arbitrary Lisp forms evaluated in a cleanly loaded context, no matter what else it might put in its memory in the meantime. (Things are set up such that the child processes responsible for actually evaluating the Lisp forms never see the Lisp forms received for evaluation by other child processes, either.)

So suppose now we have an ASDF system :com.silentflame.cool-web-service, and there is a function (start-server PORT) which we should call to start listening for connections. Then we can make our consfig depend upon that ASDF system, and do something like this:

CONSFIG> (deploy-these ((:ssh :user "root") :sbcl) server.example.org
           ;; Set up Apache to proxy requests to our service.
           (apache:https-vhost ...)
           ;; Now apply a property to dump the image.
           (image-dumped "/usr/local/bin/cool-web-service"
                         '(cool-web-service:start-server 1234)))

Consfigurator will: SSH to server.example.org; upload all the ASDF source for your consfig and its dependencies; compile and load that code into a remote SBCL process; call fork(2) and set up the control pipe; receive the applications of APACHE:HTTPS-VHOST and IMAGE-DUMPED shown above from your laptop, on stdin; apply the APACHE:HTTPS-VHOST property to ensure that Apache is proxying connections to port 1234; send a request into the control pipe to have the child process fork again and dump an executable which, when started, will evaluate the form (cool-web-service:start-server 1234). And that form will get evaluated in a pristine Lisp image, where the only meaningful things that have happened is that some ASDF systems have been loaded and a single fork(2) has taken place. You’d probably need to add some other properties to add some mechanism for actually invoking /usr/local/bin/cool-web-service and restarting it when the executable is updated.

(Background: The primary reason why Consfigurator’s remote Lisp images need to call fork(2) is that they need to do things like setuid from root to other accounts and enter chroots without getting stuck in those contexts. Previously we forked right before entering such contexts, but that meant that Consfigurator deployments could never be multithreaded, because it might later be necessary to fork, and you can’t usually do that once you’ve got more than one thread running. So now we fork before doing anything else, so that the parent can then go multithreaded if desired, but can still execute subdeployments in contexts like chroots by sending Lisp forms to evaluate in those contexts into the control pipe.)

Posted Wed 21 Jul 2021 20:30:09 UTC Tags:

Tonight I’m provisioning a new virtual machine at Hetzner and I wanted to share how Consfigurator is helping with that. Hetzner have a Debian “buster” image you can start with, as you’d expect, but it comes with things like cloud-init, preconfiguration to use Hetzner’s apt mirror which doesn’t serve source packages(!), and perhaps other things I haven’t discovered. It’s a fine place to begin, but I want all the configuration for this server to be explicit in my Consfigurator consfig, so it is good to start with pristine upstream Debian. I could boot one of Hetzner’s installation ISOs but that’s slow and manual. Consfigurator can replace the OS in the VM’s root filesystem and reboot for me, and we’re ready to go.

Here’s the configuration:

(defhost foo.silentflame.com (:deploy ((:ssh :user "root") :sbcl))
  (os:debian-stable "buster" :amd64)

  ;; Hetzner's Debian 10 image comes with a three-partition layout and boots
  ;; with traditional BIOS.
  (disk:has-volumes
   (physical-disk
    :device-file "/dev/sda" :boots-with '(grub:grub :target "i386-pc")))

  (on-change (installer:cleanly-installed-once
              nil
              ;; This is a specification of the OS Hetzner's image has, so
              ;; Consfigurator knows how to install SBCL and debootstrap(8).
              ;; In this case it's the same Debian release as the replacement.
              '(os:debian-stable "buster" :amd64))

    ;; Clear out the old OS's EFI system partition contents, in case we can
    ;; switch to booting with EFI at some point (if we wanted we could specify
    ;; an additional x86_64-efi target above, and grub-install would get run
    ;; to repopulate /boot/efi, but I don't think Hetzner can boot from it yet).
    (file:directory-does-not-exist "/boot/efi/EFI")

    (apt:installed "linux-image-amd64")
    (installer:bootloaders-installed)

    (fstab:entries-for-volumes
     (disk:volumes
       (mounted-ext4-filesystem :mount-point "/")
       (partition
        (mounted-fat32-filesystem
         :mount-options '("umask=0077") :mount-point "/boot/efi"))))
    (file:lacks-lines "/etc/fstab" "# UNCONFIGURED FSTAB FOR BASE SYSTEM")

    (file:is-copy-of "/etc/resolv.conf" "/old-os/etc/resolv.conf")
    (mount:unmounted-below-and-removed "/old-os"))

  (apt:mirror "http://ftp.de.debian.org/debian")
  (apt:no-pdiffs)
  (apt:standard-sources.list)
  (sshd:installed)
  (as "root" (ssh:authorized-keys +spwsshkey+))
  (sshd:no-passwords)
  (timezone:configured "Etc/UTC")
  (swap:has-swap-file "2G")

  (network:clean-/etc/network/interfaces)
  (network:static "enp1s0" "xxx.xxx.xxx.xxx" "xxx.xxx.1.1" "255.255.255.255"))

and to use it you evaluate this at the REPL:

CONSFIG> (deploy ((:ssh :user "root" :hop "xxx.xxx.xxx.xxx") :sbcl) foo.silentflame.com)

Here the :HOP parameter specifies the IP address of the new machine, as DNS hasn’t been updated yet. Consfigurator installs SBCL and debootstrap(8), prepares a minimal system, replaces the contents of /, gets to work applying the other properties, and then reboots. This gets us a properly populated fstab:

UUID=...            /           ext4    relatime    0   1
PARTUUID=...        /boot/efi   vfat    umask=0077  0   2
/var/lib/swapfile   swap        swap    defaults    0   0

(slightly doctored for more readable alignment)

There’s ordering logic so that the swapfile will end up after whatever filesystem contains it; a UUID is used for ext4 filesystems, but for fat32 filesystems, to be safe, a PARTUUID is used.

The application of (INSTALLER:BOOTLOADERS-INSTALLED) handles calling both update-grub(8) and grub-install(8), relying on the metadata specified about /dev/sda. Next time we execute Consfigurator against the machine, it’ll ignore all the property applications attached to the application of (INSTALLER:CLEANLY-INSTALLED-ONCE) with ON-CHANGE, and just apply everything following that block.

There are a few things I don’t have good solutions for. When you boot Hetzner’s image the primary network interface is eth0, but then for a freshly debootstrapped Debian you get enp1s0, and I haven’t got a good way of knowing what it’ll be (if you know it’ll have the same name, you can use (NETWORK:PRESERVE-STATIC-ONCE) to create a file in /etc/network/interfaces.d based on the current default route and corresponding interface).

Another tricky thing is SSH host keys. It’s easy to use Consfigurator to add host keys to your laptop’s ~/.ssh/known_hosts, but in this case the host key changes back and forth from whatever the Hetzner image has and the newly generated key you get afterwards. One option might be to copy the old host keys out of /old-os before it gets deleted, like how /etc/resolv.conf is copied.

This work is based on Propellor’s equivalent functionality. I think my approach to handling /etc/fstab and bootloader installation is an improvement on what Joey does.

Posted Sat 10 Jul 2021 04:20:43 UTC Tags:

If you’re writing a lot of Common Lisp and you want to follow the convention of using all uppercase to refer to symbols in docstrings, comments etc., you really need something better than the shift key. Similarly if you’re writing C and you have VARIOUS_LONG_ENUMS.

The traditional way is a caps lock key. But that means giving up a whole keyboard key, all of the time, just for block capitalisation, which one hardly uses outside of programming. So a better alternative is to come up with some Emacs thing to get block capitalisation, as Emacs key binding is much more flexible than system keyboard layouts, and can let us get block capitalisation without giving up a whole key.

The simplest thing would be to bind some sequence of keys to just toggle caps lock. But I came up with something a bit fancier. With the following, you can type M-C, and then you get block caps until the point at which you’ve probably finished typing your symbol or enum name.

(defun spw/transient-caps-self-insert (&optional n)
  (interactive "p")
  (insert-char (upcase last-command-event) n))

(defun spw/activate-transient-caps ()
  "Activate caps lock while typing the current whitespace-delimited word(s).
This is useful for typing Lisp symbols and C enums which consist
of several all-uppercase words separated by hyphens and
underscores, such that M-- M-u after typing will not upcase the
whole thing."
  (interactive)
  (let* ((map (make-sparse-keymap))
     (deletion-commands '(delete-backward-char
                          paredit-backward-delete
                          backward-kill-word
                          paredit-backward-kill-word
                          spw/unix-word-rubout
                          spw/paredit-unix-word-rubout))
     (typing-commands (cons 'spw/transient-caps-self-insert
                            deletion-commands)))
     (substitute-key-definition 'self-insert-command
                                #'spw/transient-caps-self-insert
                                map
                                (current-global-map))
    (set-transient-map
     map
     (lambda ()
       ;; try to determine whether we are probably still about to try to type
       ;; something in all-uppercase
       (and (member this-command typing-commands)
            (not (and (eq this-command 'spw/transient-caps-self-insert)
                      (= (char-syntax last-command-event) ?\ )))
            (not (and (or (bolp) (= (char-syntax (char-before)) ?\ ))
                      (member this-command deletion-commands))))))))

(global-set-key "\M-C" #'spw/activate-transient-caps)

A few notes:

  • I have caps lock on left-ctrl on standard keyboard layouts, and on the sequence keypd-capslock-keypd on the Kinesis Advantage. These are about equally inconvenient, but good enough for those rare cases one needs caps lock outside of Emacs.

  • I also had the idea of binding this to Delete, because I don’t use that at all in Emacs, but Delete is relatively hard to hit on conventional keyboards, sometimes missing, and might not work in some text terminals.

  • I’ve actually trained myself to set the mark, type my symbol or enum and then just C-x C-u to upcase it, so I’m not actually using the above, but I thought someone else might like it, so still worth posting.

Posted Sun 20 Jun 2021 00:26:03 UTC

I recently bought a Pinebook Pro. This was mainly out of general interest, but also because I wanted to have a spare portable computer. When I was recently having some difficulty with my laptop not charging, I realised that I am dependent on having access to Emacs, notmuch.el and my usual git repositories in the way that most people are dependent on their smartphones – all the info I need to get things done is in there, and it’s very disabling not to have it. So, good to have a spare.

I decided to get the machine running the hard way, and have been working to add a facility to install the device-specific bootloader to Consfigurator. It has been good to learn about how ARM machines boot. The only really hard part turned out to be coming up with the right abstractions within Consfigurator, thanks to the hard work of the Debian U-Boot maintainers. This left me with a chroot and a corresponding disk image, properly partitioned and with the bootloader installed. It was only then that the difficulties began: getting a kernel and initrd combination which can output to the Pinebook Pro’s screen and take input from its keyboard is not really straightforward yet, but that’s required for inputting disk encryption passwords, which are required on portable devices. I don’t have the right hardware to make a serial connection to the machine, so all this took a lot of trial and error. I’ve ended up using Manjaro’s patched upstream kernel build for now, because that compiles in the right drivers, and debugging an initrd without a serial connection is far too inefficient.

What I keep having to remind myself is that this device isn’t really a laptop in the usual sense – it’s a single board computer that’s powering several pieces of hardware which together roughly constitute a laptop. I think something which epitomises this is how the power light doesn’t come on when you hit the power button, but only when the bootloader or operating system kernel thinks to turn on the LED. You start up this SBC and it loads up some software and then once it has got itself going – several seconds later – that software starts turning on the screen, keyboard, power LEDs etc. Whereas on an ordinary laptop it’s more than you turn on the keyboard, screen, power LEDs etc. all at once, and then /they/ go off and load some software. Of course this description is nothing like what’s actually going on, but it’s my attempt to capture how it feels as a user, who is installing operating systems, but otherwise treating the laptop’s hardware, including things like boot ROMs, as a black box. There are tangible differences between what it is like to do that with an ordinary laptop and with the Pinebook Pro.

Thanks to Vagrant Cascadian for all the work on U-Boot in Debian and for help on IRC, Cyril Brulebois for help with crossbuilding, and Birger Schacht for a useful blog post.

Posted Sat 15 May 2021 00:24:48 UTC

One of my goals for Consfigurator is to make it capable of installing Debian to my laptop, so that I can stop booting to GRML and manually partitioning and debootstrapping a basic system, only to then turn to configuration management to set everything else up. My configuration management should be able to handle the partitioning and debootstrapping, too.

The first stage was to make Consfigurator capable of debootstrapping a basic system, chrooting into it, and applying other arbitrary configuration, such as installing packages. That’s been in place for some weeks now. It’s sophisticated enough to avoid starting up newly installed services, but I still need to add some bind mounting.

Another significant piece is teaching Consfigurator how to partition block devices. That’s quite tricky to do in a sufficiently general way – I want to cleanly support various combinations of LUKS, LVM and regular partitions, including populating /etc/crypttab and /etc/fstab. I have some ideas about how to do it, but it’ll probably take a few tries to get the abstractions right.

Let’s imagine that code is all in place, such that Consfigurator can be pointed at a block device and it will install a bootable Debian system to it. Then to install Debian to my laptop I’d just need to take my laptop’s disk drive out and plug it into another system, and run Consfigurator on that system, as root, pointed at the block device representing my laptop’s disk drive. For virtual machines, it would be easy to write code which loop-mounts an empty disk image, and then Consfigurator could be pointed at the loop-mounted block device, thereby making the disk image file bootable.

This is adequate for virtual machines, or small single-board computers with tiny storage devices (not that I actually use any of those, but I want Consfigurator to be able to make disk images for them!). But it’s not much good for my laptop. I casually referred to taking out my laptop’s disk drive and connecting it to another computer, but this would void my laptop’s warranty. And Consfigurator would not be able to update my laptop’s NVRAM, as is needed on UEFI systems.

What’s wanted here is a live system which can run Consfigurator directly on the laptop, pointed at the block device representing its physical disk drive. Ideally this live system comes with a chroot with the root filesystem for the new Debian install already built, so that network access is not required, and all Consfigurator has to do is partition the drive and copy in the contents of the chroot. The live system could be set up to automatically start doing that upon boot, but another option is to just make Consfigurator itself available to be used interactively. The user boots the live system, starts up Emacs, starts up Lisp, and executes a Consfigurator deployment, supplying the block device representing the laptop’s disk drive as an argument to the deployment. Consfigurator goes off and partitions that drive, copies in the contents of the chroot, and executes grub-install to make the laptop bootable. This is also much easier to debug than a live system which tries to start partitioning upon boot. It would look something like this:

    ;; melete.silentflame.com is a Consfigurator host object representing the
    ;; laptop, including information about the partitions it should have
    (deploy-these :local ...
      (chroot:partitioned-and-installed
        melete.silentflame.com "/srv/chroot/melete" "/dev/nvme0n1"))

Now, building live systems is a fair bit more involved than installing Debian to a disk drive and making it bootable, it turns out. While I want Consfigurator to be able to completely replace the Debian Installer, I decided that it is not worth trying to reimplement the relevant parts of the Debian Live tool suite, because I do not need to make arbitrary customisations to any live systems. I just need to have some packages installed and some files in place. Nevertheless, it is worth teaching Consfigurator how to invoke Debian Live, so that the customisation of the chroot which isn’t just a matter of passing options to lb_config(1) can be done with Consfigurator. This is what I’ve ended up with – in Consfigurator’s source code:

(defpropspec image-built :lisp (config dir properties)
  "Build an image under DIR using live-build(7), where the resulting live
system has PROPERTIES, which should contain, at a minimum, a property from
CONSFIGURATOR.PROPERTY.OS setting the Debian suite and architecture.  CONFIG
is a list of arguments to pass to lb_config(1), not including the '-a' and
'-d' options, which Consfigurator will supply based on PROPERTIES.

This property runs the lb_config(1), lb_bootstrap(1), lb_chroot(1) and
lb_binary(1) commands to build or rebuild the image.  Rebuilding occurs only
when changes to CONFIG or PROPERTIES mean that the image is potentially
out-of-date; e.g. if you just add some new items to PROPERTIES then in most
cases only lb_chroot(1) and lb_binary(1) will be re-run.

Note that lb_chroot(1) and lb_binary(1) both run after applying PROPERTIES,
and might undo some of their effects.  For example, to configure
/etc/apt/sources.list, you will need to use CONFIG not PROPERTIES."
  (:desc (declare (ignore config properties))
         #?"Debian Live image built in ${dir}")
  (let* (...)
    ;; ...
    `(eseqprops
      ;; ...
      (on-change
          (eseqprops
           (on-change
               (file:has-content ,auto/config ,(auto/config config) :mode #o755)
             (file:does-not-exist ,@clean)
             (%lbconfig ,dir)
             (%lbbootstrap t ,dir))
           (%lbbootstrap nil ,dir)
           (deploys ((:chroot :into ,chroot)) ,host))
        (%lbchroot ,dir)
        (%lbbinary ,dir)))))

Here, %lbconfig is a property running lb_config(1), %lbbootstrap one which runs lb_bootstrap(1), etc. Those properties all just change directory to the right place and run the command, essentially, with a little extra code to handle failed debootstraps and the like.

The ON-CHANGE and ESEQPROPS combinators work together to sequence the interaction of the Debian Live suite and Consfigurator.

  • In the innermost ON-CHANGE expression: create the file auto/config and populate it with the call to lb_config(1) that we need to make, as described in the Debian Live manual, chapter 6.

    • If doing so resulted in a change to the auto/config file – e.g. the user added some more options – ensure that lb_config(1) and lb_bootstrap(1) both get rerun.
  • Now in the inner ESEQPROPS expression, use DEPLOYS to configure the chroot, essentially by forking into the chroot and recursively reinvoking Consfigurator.

  • Finally, if any of the above resulted in a change being made, call lb_chroot(1) and lb_binary(1).

This way, we only rebuild the chroot if the configuration changed, and we only rebuild the image if the chroot changed.

Now over in my personal consfig:

(try-register-data-source
 :git-snapshot :name "consfig" :repo #P"src/cl/consfig/" ...)

(defproplist hybrid-live-iso-built :lisp ()
  "Build a Debian Live system in /srv/live/spw.

Typically this property is not applied in a DEFHOST form, but rather run as
needed at the REPL.  The reason for this is that otherwise the whole image will
get rebuilt each time a commit is made to my dotfiles repo or to my consfig."
  (:desc "Sean's Debian Live system image built")
  (live-build:image-built.
      '("--archive-areas" "main contrib non-free" ...)
      "/srv/live/spw"
    (os:debian-stable "buster" :amd64)
    (basic-props)
    (apt:installed "whatever" "you" "want")

    (git:snapshot-extracted "/etc/skel/src" "dotfiles")
    (file:is-copy-of "/etc/skel/.bashrc" "/etc/skel/src/dotfiles/.bashrc")

    (git:snapshot-extracted "/root/src/cl" "consfig")))

The first argument to LIVE-BUILD:IMAGE-BUILT. is additional arguments to lb_config(1). The third argument onwards are the properties for the live system. The cool thing is GIT:SNAPSHOT-EXTRACTED – the calls to this ensure that a copy of my Emacs configuration and my consfig end up in the live image, ready to be used interactively to install Debian, as described above. I’ll need to add something like (chroot:host-chroot-bootstrapped melete.silentflame.com "/srv/chroot/melete") too.

As with everything Consfigurator-related, Joey Hess’s Propellor is the giant upon whose shoulders I’m standing.

Posted Thu 08 Apr 2021 23:35:00 UTC

I was shocked to learn today that Richard Stallman has been reinstated as a member of the board of the Free Software Foundation. I think this is plain inappropriate, but I cannot see how anyone who doesn’t think that could fail to see the reinstatement as counterproductive. As Bradley M. Kuhn put it,

The question is whether an organization should have a designated leader who is on a sustained, public campaign advocating about an unrelated issue that many consider controversial. It really doesn’t matter what your view about the controversial issue is; a leader who refuses to stop talking loudly about unrelated issues eventually creates an untenable distraction from the radical activism you’re actively trying to advance. The message of universal software freedom is a radical cause; it’s basically impossible for one individual to effectively push forward two unrelated controversial agendas at once. In short, the radical message of software freedom became overshadowed by RMS’ radical views about sexual morality.

There is an open letter calling for the removal of the entire Board of the Free Software Foundation in response. I haven’t signed the letter because the Free Software Foundation Board’s vote to reinstate Stallman was not unanimous, so the call to remove all of them does not make sense to me. I agree with the open letter’s call to remove Stallman from other positions of leadership. I hope that this whole situation can be resolved quickly.

Posted Tue 23 Mar 2021 23:01:11 UTC

I had thought that Emacs’ C-t was mainly about correcting typos. It turns out to be extremely useful when working on Lisp macros which themselves write macros. This typically involves nested quasiquotation, where you can have multiple alternating sequences of open parentheses and backticks, or of commas, quotation marks and ampersats. While you’re working on it you often need to reorder these character sequences and C-t does a great job.

Posted Sat 13 Mar 2021 22:04:57 UTC