I’ve been blogging since 2005, but not all old posts have been imported here.

I’ve come up with a new reprepro wrapper for adding rebuilds of existing Debian packages to a local repository: reprepro-rebuilder. It should make it quicker to update local rebuilds of existing packages, patched or unpatched, working wholly out of git. Here’s how it works:

  1. Start with a git branch corresponding to the existing Debian package you want to rebuild. Probably you want dgit clone foo.

  2. Say reprepro-rebuilder unstable, and the script will switch you to a branch PREFIX/unstable, where PREFIX is a short name for your reprepro repository, and update debian/changelog for a local rebuild. If the branch already exists, it will be updated with a merge.

  3. You can now do any local patching you might require. Then, say reprepro-rebuilder --release. (The command from step (2) will offer to release immediately for the case that no additional patching is required.)

  4. At this point, your reprepro will contain a source package coresponding to your local rebuild. You can say reprepro-rebuilder --wanna-build to build any missing binaries for all suites, for localhost’s Debian architecture. (Again, the command from step (3) will offer to do this immediately after adding the source package.)

Additionally, if you’re rebuilding for unstable, reprepro-rebuilder will offer to rebuild for backports, too, and there are a few more convenience features, such as offering to build binaries for testing between steps (2) and (3). You can leave the script waiting to release while you do the testing.

I think that the main value of this script is keeping track of the distinct steps of a relatively fiddly, potentially slow-running workflow for you, including offering to perform your likely next step immediately. This means that you can be doing something else while the rebuilds are trundling along: you just start reprepro-rebuilder unstable in a shell, and unless additional patching is required between steps (2) and (3), you just have to answer script prompts as they show up and everything gets done.

If you need to merge from upstream fairly regularly, and then produce binary packages for both unstable and backports, that’s quite a lot of manual steps that reprepro-rebuilder takes care of for you. But the script’s command line interface is flexible enough for the cases where more intervention is required, too. For example, for my Emacs snapshot builds, I have another script to replace steps (1) and (2), which merges from a specific branch that I know has been manually tested, and generates a special version number. Then I say reprepro-rebuilder --release and the script takes care of preparing packages for unstable and bullseye-backports, and I can have my snapshots on all of my machines without a lot of work.

Posted Thu 08 Sep 2022 19:45:35 UTC

The ThinkPad x220 that I had been using as an ssh terminal at home finally developed one too many hardware problems a few weeks ago, and so I ordered a Raspberry Pi 4b to replace it. Debian builds minimal SD card images for these machines already, but I wanted to use the usual ext4-on-LVM-on-LUKS setup for GNU/Linux workstations. So I used Consfigurator to build a custom image.

There are two key advantages to using Consfigurator to do something like this:

  1. As shown below, it doesn’t take a lot of code to define the host, it’s easily customisable without writing shell scripts, and it’s all declarative. (It’s quite a bit less code than Debian’s image-building scripts, though I haven’t carefully compared, and they are doing some additional setup beyond what’s shown below.)

  2. You can do nested block devices, as required for ext4-on-LVM-on-LUKS, without writing an intensely complex shell script to expand the root filesystem to fill the whole SD card on first boot. This is because Consfigurator can just as easily partition and install an actual SD card as it can write out a disk image, using the same host definition.

Consfigurator already had all the capabilities to do this, but as part of this project I did have to come up with the high-level wrapping API, which didn’t exist yet. My first SD card write wouldn’t boot because I had to learn more about kernel command lines; the second wouldn’t boot because of a minor bug in Consfigurator regarding /etc/crypttab; and the third build is the one I’m using, except that the first boot runs into a bug in cryptsetup-initramfs. So as far as Consfigurator is concerned I would like to claim that it worked on my second attempt, and had I not been using LUKS it would have worked on the first :)

The code

(defhost erebus.silentflame.com ()
  "Low powered home workstation in Tucson."
  (os:debian-stable "bullseye" :arm64)
  (timezone:configured "America/Phoenix")

  (user:has-account "spwhitton")
  (user:has-enabled-password "spwhitton")

  (disk:has-volumes
   (physical-disk
    (partitioned-volume

     ((partition
       :partition-typecode #x0700 :partition-bootable t :volume-size 512
       (fat32-filesystem :mount-point #P"/boot/firmware/"))

      (partition
       :volume-size :remaining

       (luks-container
        :volume-label "erebus_crypt"
        :cryptsetup-options '("--cipher" "xchacha20,aes-adiantum-plain64")

        (lvm-physical-volume :volume-group "vg_erebus"))))))

   (lvm-logical-volume
    :volume-group "vg_erebus"
    :volume-label "lv_erebus_root" :volume-size :remaining

    (ext4-filesystem :volume-label "erebus_root" :mount-point #P"/"
                     :mount-options '("noatime" "commit=120"))))

  (apt:installed "linux-image-arm64" "initramfs-tools"
                 "raspi-firmware" "firmware-brcm80211"
                 "cryptsetup" "cryptsetup-initramfs" "lvm2")
  (etc-default:contains "raspi-firmware"
                        "ROOTPART" "/dev/mapper/vg_erebus-lv_erebus_root"
                        "CONSOLES" "ttyS1,115200 tty0"))

and then you just insert the SD card and, at the REPL on your laptop,

CONSFIG> (hostdeploy-these laptop.example.com
           (disk:first-disk-installed-for nil erebus.silentflame.com #P"/dev/mmcblk0"))

There is more general information in the OS installation tutorial in the Consfigurator user’s manual.

Other niceties

  • Configuration management that’s just as easily applicable to OS installation as it is to the more usual configuration of hosts over SSH drastically improves the ratio of cost-to-benefit for including small customisations one is used to.

    For example, my standard Debian system configuration properties (omitted from the code above) meant that when I was dropped into an initramfs shell during my attempts to make an image that could boot itself, I found myself availed of my custom Space Cadet-inspired keyboard layout, without really having thought at any point “let’s do something to ensure I can have my usual layout while I’m figuring this out.” It was just included along with everything else.

  • As compared with the ThinkPad x220, it’s nice how the Raspberry Pi 4b is silent and doesn’t have any LEDs lit by default once it’s booted. A quirk of my room is that one plug socket is controlled by a switch right next to the switch for the ceiling light, so I’ve plugged my monitor into that outlet. Then when I’ve finished using the new machine I can flick that switch and the desk becomes completely silent and dark, without actually having to suspend the machine to RAM, thereby stopping cron jobs, preventing remote access from the office to fetch uncommitted files, etc..

Posted Wed 03 Aug 2022 00:04:30 UTC

I’d like to share some pointers for using Gnus together with notmuch rather than notmuch together with notmuch’s own Emacs interface, notmuch.el. I set about this because I recently realised that I had been poorly reimplementing lots of Gnus features in my init.el, primarily around killing threads and catching up groups, supported by a number of complex shell scripts. I’ve now switched over, and I’ve been able to somewhat simplify what’s in my init.el, and drastically simplify my notmuch configuration outside of Emacs. I’m always more comfortable with less Unix and more Lisp when it’s feasible.

  • The basic settings are gnus-search-default-engines and gnus-search-notmuch-remove-prefix, explained in (info "(gnus) Searching"), and an entry for your maildir in gnus-secondary-select-methods, explained in (info "(gnus) Maildir"). Then you will have G G and G g in the group buffer to make and save notmuch searches.

  • I think it’s important to have something equivalent to notmuch-saved-searches configured programmatically in your init.el, rather than interactively adding each saved search to the group buffer. This is because, as notmuch users know, these saved searches are more like permanent, virtual inboxes than searches. You can learn how to do this by looking at how gnus-group-make-search-group calls gnus-group-make-group. I have some code running in gnus-started-hook which does something like this for each saved search:

      (if (gnus-group-entry group)
          (gnus-group-set-parameter group 'nnselect-specs ...)
        (gnus-group-make-group ...))
    

    The idea is that if you update your saved search in your init.el, rerunning this code will update the entries in the group buffer. An alternative would be to just kill every nnselect search in the group buffer each time, and then recreate them. In addition to reading gnus-group-make-search-group, you can look in ~/.newsrc.eld to see the sort of nnselect-specs group parameters you’ll need your code to produce.

    I’ve very complicated generation of my saved searches from some variables, but that’s something I had when I was using notmuch.el, too, so perhaps I’ll describe some of the ideas in there in another post.

  • You’ll likely want to globally bind a function which starts up Gnus if it’s not already running and then executes an arbitrary notmuch search. For that you’ll want (unless (gnus-alive-p) (gnus)), and not (unless (gnus-alive-p) (gnus-no-server)). This is because you need Gnus to initialise nnmaildir before doing any notmuch searches. Gnus passes --output=files to notmuch and constructs a summary buffer of results by selecting mail that it already knows about with those filenames.

  • When you’re programmatically generating the list of groups, you might also want to programmatically generate a topics topology. This is how you do that:

      (with-current-buffer gnus-group-buffer
        (gnus-topic-mode 0)
        (setq gnus-topic-alist nil gnus-topic topology nil)
        ;; Now push to those two variables.  You can also use
        ;; `gnus-topic-move-matching' to move nnmaildir groups into, e.g.,
        ;; "misc".
        (gnus-topic-mode 1)
        (gnus-group-list-groups))
    

    If you do this in gnus-started-hook, the values for those variables Gnus saves into ~/.newsrc.eld are completely irrelevant and do not need backing up/syncing.

  • When you want to use M-g to scan for new mail in a saved search, you’ll need to have Gnus also rescan your nnmaildir inbox, else it won’t know about the filenames returned by notmuch and the messages won’t appear. This is similar to the gnus vs. gnus-no-server issue above. I’m using :before advice to gnus-request-group-scan to scan my nnmaildir inbox each time any nnselect group is to be scanned.

  • If you are used to linking to mail from Org-mode buffers, the existing support for creating links works fine, and the standard gnus: links already contain the Message-ID. But you’ll probably want opening the link to perform a notmuch search for id:foo rather than trying to use Gnus’s own jump-to-Message-ID code. You can do this using :around or :override advice for org-gnus-follow-link: look at gnus-group-read-ephemeral-search-group to do the search, and then call gnus-summary-goto-article.

I don’t think that the above is especially hacky, and don’t expect changes to Gnus to break any of it. Implementing the above for your own notmuch setup should get you something close enough to notmuch.el that you can take advantage of Gnus’ unique features without giving up too much of notmuch’s special features. However, it’s quite a bit of work, and you need to be good at Emacs Lisp. I’d suggest reading lots of the Gnus manual and determining for sure that you’ll benefit from what it can do before considering switching away from notmuch.el.

Reading through the Gnus manual, it’s been amazing to observe the extent to which I’d been trying to recreate Gnus in my init.el, quite oblivious that everything was already implemented for me so close to hand. Moreover, I used Gnus ten years ago when I was new to Emacs, so I should have known! I think that back then I didn’t really understand the idea that Gnus for mail is about reading mail like news, and so I didn’t use any of the features, back then, that more recently I’ve been unknowingly reimplementing.

Posted Mon 11 Jul 2022 17:03:23 UTC Tags:

I recently released Consfigurator 1.0.0 and I’m now returning to my Common Lisp reading. Building Consfigurator involved the ad hoc development of a cross between a Haskell-style functional DSL and a Lisp-style macro DSL. I am hoping that it will be easier to retain lessons about building these DSLs more systematically, and making better use of macros, by finishing my studying of macrology books and papers only after having completed the ad hoc DSL. Here’s my current list:

  • Finishing off On Lisp and Let Over Lambda.

  • Richard C. Waters. 1993. “Macroexpand-All: an example of a simple lisp code walker.” In Newsletter ACM SIGPLAN Lisp Pointers 6 (1).

  • Naive vs. proper code-walking.

  • Michael Raskin. 2017. “Writing a best-effort portable code walker in Common Lisp.” In Proceedings of 10th European Lisp Symposium (ELS2017).

  • Cullpepper et. al. 2019. “From Macros to DSLs: The Evolution of Racket”. Summet of Advances in Programming Languages.

One thing that I would like to understand better is the place of code walking in macro programming. The Raskin paper explains that it is not possible to write a fully correct code walker in ANSI CL. Consfigurator currently uses Raskin’s best-effort portable code walker. Common Lisp: The Language 2 includes a few additional functions which didn’t make it into the ANSI standard that would make it possible to write a fully correct code walker, and most implementations of CL provide them under one name or another. So one possibility is to write a code walker in terms of ANSI CL + those few additional functions, and then use a portability layer to get access to those functions on different implementations (e.g. trivial-cltl2).

However, both On Lisp and Let Over Lambda, the two most substantive texts on CL macrology, both explicitly put code walking out-of-scope. I am led to wonder: does the Zen of Common Lisp-style macrology involve doing without code walking? One key idea with macros is to productively blur the distinction between designing languages and writing code in those languages. If your macros require code walking, have you perhaps ended up too far to the side of designing whole languages? Should you perhaps rework things so as not to require the code walking? Then it would matter less that those parts of CLtL2 didn’t make it into ANSI. Graham notes in ch. 17 of On Lisp that read macros are technically more powerful than defmacro because they can do everything that defmacro can and more. But it would be a similar sort of mistake to conclude that Lisp is about read macros rather than defmacro.

There might be some connection between arguments for and against avoiding code walking in macro programming and the maintainance of homoiconicity. One extant CL code walker, hu.dwim.walker, works by converting back and forth between conses and CLOS objects (Raskin’s best-effort code walker has a more minimal interface), and hygienic macro systems in Scheme similarly trade away homoiconicity for additional metadata (one Lisp programmer I know says this is an important sense in which Scheme could be considered not a Lisp). Perhaps arguments against involving much code walking in macro programming are equivalent to arguments against Racket’s idea of language-oriented programming. When Racket’s designers say that Racket’s macro system is “more powerful” than CL’s, they would be right in the sense that the system can do all that defmacro can do and more, but wrong if indeed the activity of macro programming is more powerful when kept further away from language design. Anyway, these are some hypotheses I am hoping to develop some more concrete ideas about in my reading.

Posted Sun 08 May 2022 20:41:17 UTC

Consfigurator has long has combinators OS:TYPECASE and OS:ETYPECASE to conditionalise on a host’s operating system. For example:

(os:etypecase
  (debian-stable (apt:installed-backport "notmuch"))
  (debian-unstable (apt:installed "notmuch")

You can’t distinguish between stable releases of Debian like this, however, because while that information is known, it’s not represented at the level of types. You can manually conditionalise on Debian suite using something like this:

(defpropspec notmuch-installed :posix ()
  (switch ((os:debian-suite (get-hostattrs-car :os)) :test #'string=)
    ("bullseye" '(apt:installed-backport "notmuch"))
    (t          '(apt:installed "notmuch"))))

but that means stepping outside of Consfigurator’s DSL, which has various disadvantages, such as a reduction in readability. So today I’ve added some new combinators, so that you can say

(os:debian-suite-case
  ("bullseye" (apt:installed-backport "notmuch"))
  (t          (apt:installed "notmuch")))

For my own use I came up with this additional simple wrapper:

(defmacro for-bullseye (atomic)
  `(os:debian-suite-case
     ("buster")
     ("bullseye" ,atomic)
     ;; Check the property is actually unapplicable.
     ,@(and (get (car atomic) 'punapply) `((t (unapplied ,atomic))))))

So now I can say

(for-bullseye (apt:pinned '("elpa-org-roam") '(os:debian-unstable) 900))

which is a succinct expression of the following: “on bullseye, pin elpa-org-roam to sid with priority 900, drop the pin when we upgrade the machine to bookworm, and don’t do anything at all if the machine is still on buster”.

As a consequence of my doing Debian development but running Debian stable everywhere, I accumulate a number of tweaks like this one over the course of each Debian stable release. In the past I’ve gone through and deleted them all when it’s time to upgrade to the next release, but then I’ve had to add properties to undo changes made for the last stable release, and write comments saying why those are there and when they can be safely removed, which is tedious and verbose. This new combinator is cleaner.

Posted Tue 03 May 2022 23:13:37 UTC Tags:

I am pleased to announce Consfigurator 1.0.0.

Reaching version 1.0.0 signifies that we will try to avoid API breaks. You should be able to use Consfigurator to manage production systems.

You can find the source at https://git.spwhitton.name/consfigurator for browsing online or git cloning.

Releases are made by publishing signed git tags to that repository. The tag for this release is named ‘v1.0.0’, and is signed by me.

On Debian/etc. systems, apt-get install cl-consfigurator

-8<-

Consfigurator is a system for declarative configuration management using Common Lisp. You can use it to configure hosts as root, deploy services as unprivileged users, build and deploy containers, install operating systems, produce disc images, and more. Some key advantages:

  • Apply configuration by transparently starting up another Lisp image on the machine to be configured, so that you can use the full power of Common Lisp to inspect and control the host.

  • Also define properties of hosts in a more restricted language, that of :POSIX properties, to configure machines, containers and user accounts where you can’t install Lisp. These properties can be applied using just an SSH or serial connection, but they can also be applied by remote Lisp images, enabling code reuse.

  • Flexibly chain and nest methods of connecting to hosts. For example, you could have Consfigurator SSH to a host, sudo to root, start up Lisp, use the setns(2) system call to enter a Linux container, and then deploy a service. Secrets, and other prerequisite data, are properly passed along.

  • Combine declarative semantics for defining hosts and services with a multiparadigmatic general-purpose programming language that won’t get in your way.

Declarative configuration management systems like Consfigurator and Propellor share a number of goals with projects like the GNU Guix System and NixOS. However, tools like Consfigurator and Propellor try to layer the power of declarative and reproducible configuration semantics on top of traditional, battle-tested UNIX system administration infrastructure like distro package managers, package archives and daemon configuration mechanisms, rather than seeking to replace any of those. Let’s get as much as we can out of all that existing distro policy-compliant work!

Posted Sat 30 Apr 2022 19:50:47 UTC

I like to look at the Emacs subreddit and something I’ve noticed recently is people asking “should I start by writing my own Emacs config, or should I use this or that prepackaged one?” There is also this new config generator published by Philip Kaludercic. I find implicit in these the idea that one’s init.el is a singular product. To start using Emacs, newcomers seem to think, you need to couple it with a completed init.el, and so there is the question of writing your own or using one someone else has written. I think that an appropriate analogy is certain shell scripts. If you want to burn backups to DVDs you might download someone’s DVD burning shell script which tries to make that easy, or you might write your own. In both cases, you are likely to want to tweak the script after you’ve started using it, but there is nevertheless a discrete point at which you go from having part of a script and not being able to burn DVDs, to having a completed script and now being able to burn DVDs. Similarly, the idea that you can’t start using Emacs until you couple it with an init.el is like thinking that there is a process of producing or downloading an init.el, and only after that can you begin using Emacs.

This thinking makes sense if you’re developing one of the large Emacs configuration frameworks like Spacemacs or Doom Emacs. The people behind those projects are seeking to build something quite different from Emacs, using Emacs as a base, and for many people using that new, quite different thing is preferable to using Emacs. Then indeed, until you’ve finished developing your configuration framework’s init.el to a degree that you’re ready to release version 0.1 of your framework, you haven’t got something that’s ready to use. Like the shell script, there’s a discrete point after which you have a product, and there’s lots of labour that must precede it. (I think it’s pretty cool that Emacs is flexible enough to be something on its own and also a base for these projects.)

However, this temporal structure does not make sense to me when it comes to using just Emacs. I find the idea that one’s init.el is a singular product strange. In particular, my init.el is not something which transforms Emacs into something else, like the init.el that’s part of Doom Emacs does. It’s just a long collection of incrementally developed, largely unrelated hacks and tweaks. You could insist that it transforms default Emacs into Sean’s Emacs, but I think that misleadingly implies that there’s an overarching design and cohesion to my init.el, which just isn’t there – and it would be weird if it was there, because then I would be doing something more like the developers behind Doom Emacs. So if you’re not going to use one of the large configuration frameworks, then there is no task of “writing your own init.el” that stands before you. You just start using Emacs, and as part of that you’re going to write functions and rebind keys, etc., and your init.el is the file in which those changes are collected. The choice is not between writing your own init.el or downloading a prepackaged one. It’s between using Emacs, or using another product that has been built out of Emacs. The latter necessarily involves a completed init.el, but that’s an implementation detail.

I am very happy Kaludercic’s configuration generator has been made available, but I would be inclined to rename it. A new user of Emacs is likely to be overwhelmed with unintuitive defaults that have stuck around for mostly historical reasons. There are a lot of them, so it is a lot to ask of new users that they just identify the defaults that don’t suit them and add lines to their init.el to change those. When too many things are unintuitive, it’s hard to know where to start. Kaludercic’s configuration generator is a way to walk newcomers through the most significant defaults in a way that something structured like a reference manual would struggle to do. The result is some Lisp code, but I would prefer not to refer to that result as an Emacs configuration. It’s a series of configuration snippets that you can add to your Emacs configuration to help deal with the newcomer’s problem of too many unintuitive defaults.

I’m not sure it’s important to actually rename Kaludercic’s tool to something which says it’s a generator of configuration snippets rather than a generator of configurations. But I would like to challenge the idea that to start using Emacs you first need to couple it with a completed init.el. If you’re going to use Emacs, rather than Spacemacs or Doom Emacs, you can just start using it. If you find yourself butting up against a lot of unintuitive defaults, then you can use a walkthrough tool like Kaludercic’s to figure out what you need to add to your init.el to deal with those. But that is better understood as just more of the tweaking and customisation that Emacs users are always getting up to, not some prerequisite labour.

Posted Tue 26 Apr 2022 06:01:25 UTC Tags:

Here are a few new features I’ve added to GNU ELPA and upstream GNU Emacs recently. Text is adapted from the in-tree documentation I wrote for the new features. Thanks to everyone who offered feedback on my patches.

New feature to easily bypass Eshell’s own pipelining

Prefixing |, < or > with an asterisk, i.e. *|, *< or *>, will cause the whole command to be passed to the operating system shell. This is particularly useful to bypass Eshell’s own pipelining support for pipelines which will move a lot of data.

This has long been an obstacle when it comes to using Eshell as one’s main shell. The new syntax is easy to use and covers a lot of different use cases.

New Eshell module to help supplying absolute file names to remote commands

After enabling the new eshell-elecslash module, typing a forward slash as the first character of a command line argument will automatically insert the Tramp prefix. The automatic insertion applies only when default-directory is remote and the command is a Lisp function. This frees you from having to keep track of whether commands are Lisp function or external when supplying absolute file name arguments.

This is another attempt to solve an Eshell papercut. Suppose you execute

 cd /ssh:root@@example.com:
 find /etc -name "*gnu*"

and in reviewing the output of the command, you identify a file /etc/gnugnu that should be moved somewhere else. So you type

mv /etc/gnugnu /tmp

But since mv refers to the local Lisp function eshell/mv, not a remote shell command (unlike find(1)), to say this is to request that the local file /etc/gnugnu be moved into the local /tmp directory. After you enable eshell-elecslash, to then when you type the above mv invocation you will get the following input, which is what you intended:

mv /ssh:root@example.com:/etc/gnugnu /ssh:root@example.com:/tmp

imenu is now bound to M-g i globally

This is a useful command but everyone has to come up with their own binding for it. No longer.

New macro-writing macros, cl-with-gensyms and cl-once-only

These two macros are quite interesting. In the history of Common Lisp-style macros, these are the only two macro-writing macros that have emerged as essential tools for intermediate and advanced macrology. Most any other macro-writing macros are either project- or programmer-specific. In his book on Lisp macros Doug Hoyte proposes an alternative to defmacro, defmacro!, which is just the same as defmacro except that it builds in facilities equivalent to cl-with-gensyms and cl-once-only. I’ve long wanted to have these macros available in core Emacs Lisp, too, and now they are.

New package on GNU ELPA: transient-cycles

Many commands can be conceptualised as selecting an item from an ordered list or ring. Sometimes after running such a command, you find that the item selected is not the one you would have preferred, but the preferred item is nearby in the list. If the command has been augmented with transient cycling, then it finishes by setting a transient map with keys to move backwards and forwards in the list of items, so you can select a nearby item instead of the one the command selected. From the point of view of commands subsequent to the deactivation of the transient map, it is as though the first command actually selected the nearby item, not the one it really selected.

Protesilaos Stavrou helped me test the package and has written up some usage notes.

This is an idea I came up with in 2020, and refined in my init.el since then. This year I made it into a package.

Posted Thu 21 Apr 2022 20:43:28 UTC Tags:

Debian contributors end up with a number of Debian installations on their workstations for various purposes; there are perhaps five or six on my laptop right now. In addition to this, I now have three copies of GNU Emacs, too.

/usr/bin/emacs

This is the version of Emacs in Debian stable. I have to keep it available for testing various of the Debian packages I maintain.

/opt/emacs-snapshot/bin/emacs

This is a build of Emacs 29.0.50 out of upstream git, with native Wayland support, and some custom Debian packaging to integrate it with Debian’s emacsen-common infrastructure: Emacs addon packages are pre-bytecompiled against this version of Emacs, and re-bytecompilation is fully automated. This is nice if you want to use development snapshots of Emacs but get your addon packages from Debian, as I do. These are, roughly, the extent of my changes on top of upstream:

diff --git a/.gitignore b/.gitignore
index 78557a5e87..9743ccca0c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,6 +23,11 @@
 # in Git 1.8.2 (March 2013).


+# Debian packaging
+config.guess
+config.sub
+debian/emacs-snapshot/
+
 # Personal customization.
 .dir-locals-2.el

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index de4cebccca..1596b5a1bf 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -311,6 +311,7 @@ package-directory-list
       (and (stringp f)
            (equal (file-name-nondirectory f) "site-lisp")
            (push (expand-file-name "elpa" f) result)))
+    (push "/usr/share/emacs-snapshot/site-lisp/elpa" result)
     (nreverse result))
   "List of additional directories containing Emacs Lisp packages.
 Each directory name should be absolute.
diff --git a/lisp/startup.el b/lisp/startup.el
index b79467339b..c54ab89b33 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1374,7 +1374,12 @@ command-line
     ;; be loaded from site-run-file and wants to test if -q was given
     ;; should check init-file-user instead, since that is already set.
     ;; See cus-edit.el for an example.
-    (if site-run-file
+    (when site-run-file
+        (let ((file "/etc/emacs/site-start.d/00debian.el"))
+          ;; When the Emacs build invokes Emacs, such as in the
+          ;; Makefile rule for ${unidir}/emoji-labels.el, 00debian.el
+          ;; might not exist.  Should be fine to just skip the load.
+          (when (file-readable-p file) (load file)))
         ;; Sites should not disable the startup screen.
         ;; Only individuals should disable the startup screen.
         (let ((inhibit-startup-screen inhibit-startup-screen))

If you’re using Wayland on Debian, or you just want or need a newer Emacs, you might like to install these snapshot builds yourself from http://silentflame.com/debian/, suites “bullseye-backports” and “unstable”.

~spwhitton/src/emacs/{lib-src/emacsclient,src/emacs}

The snapshot builds described in the previous section are what I use on machines other than my laptop. On my laptop, I have my Wayland compositor configured to use the emacs and emacsclient binaries in ~/src/emacs, if they exist. This is a build of a branch with work-in-progress patches I’m planning to send upstream, plus bug fixes, reversions of commits that introduced regressions which affect my setup, etc..

Why have this in addition to the emacs-snapshot.deb builds – if there are temporary fixup commits, why not apply them and build a .deb just for local use? Well, that is indeed what I was doing until recently, but if you’re developing patches for upstream Emacs it’s pretty inconvenient. Hyperlinks to function definitions in Help buffers take you somewhere read-only in /opt, for example, and rebuilds are slow because, by default, Debian package builds begin by cleaning the tree.

In addition to the diff above (minus the .gitignore change), and the temporary fixup commits, I also apply the following change on this branch. Having the above diff applied implies that a recent emacs-snapshot.deb is installed, to provide the byte-compiled addons. This informal dependency should work well enough so long as I update the snapshot .debs described in the previous section often enough.

diff --git a/lib-src/emacsclient.c b/lib-src/emacsclient.c
index 7769e015ed..3af4f7d0cf 100644
--- a/lib-src/emacsclient.c
+++ b/lib-src/emacsclient.c
@@ -1773,7 +1773,10 @@ start_daemon_and_retry_set_socket (void)
     }
   else
     {
-      char emacs[] = "emacs";
+      const char *devel_build_rel = "/src/emacs/src/emacs";
+      char *emacs = xmalloc (strlen (egetenv ("HOME"))
+                + strlen (devel_build_rel) + 1);
+      strcpy ( stpcpy (emacs, egetenv ("HOME")), devel_build_rel);
       char daemon_option[] = "--daemon";
       char *d_argv[3];
       d_argv[0] = emacs;
@@ -1790,7 +1793,7 @@ start_daemon_and_retry_set_socket (void)
      d_argv[1] = daemon_arg;
    }
 # endif
-      execvp ("emacs", d_argv);
+      execvp (emacs, d_argv);
       message (true, "%s: error starting emacs daemon\n", progname);
       exit (EXIT_FAILURE);
     }

And I use something like this to have my Wayland compositor use these builds:

#!/bin/sh

if [ -e "$HOME/src/emacs/admin/README.melete" \ # confirm we're on the branch
    -a -x "$HOME/src/emacs/lib-src/emacsclient" \
    -a -d "/usr/share/emacs-snapshot/site-lisp/elpa" ]; then
    EMACSCLIENT="$HOME/src/emacs/lib-src/emacsclient"
else
    EMACSCLIENT=$(command -v emacsclient)
fi

$EMACSCLIENT -a "" -nc

There’s something similar in my shell configuration.

Posted Sun 26 Dec 2021 21:09:48 UTC

For some months now I’ve been working on some patches to Consfigurator to add support for Linux containers. My goal is to make Consfigurator capable of both performing the initial setup of a container and of entering the running container to apply configuration. For the case of unprivileged LXCs running as non-root, my work-in-progress branch can now do both of these things. As Consfigurator enters the container directly using system calls, it should be decently fast at configuring multiple containers on a host, and it will also be possible to have it do this in parallel. The initial setup for the container uses Consfigurator’s existing support for building root filesystems, and it should be easy to extend that to support arbitrary GNU/Linux distributions by teaching Consfigurator how to invoke bootstrapping tools other than debootstrap(8).

Here’s an example:

(defhost lxc1.silentflame.com ()
  (os:debian-stable "bullseye" :amd64)
  (basic-props)
  (apt:installed "systemd" "netcat")
  (apache:https-vhost ...))

(defhost lxctest.laptop.silentflame.com ()
  (os:debian-stable "bullseye" :amd64)
  (apt:proxy "http://192.168.122.1:3142")
  (basic-props)
  (apt:installed "linux-image-amd64" "lxc")

  (lxc:usernet-usable-by "spwhitton" "lxcbr0")
  (lxc:user-containers-autostart "spwhitton")
  (lxc:user-container-for '(:additional-lines
                            ("lxc.net.0.type = veth"
                             "lxc.net.0.flags = up"
                             "lxc.net.0.link = lxcbr0"
                             ...))
                          "spwhitton"
                          lxc1.silentflame.com))

(defhost laptop.silentflame.com ()
  ...
  (libvirt:kvm-boots-chroot-for '(:always-deploys t)
                                lxctest.laptop.silentflame.com))

This code is a simplified definition of my testing setup for this work. It defines three hosts: a container lxc1, a container host lxctest, and my laptop. When Consfigurator is asked to deploy the laptop, it will set up the root filesystem for lxctest and then boot it as a KVM virtual machine. Preparing that root filesystem will include setting up the root filesystem for lxc1, too, including shifting the ownership and ACLs to match the user namespace LXC will use when booting the container. Thus, once the deployment of the laptop is finished, it will be possible to boot the lxctest VM, connect to it as the user spwhitton, and start lxc1.

Consfigurator includes only minimal support for setting up container networking, as there are so many different ways in which you might want to do it. In my own consfig I’ve been developing properties to connect containers directly to my tinc VPN. A single tinc daemon runs on the container host, and other tinc daemons route a whole subnet, containing the addresses for each of the containers, to the container host’s tinc daemon. As the LXCs Consfigurator sets up run as non-root, some sort of setuid facility is required to configure this networking. Consfigurator’s ability to dump executable Lisp images is helping here. I define a function which runs as root to set up the networking:

(defun route-athenet-container-veth (host)
  (let ((user (getenv "USERV_USER"))
        (peer (getenv "USERV_U_PEER"))
        (ip (car (uiop:command-line-arguments))))
    (unless (string-prefix-p (format nil "veth~D_" (getenv "USERV_UID")) peer)
      (error "~A does not belong to requester." peer))
    (unless (member (cons user ip) (get-hostattrs 'veth-ips host) :test #'equal)
      (error "~A does not have permission to route ~A." user ip))
    (flet ((r (&rest args)
             ;; Explicitly passing nil means UIOP will not invoke a shell.
             (run-program args :force-shell nil)))
      (eswitch ((getenv "USERV_U_HOOK_TYPE") :test #'string=)
        ("up"
         (apply #'r
                "sysctl" "-w"
                "net.ipv6.conf.all.forwarding=1"
                ...)
         (r "ip" "addr" "flush" "dev" peer "scope" "link")
         (r "ip" "-6" "addr" "add" "fe80::1/64" "dev" peer)
         (r "ip" "-6" "route" "add" (strcat ip "/128") "dev" peer)
         ...)
        ("down"
         ...
         (r "ip" "-6" "route" "del" (strcat ip "/128") "dev" peer))))))

and then apply the following property to lxctest to dump an image which will call this function and then exit:

(image-dumped
 "/usr/lib/userv/route-athenet-container-veth"
 `(route-athenet-container-veth ,(intern (string-upcase (get-hostname)))))

I’m using GNU userv to enable ordinary users to run this image as root, so there there’s a small script which converts LXC’s LXC_HOOK_* environment variables into appropriate command line arguments to userv(1) such that the function above is able to access that information from its environment (the USERV_U_* variables above). You could just as easily do this with sudo, by giving permission for the relevant LXC_HOOK_* environment variables to survive the switch to root.

What’s particularly nice about this is that there’s no need to write any code to keep a config file updated, specifying which users are allowed to route which IPs to their containers. ROUTE-ATHENET-CONTAINER-VETH receives a HOST value for the container host and can just look at the metadata established by properties for particular containers. Each time this metadata is updated and lxctest is deployed, a fresh image is dumped containing the updated metadata.

This work has provided opportunities to make various other improvements to Consfigurator, especially with regard to dumping and reinvoking images. Making SBCL capable of entering user namespaces required a change upstream, which made it into the recent SBCL 2.1.8 release. I’m very grateful to the SBCL developers for their engagement with my project. I’ve been able to add a workaround so that Consfigurator can still enter user namespaces when run on the version of SBCL included in Debian stable. I also discovered that deploying all of my laptop, lxctest and lxc1 at once generates enough output to fill up a pipe, thus revealing a deadlock in Consfigurator’s IPC, which it was good to become aware of and fix. That involved writing my first multi-threaded Lisp, as there are two pipes that need to be kept from filling up, and to my surprise it worked first time. Take that Haskell :)

Posted Sun 24 Oct 2021 16:17:51 UTC