Try it out / quick start
Install Consfigurator (Installation).
Create a new directory
consfigsomewhere where ASDF will pick it up, such as
Define a Lisp system which represents your configuration.
(asdf:defsystem :com.example.consfig :serial t :depends-on (#:consfigurator #:cl-interpol) :components ((:file "package") (:file "consfig")))
(in-package :cl-user) ;; this macro is a simple wrapper of DEFPACKAGE which sets up local ;; nicknames for packages providing properties and data sources (consfigurator:defpackage-consfig :com.example.consfig (:use #:cl #:alexandria #:consfigurator))
If you use Emacs and you have
consfigurator.elinstalled, it will help to activate our major mode in ~/common-lisp/consfig/.dir-locals.el:
((auto-mode-alist . (("\\.lisp\\'" . consfigurator-lisp-mode))))
Define some hosts and deployments.
(in-package :com.example.consfig) (in-consfig "com.example.consfig") (named-readtables:in-readtable :consfigurator) (defhost athena.example.com (:deploy ((:ssh :user "root") :sbcl)) "Web and file server." (os:debian-stable "buster" :amd64) (apt:mirrors "http://my.local.mirror.example.com/") (apt:uses-local-cacher) ; sets up apt-cacher-ng (apt:standard-sources.list) ;; Set key--value pairs in INI-style files. Consfigurator will ;; uncomment existing lines in preference to adding new ones, for ;; readability. (file:contains-ini-settings "/etc/systemd/logind.conf" '("Login" "KillUserProcesses" "no")) (apt:service-installed-running "apache2") ;; Apply some properties as a non-root user. ;; (as "spwhitton" ;; (gnupg:public-key-imported "8DC2 487E 51AB DD90 B5C4 753F 0F56 D055 3B6D 411B")) (file:has-content "/etc/foo" "Here is my file content. It's multiline. CL-INTERPOL and CL-HEREDOC are also available; the latter is particularly useful for shell scripts.") (file:has-content "/etc/bar" '("or" "specify" "a" "list" "of" "lines")) (file:contains-lines "/etc/some.conf" "FOO=bar") ; preserve rest of file contents ;; This will call debootstrap(1) in a way which respects the apt ;; cacher and mirror configured above, so setting up multiple ;; chroots with the same OS will be fast. (chroot:os-bootstrapped. nil "/srv/chroot/test" (os:debian-unstable :amd64) (apt:standard-sources.list) ;; These two properties are not for debootstrap(1) for but apt ;; inside the chroot. (apt:uses-parent-proxy) ; use the apt-cacher-ng set up outside chroot (apt:uses-parent-mirrors))) ; use the apt mirror set up above
This assumes that you have your SSH agent etc. set up such that you can ssh to root on athena without having to type a password into ssh’s stdin. You should also enable SSH connection sharing, including
Get a Lisp REPL started up –
M-x slimein Emacs or
sbclat a shell prompt. Evaluate
(asdf:load-system "com.example.consfig"), then
C-c ~in Emacs).
You should now be able to evaluate
(athena.example.com)to deploy properties to athena, using the connection chain of SSH and then handing over to a remote Lisp image.
Other things to try
Note that some of these violate some of the ideas of declarative configuration management, because they apply individual properties without updating the definitions of hosts. Sometimes that’s the right thing to do, though, and Consfigurator makes it easy to reuse your property definitions in these non-declarative ways.
Try deploying properties to athena using a different connection type
Evaluate something like:
(deploy :ssh athena.example.com)
Apply a security update to all your systems
It’s useful to be able to quickly apply a security update across multiple
machines without otherwise interacting with their configuration. Supposing
you have defined a variable
*ALL-MY-SERVERS* which is a list hosts defined
DEFHOST, you can evaluate:
(dolist (server *all-my-servers*) (deploy-these :ssh server (cmd:single "apt-get update && apt-get upgrade openssl")))
Regex replace a file across hosts
*ALL-MY-SERVERS* as in the previous example,
(dolist (server *all-my-servers*) (deploy-these :ssh server (file:regex-replace-lines "/etc/baz" #?/^foo/ "bar")))
(relies on CL-INTERPOL syntax being enabled, as it is in the example consfig above)
Concepts and terminology
We make some simplifications. More precise definitions appear later in this manual.
A machine, container, chroot, or similar. Has a plist of static informational host attributes (“hostattrs”), usually including at least a hostname, and a property application specification defining the properties it has.
Some configuration which a host can have or lack, and which can be added to a host by running some code, possibly just by applying a series of other properties.
For example: the presence of some lines in a config file; a package being installed or absent; the availability of a website.
A means by which properties can be applied to hosts, and multihop connections to other hosts can be established. There are two types of connections: those which interact with the remote host by means of a POSIX shell, and those which apply properties by executing them in a Lisp image running on the host.
POSIX connections can pass input to and return output from processes, but cannot start asynchronous processes for interaction with your Lisp functions. This is so that POSIX connections can be defined to control hosts for which any kind of shell multiplexing is hard or impossible, such as with serial connections providing only a single interactive POSIX sh. For asynchronous interaction, use a Lisp connection.
The combination of a connection and a host. Executing a connection deploys
all of a host’s usual properties to that host by means of the given
connection. To deploy just a few particular properties, you can use
A deployment is itself a property. This is one way in which connections can be nested: one remote host can be used to deploy others, as a controller.
The Lisp image you control directly when you execute deployments. Typically
running on your development laptop/workstation (and not as the
Property application specification (“propspec”)
A specification, in Consfigurator’s DSL, for the properties that a host has and/or lacks. For example,:
(eseqprops (apt:installed postfix) (etc-default:contains "locale" "LANG" "en_GB.UTF-8") (unapplied (com.example.consfig.services:mail-satellite)))
Property application specifications are applied in order, so properties later in the list usually implicitly depend on properties earlier in the list, though some property combinators can change this.
Unevaluated property application specification (“unevaluated propspec”)
A property application specification, except in atomic property applications
of the form
(PROPERTY . ARGS),
ARGS are expressions to be evaluated to
produce the arguments to pass to
PROPERTY, rather than those arguments
themselves. An unevaluated property application specification can be
converted into a property application specification by evaluating each of
The main places you will find an unevaluated property application
specification is in calls to
DEFPROPLIST. Theses macros converts an unevaluated property application
specification into code which will produce the corresponding property
File contents required to apply a property which should be generated or
extracted, by the root Lisp, at the time of deployment: a tarball containing
the latest version of the web service to be deployed; a secret extracted from
an encrypted store; a git bundle from localhost which the target host cannot
git clone to itself.
Prerequisite data is versioned. To replace a secret key, for example, you change the data and bump the version. If there is no version bump, Consfigurator will assume connections can re-use old copies of prerequisite data; this avoids uploading the same data over and over again.
In addition to secrets management, prerequisite data is Consfigurator’s
mechanism for the common need to upload files to controlled hosts. The same
mechanism is used internally to upload the Lisp code needed to start up remote
Lisp images for
An ASDF system in which you define your hosts and initialise sources of
prerequisite data. This system might also define some site-specific
properties, default deployments, and helper functions. Typically the system
example.com is your primary domain
The system can contain multiple packages, perhaps to divide up your
definitions of hosts and default deployments from your site-specific
properties (e.g. you might have a package called
You can have multiple independent Consfigs loaded into the root Lisp at once,
but if you do, then you should avoid using the
*CONSFIG* global variable.
All unqualified names of Lisp symbols refer to those exported from the
CONSFIGURATOR package, because it is assumed that this package is imported
unqualified into both user consfigs and Lisp packages providing properties,
connection types and sources of prerequisite data.
FOO.BAR:BAZ means a symbol
BAZ defined in
CONSFIGURATOR.PROPERTY.FOO.BAR, except that
DATA.FOO:BAR means a
BAR defined in
CONSFIGURATOR.PROPERTY.DATA.FOO. These are the
recommended package nicknaming schemes for use in consfigs, e.g.:
(defpackage :com.example.consfig (:use #:cl #:consfigurator) (:local-nicknames (#:file #:consfigurator.property.file) (#:cmd #:consfigurator.property.cmd) (#:data.pgp #:consfigurator.data.pgp)))
You can use the
DEFPACKAGE-CONSFIG macro to set up all these local
Portability and stability
The core library should be portable between standards-conforming implementations of ANSI Common Lisp which include support for a few additional, widely-implemented features such as package-local nicknames. Optional packages providing properties and connection types might use implementation-specific functionality. Little to no testing is done by the author on implementations other than SBCL, so testing and portability patches are welcome.
Lisp implementations which will run on the hosts you wish to configure must support multithreading and must expose some mechanism for safely calling fork(2) in the presence of non-user threads, like
SB-POSIX:FORKin the case of SBCL. The root Lisp does not need to fork(2). With some additional portability patches, it should be possible to host the root Lisp even on systems to which Consfigurator probably can’t apply properties, such as Microsoft Windows.
As both Consfigurator and its dependency Osicat make use of CFFI-Grovel, loading Consfigurator into Lisp currently always additionally requires a C toolchain, and development headers for libacl. On GNU/Linux, development headers for libcap are also required. It might be possible to conditionalise further so as to avoid any dependency on a C toolchain for the root Lisp.
Little attempt is made by the author to support systems other than Debian GNU/Linux, but again, portability patches are welcome, and the design of Consfigurator should enable supporting other systems.
Many of the good ideas here come straight from Joey Hess’s Propellor. I’m working on Consfigurator because I think Propellor is great, but wanted to add Consfigurator’s POSIX-type connections and arbitrary connection nesting, and I wanted to implement that in Lisp (Propellor only supports something equivalent to a single, unnested Lisp-type connection). Additionally, after five years of using and extending Propellor, I’ve come to disagree with Joey about whether Haskell’s type system helps or hinders using and extending Propellor.