Prerequisite data

Naming

An item of prerequisite data is identified by two strings, called IDEN1 and IDEN2; together these are the prerequisite data identifiers for an item of prerequisite data. Typically IDEN1 specifies the context in which the data is relevant, and IDEN2 identifies the data within its context. IDEN2 is very often the filename in which the prerequisite data will eventually be stored. It might also be a human-readable string describing the purpose of the data. The following are the valid forms of IDEN1, and their meanings.

• (HOSTNAME . PATH) means the data that should be uploaded to PATH on HOSTNAME (and usually nowhere else, except in the case of, e.g., a public key). PATH must be absolute, not relative.

• ("--lisp-system" . SYSTEM) means the data is Lisp code which, when loaded, defines the packages and symbols contained in the ASDF system SYSTEM

• ("--user-passwd--HOSTNAME" . USER) means the data is the password for user USER on HOSTNAME

• ("--git-snapshot" . NAME) means the data is a snapshot of a git repo identified by NAME; see DATA.GIT-SNAPSHOT

• ("--pgp-pubkey" . FINGERPRINT) means the/a OpenPGP public key with fingerprint FINGERPRINT, ASCII-armoured

• ("--pgp-seckey" . FINGERPRINT) means the/a OpenPGP secret key with fingerprint FINGERPRINT, ASCII-armoured

• ("--luks-passphrase" . VOLUME-LABEL) means a LUKS passphrase for volume with label VOLUME-LABEL.

• Any other IDEN1 beginning with exactly two hyphens is reserved for future use.

• (_CONTEXT . ITEM) is an arbitrary prerequisite data context named CONTEXT; typically CONTEXT will be a network or grouping name, rather than referring to a single host. ITEM might be a path or some other identifier. Reserved for consfigs; will not be used by property definitions included with Consfigurator, and should not be used by third party extensions.

• (---CONTEXT . ITEM) is, similarly, an arbitrary prerequisite data context named CONTEXT. This form is intended for contexts similar to the reserved names beginning with two hyphens: types of information rather than site-local network or grouping names. This form will not be used by property definitions included with Consfigurator, but may be used by both consfigs and third party extensions.

Any other forms are invalid. In particular, an IDEN1 that is not a valid hostname and does not begin with a hyphen or an underscore must not be used.

Mechanics

Properties declare that they need certain pieces of prerequisite data by adding static informational attributes, and a deployment of those properties will make an attempt to provide the data. Properties then either call the GET-DATA-STREAM function or the GET-DATA-STRING function, or depend on the DATA-UPLOADED property, to get access to the requested data.

A Lisp connection gathers all needed prerequisite data once at the beginning, and copies it to an on-disk cache inside the home directory of the remote UID which will run the Lisp image. A POSIX connection only attempts to obtain prerequisite data when a property’s check function indicates the property is not already applied.

Sources of prerequisite data

Sources of prerequisite data register two functions. The second returns either a string of the prerequisite data itself, or a path to a file containing the data. The first returns the latest version number of the data that source is able to provide – i.e., the version number of the data that the second function would return if called.

Consfigurator will call the first function to find out if it needs to call the first rather than just using its caches. The first function should return nil if it can’t obtain the prerequisite data on this host, perhaps because it can’t decrypt the store. If a prerequisite data source wants to effectively bypass caching and provide fresh data every time Consfigurator deploys the host, it can use GET-UNIVERSAL-TIME as its first function.

Versions are compared using UIOP:VERSION< and UIOP:VERSION<=.

Security issues

Nothing is done to prevent prerequisite data being swapped out, so ensure your swap is encrypted.

Certain connection types require storing unencrypted copies of prerequisite data under ~/.cache/consfigurator/data. Consfigurator only stores data there when it has to, only the subset of the data that has to be uploaded for the requested deployment to be successful, and never in the root Lisp.

API reference

Prerequisite data

Class: DATA

An item of prerequisite data as provided by a registered prerequisite data source, or, outside of the root Lisp, as fished out of a local cache of prerequisite data.

Class: STRING-DATA

An item of prerequisite data directly accessible to Lisp.

Class: FILE-DATA

An item of prerequisite data accessible via the filesystem.

Generic function: REGISTER-DATA-SOURCE

(register-data-source type &key)

Initialise and register a source of prerequisite data in this Lisp image. Registered data sources are available to all deployments executed from the root Lisp, regardless of the consfig which defines the host to which properties are to be applied. (This could only cause problems if you have different consfigs with prerequisite data which is identified by the same two strings, in which case you will need to wrap your deployments with registering and unregistering data sources. Usually items of prerequisite data are identified using things like hostnames, so this shouldn’t be necessary.)

Implementations of this function return a pair of functions.

Signals a condition MISSING-DATA-SOURCE when unable to access the data source (e.g. because can’t decrypt it). This condition is captured and ignored in all new Lisp images started up by Consfigurator, since prerequisite data sources are not expected to be available outside of the root Lisp.

Simple error: MISSING-DATA-SOURCE

(missing-data-source)

Function: TRY-REGISTER-DATA-SOURCE

(try-register-data-source &rest args)

Register sources of prerequisite data. This function is typically called in consfigs. Any relative pathnames in ARGS will be resolved as paths under the home directory of the user Lisp is running as, before being passed to implementations of REGISTER-DATA-SOURCE.

Function: RESET-DATA-SOURCES

(reset-data-sources)

Forget all data sources registered in this Lisp image and items of string data obtained from data sources by this Lisp image. This function is typically called at the REPL.

Macro: WITH-RESET-DATA-SOURCES

(with-reset-data-sources &body body)

Run BODY with initially empty data sources and string data.

This macro is typically used for testing or debugging.

Function: GET-DATA-STRING

(get-data-string iden1 iden2)

Return the content of an item of prerequisite data as a string.

This function is called by property :APPLY and :UNAPPLY subroutines.

Function: GET-DATA-STREAM

(get-data-stream iden1 iden2)

Return a stream which will produce the content of an item of prerequisite data. The elements of the stream are always octets. If the item of prerequisite data was provided by the prerequisite data source as a string, it will be encoded in UTF-8.

This function is called by property :APPLY and :UNAPPLY subroutines.

Macro: WITH-DATA-STREAM

(with-data-stream s &body body)

Condition class: MISSING-DATA

Function: DATA-SOURCE-PROVIDING-P

(data-source-providing-p iden1 iden2)

Is there a data source which can provide the item of prerequisite data identified by IDEN1 and IDEN2?

This function is for implementation of REGISTER-DATA-SOURCE to check for clashes. It should not be called by properties.

Function: MAYBE-WRITE-REMOTE-FILE-DATA

(maybe-write-remote-file-data path iden1 iden2 &key mode)

Wrapper around WRITE-REMOTE-FILE which returns :NO-CHANGE and avoids touching PATH if PATH’s content is already the prerequisite data identified by IDEN1 and IDEN2 and PATH has mode MODE.

Generic function: CONNECTION-UPLOAD

(connection-upload connection data)

Subroutine to upload an item of prerequisite data to the remote cache. The default implementation will work for any connection which implements CONNECTION-WRITE-FILE and CONNECTION-RUN, but connection types which work by calling CONTINUE-DEPLOY* or CONTINUE-DEPLOY*-PROGRAM will need their own implementation.

Generic function: CONNECTION-CLEAR-DATA-CACHE

(connection-clear-data-cache connection iden1 iden2)

Delete all versions of the data identified by IDEN1 and IDEN2 from the remote cache of CONNECTION. Called by UPLOAD-ALL-PREREQUISITE-DATA before uploading new versions of data, to avoid them piling up.

Function: UPLOAD-ALL-PREREQUISITE-DATA

(upload-all-prerequisite-data &optional connection)

Upload all prerequisite data required by the current deployment to the remote cache of the current connection hop, or to the remote cache of CONNECTION.

This is called by implementations of ESTABLISH-CONNECTION which call CONTINUE-DEPLOY* or CONTINUE-DEPLOY*-PROGRAM.

Function: DATA-PATHNAME

(data-pathname root &rest segments)

Function: LOCAL-DATA-PATHNAME

(local-data-pathname &optional iden1 iden2 version)

Get a pathname where an item of prerequisite data may be cached, ensuring that parent directories exist. This is exported for use by prerequisite data sources which work by generating new files and need somewhere to store them. It should not be used by properties, or data sources which return objects referencing existing files.

Note that since prerequisite data sources are queried only in the root Lisp, but items of prerequisite data are never uploaded to the root Lisp, there is no risk of clashes between fresly generated files and cached copies of files.

Function: REMOTE-DATA-PATHNAME

(remote-data-pathname &rest args)

Remote caches

Generic function: GET-REMOTE-CACHED-PREREQUISITE-DATA

(get-remote-cached-prerequisite-data connection)

Return a list of items of prerequisite data in the cache on the remote side of CONNECTION, where each entry is of the form

'(iden1 iden2 version).

Local caches

Function: GET-LOCAL-CACHED-PREREQUISITE-DATA

(get-local-cached-prerequisite-data where)

Scan a local cache of prerequisite data at WHERE, and return a list of items of prerequisite data where each entry is of the form

'(iden1 iden2 version).

This is exported for use by implementations of CONNECTION-UPLOAD, which should always supply a value for WHERE.

Function: GET-HIGHEST-LOCAL-CACHED-PREREQUISITE-DATA

(get-highest-local-cached-prerequisite-data iden1 iden2)

Get the highest version of prerequisite data identified by IDEN1 and IDEN2 available in the local cache.

This is exported for use by prerequisite data sources which work by generating new files and need somewhere to store them. It should not be used by properties, or data sources which return objects referencing existing files.

Passphrases

Class: WRAPPED-PASSPHRASE

Function: WRAP-PASSPHRASE

(wrap-passphrase passphrase)

Make an object which is unprintable by default to contain a passphrase.

Function: GET-DATA-PROTECTED-STRING

(get-data-protected-string iden1 iden2)

Like GET-DATA-STRING, but wrap the content in an object which is unprintable by default. Intended for code which fetches passwords and wants to lessen the chance of those passwords showing up in the clear in the Lisp debugger.

Variable: *DATA-SOURCE-GNUPGHOME*

Home directory for gnupg when used in a data source.

Because gnupg uses Unix domain sockets internally, this path should be short enough to avoid the 108 char limit on socket paths.