tori

tori is a tool to track system configurations and replicate them.

For an overview, see the discover page. If you’d like a more detailed description of what tori is, its purpose, origins and goals, see the announcement blog post.

For updates and access to source code, visit the project website .

Installation

First of all, check if your operating system is supported.

tori is still in very early development and not yet packaged. To try it, clone its repository to your system:

git clone https://brew.bsd.cafe/jutty/tori.git /path/to/repository

Next, create ~/.config/tori/tori.conf with the following content:

tori_root = /path/to/repository

If you clone it to the default location, ~/.local/share/tori, the above step is not necessary.

Finally, you need to symlink the tori file at the repository root to somewhere on your $PATH:

ln -s /path/to/repository/tori $HOME/.local/bin/tori

Usage

You are now ready to start creating your configuration in ~/.config/tori. You could start by creating a packages file there with the packages you expect to have installed.

When you want to compare the configuration with your actual system, you can use tori check.

If you did that at this point, with only a few packages in your ~/.config/tori/packages file, tori would likely give you a long list of all the other packages you have manually installed but have not added to this package list. Among the options on how to proceed, you could now “Add all to configuration”.

Then, if you run tori check again it should exit immediately, as all the packages match.

To learn more about all the ways you can declare your system using the configuration directory, see the documentation page on configuration.

Currently, the following commands are implemented:

• check: check for divergences between the configuration and the system
• cache: force an update of the local package cache
• help: show a usage summary with supported options
• version: print the current version and its release date

To issue a command, use tori <command>, as in tori check.

Configuration

tori looks for configuration in ~/.config/tori.

In this directory, you can create the following files and directories:

• tori.conf: configures options that will determine how tori itself behaves
• packages: a list of packages containing one package name per line
• base: a directory containing any number of files that tori will copy to specific locations

Configuration options

The tori.conf file must use the following format:

tori_root = ~/.local/apps/my-tori-installation

For a complete reference on this file, including its syntax and all available options, see the tori.conf documentation page.

Package lists

tori will read the packages file and check if it matches the list of currently installed packages. If it does not match, it by default will ask you how to proceed.

The application only concerns itself with manually installed packages. Any dependencies pulled automatically should not be picked up by tori unless you have manually installed that dependency or marked it as such.

This is ensured by specifically querying your package manager for a list of manually installed packages, so tori should actually not even be aware of which automatically-installed packages you have on your system.

In your packages file, empty lines and lines beginning with a # are ignored.

Base files

tori will go through the contents of the ~/.config/tori/base directory and take different actions depending on how they are laid out.

file_strategy = spare

Tree strategy

The default way tori compares files is called the tree strategy. It expects you to have top-level directories inside base that match the directories starting from your system’s root (for example, a base/home directory).

tori will recursively inspect this directory for files and compare them to the contents of the matching directories in the system.

For example, if you have the following configuration:

~/.config/tori
└── base
    └── home
        └── alice
            └── .shrc

Upon running tori check, tori will compare your configuration’s .shrc file at ~/.config/tori/base/home/alice/.shrc to /home/alice/.shrc.

If any of the files in your configuration do not exist on the corresponding location, tori will copy them to it. If they do exist and match, it won’t do anything. And if they differ from their counterparts, by default tori will ask you how to proceed.

Differs: ~/.shrc
 [1] Overwrite system
 [2] Overwrite configuration
 [3] Show difference
 [0] Exit
Choose an option number:

In the example above, ~/.config/tori/base/home/alice/.shrc and /home/alice/.shrc have differed, so the user is presented with choices on how to conciliate them.

~/.config/tori
├── base
│   └── .shrc
└── files

.shrc ~/.shrc

tori.conf

The tori.conf file is found at the root of the configuration directory, which defaults to ~/.config/tori.

Each line must use the following format:

tori_root = ~/.local/share/tori
file_strategy = tree

The values in the example above are the default ones, meaning if you do not specify these options they are what will be considered their values.

Options

The following configuration options can be used to specify how you want tori to behave.

Application behavior

• tori_root: The directory where you installed tori. This is useful when you installed tori by simply cloning its repository. It should point to the directory where the README.md and the tori files are.
  • ~/.local/share/tori (default)
  • <path>: Any valid filesystem path. See Syntax for possible home directory substitutions.

File management

• file_strategy: What file structure should be used when merging the files from your ~/.config/tori/base directories. For a more detailed explanation of what each strategy does, see the configuration page.
  • tree (default): mirror the structure of the filesystem, with base being the equivalent of its root (/)

Syntax

Configuration options are case insensitive. The spaces around the = character are optional. Blank lines and lines beginning with a # are ignored.

You can use ~ and $HOME to represent your home directory. $HOME will be replaced in every occasion with your home directory, while ~ will only be replaced for the first occurrence in the configuration value.

Values containing spaces must be wrapped in double quotes.

Configuration values must not contain the character *.

check

When you run tori check, tori will look into your configuration files and perform two tasks:

• Compare the files inside the base directory to the corresponding files on the system
• Compare the package list in the packages file to the installed packages

If any divergence is found, it will by default prompt you on what action to take.

cache

To know if package names are valid, tori will ask your package manager for a list of all available packages and store it in ~/.cache/tori.

By default, this cache will be refreshed daily. This happens before the first operation that uses the cache is triggered in a day.

If you would like to manually refresh this cache, you can use the tori cache command.

When this happens, you might see a password prompt from your authentication tool (such as sudo or doas) with the message “Updating package cache”.

Package conflict resolution

When running tori check, tori will generate two package lists: one with your installed packages and one from the packages file in your configuration directory.

In case it finds differences between the two lists, its default behavior is to ask you what to do. This approach is described in the following section.

Interactive resolution

If it finds a difference between the two package lists, tori will present you with a list of packages and a menu with several conflict resolution strategies:

[tori] 16:38:19: System and configuration packages differ

Packages on configuration but not installed: bastille fend ksh xh
  [1] Install all
  [2] Enter packages to install
  [3] Remove all from configuration
  [4] Enter packages to remove from configuration
  [5] Decide on editor
  [6] Cancel
Choose an option [1-6]:

tori will perform two separate checks that may produce two separate dialogues like the one above. It checks for packages found in the configuration that are missing from the system and for packages found in the system but missing from the configuration.

If it finds packages in your configuration that are not installed on the system, it will present you with the following choices:

• Install all: The exact list of packages just displayed is passed to the package manager’s install option. The package manager may ask for additional confirmation.
• Enter packages to install: Allows you to provide a space-separated list of packages to install. These packages will be checked for invalid characters and invalid package names. If there is an issue, you will be prompted again.
• Remove all from configuration: Removes all packages from your configuration file.
• Enter packages to remove from configuration: Allows you to provide a space-separated list of packages to remove from the configuration file.
• Decide on editor: Opens your editor and allows you to choose individually what actions to take for each package.
• Cancel: Do nothing and close the dialog

If it finds packages installed on the system but not listed in your configuration, it will present you with the following choices:

• Uninstall all: The exact list of packages just displayed is passed to the package manager’s uninstall option. The package manager may ask for additional confirmation.
• Enter packages to uninstall: Allows you to provide a space-separated list of packages to uninstall. These packages will be checked for invalid characters and invalid package names. If there is an issue, you will be prompted again.
• Add all to configuration: Adds all packages to your configuration file.
• Enter packages to add to configuration: Allows you to provide a space-separated list of packages to add to the configuration file.
• Decide on editor: Opens your editor and allows you to choose individually what actions to take for each package.
• Cancel: Do nothing and close the dialog

Decide on editor

If you choose to decide what to do in your editor, the editor set on your $EDITOR environment variable will be launched with the following text:

# Options:
#   [i]nstall     Install package to system
#   [r]emove      Remove from configuration
#   [s]kip        Do not take any action

# Providing just the value between brackets is sufficient
# Replace 'skip' below with the desired option

skip bastille
skip fend
skip ksh
skip xh

This example shows packages on configuration but not installed. The inverse set of operations are displayed on top if you have installed packages that are not in your configuration:

# Options:
#   [u]ninstall   Uninstall package from system
#   [a]dd         Add to configuration
#   [s]kip        Do not take any action

# Providing just the value between brackets is sufficient
# Replace 'skip' below with the desired option

skip bastille
skip fend
skip ksh
skip xh

Edit the skip option on each package to decide what action to take on them:

# Options:
#   [u]ninstall   Uninstall package from system
#   [a]dd         Add to configuration
#   [s]kip        Do not take any action

# Providing just the value between brackets is sufficient
# Replace 'skip' below with the desired option

uninstall bastille
u fend
a ksh
skip xh

In the example above, bastille and fend would be uninstalled, ksh would be added to the configuration and no action would be taken for xh.

Backups

tori will create backup copies of files before overwriting them.

This includes tori’s own configuration directory, meaning that when you overwrite your own configuration files through tori’s dialogs or non-interactive options, it will also backup your configuration files before doing so.

There are two types of backups: canonical and ephemeral. Each lives inside a separate directory in the backup directory, which defaults to ~/.local/state/tori/backup.

Both directories, as well as the backup directory, may be deleted at will. They will be automatically recreated whenever an overwrite operation takes place.

Canonical backups

Canonical backups are created the first time tori is told to modify a given file.

These are meant to represent the state of the file system prior to tori’s intervention. If tori is running on a recently-installed system, the canonical backups should mirror the original state of the system as of its installation.

If there is already a canonical backup, it instead proceeds to the creation of an ephemeral one.

Ephemeral backups

Ephemeral backups are timestamped backups created every time tori has to modify a file. This is mainly meant as a safety net against undesired consequences when using non-interactive options.

If an ephemeral backup is being requested when there is already a matching timestamp, meaning they were requested within the same second, the older one will be overwritten.

Directory structure

tori creates a backup directory in ~/.local/state/tori/backup that contains two other directories:

• canonical
• ephemeral

Both of them will mirror the directory structure from the root of the file system, as shown in the example below.

~/.local/state/tori
└── backup
    ├── canonical
    │   ├── etc
    │   │   ├── pf.conf
    │   │   └── rc.conf
    │   └── home
    │       └── alice
    │           └── .shrc
    └── ephemeral
        ├── etc
        │   ├── pf.conf_2024-09-03T08-54-51
        │   └── pf.conf_2024-09-03T09-02-05
        └── home
            └── alice
                └── .shrc_2024-09-03T08-46-54

In this case, the rc.conf file was overwritten once, creating a canonical backup corresponding to the first time tori touched it. pf.conf, on the other hand, was overwritten three times and .shrc twice, which is why there is a canonical copy for the first change and other timestamped ephemeral copies for subsequent ones.

Backlog

This document is meant as a list of planned changes. While their inclusion here means their implementation is desirable, the timing will depend on other priorities that could be unlisted.

Configuration

• Allow imperative configuration commands through a parsed file containing instructions

Package management

• When deciding to install/uninstall packages, there should be the option to also add/remove them from the configuration, both interactively and non-interactively
• Add install, uninstall and query commands to check the status of packages and simultaneously add/remove to/from system and configuration
  • Once install/uninstall and add/remove to/from configuration conflict resolution strategies have been implemented, the same functions can be reused to implement this

File management

• Allow merging spare files on ~/.config/tori/base by specifying where they should be copied to

Metaconfiguration

• Make configurable:

  • authorizer command (sudo/doas/su)
  • package cache update frequency (currently defaults to daily)

• Configurable hostname discernment:

  • extension, as in packages.hostname, base.hostname base/file.hostname, hostname.conf
  • directory, as in hostname/packages, hostname/base, hostname.conf

Utilities

• Add a --break flag to the log function to print a newline before any output
  • Add this flag to the log debug "user:\n$user_packages" call on scan_packages() in configuration.sh
• set_opts() would be more ergonomic if it took on and off instead of - and +

Portability

tori’s role is to manage configuration files and installed packages, allowing the transfer of this configuration between different versions of an operating system or even between different Unix and Unix-like operating systems, provided they are supported.

Given this purpose, it ideally should not require the user to install even a single package in order to run. It must be a portable application with minimal dependencies so it can perform its functions on brand new systems where very few packages have been installed and little to no configuration has been done.

To achieve this portability and independence, it is meant to run on a POSIX-compatible shell where POSIX utilities are available. If a system does not provide this, it is very unlikely tori can function.

Note that while tori expects a POSIX shell, it is not meant as a universal tool able to run on any POSIX-compliant system. A POSIX shell is required because it is the interpreter for tori’s source code and it was chosen because it comes pre-installed on most Unix-like systems.

For some of its functionalities, tori needs to be running in a supported operating system. It has specific package management and service management features that work by abstracting the actual operating system’s interfaces behind a function that detects what system is running and then runs the appropriate commands.

The currently supported operating systems are tracked in the Support Matrix page.

While it strives to do so, in some situations, tori may perform tasks by relying on resources or features not specified by POSIX, such as when there is no option or the available option presents readability or usability issues.

In these situations, tori tends to rely on specific functions that will switch their behavior depending on the operating system’s support for the operation. Regardless, all of its features are expected to work under supported operating systems.

Below is a list of assumptions made about what is supported in the running environment:

• shell
  • local
  • read with read -r -p <prompt> <variable_with_user_input> syntax
  • mkdir with -p
  • find
  • grep
  • sed
  • xargs
  • uname
  • date
    • with nanoseconds as %N
      • While nanoseconds in date is not specified by POSIX, it is available on the currently supported systems and is used only when $DEBUG is set in the environment
    • with -r for getting a modification date
      • This feature is not specified by POSIX. So far it was tested on FreeBSD and Void Linux
  • env at /usr/bin/env
    • While this may be an issue from a portability standpoint, hardcoding the path where sh is also poses another portability issue. A more robust way to find it would be desirable

The POSIX specification used as reference is the IEEE Std 1003.1-2024 Edition.

tori is developed and tested on the sh Almquist shell as shipped by FreeBSD and the dash Almquist shell as shipped by Void Linux.

Utilities

The utility functions available at src/utility.sh provide functionality that is very simple and sometimes generally used across the whole application.

log

log <level> <message>

The log function takes a log level between 1 and 5 as its first argument and a message as its second argument. The log message must be wrapped in double quotes, otherwise only the first word will be considered part of the message and the rest will be discarded.

If the environment variable DEBUG is set to a number corresponding to the provided log level or greater, the message is printed.

The current log levels are:

0. user: Always displays, with [tori] at the very left followed by a second-precision timestamp, a colon and the message. This is meant for user messages that are always relevant. This level is not affected by the value of the DEBUG environment variable
1. fatal: For critical, unrecoverable or unsupported situations. The application must immediately exit after printing the message. Displays when the log level is 1 or higher with a second-precision timestamp.
2. error: For unexpected conditions where potential undesired behavior, loss of user data or other unintended state may happen or has already happened. The application may exit immediately after printing the message. Displays when the log level is 2 or higher with a second-precision timestamp.
3. warn: For expected conditions already handled by the application that may potentially be unintended behavior, depending on the user’s intentions. A warning should not precede the application exiting. Displays when the log level is 3 or higher with a second-precision timestamp. This is the default when DEBUG is not set or is set to a non-numerical value.
4. info: For messages that do not represent errors or unintended situations, but provide more insight into the current state or behavior. This level is for development purposes. Displays when the log level is 4 or higher with a nanosecond-precision timestamp.
5. debug For messages that express any change in state or behavior, typically to help understand the application’s flow of execution. This level is for development purposes. Displays when the log level is 5 or higher with a nanosecond-precision timestamp. Levels higher than 5 are not valid, but are brought down to 5.

The log level defaults to 3 (warn). To completely disable log messages (except for the user level), set DEBUG=0.

If DEBUG is set to a value greater than 5, it will be reset to 5. If it is set to any non-numerical value, it will be set to the default, 3.

All log messages are printed to STDERR so as not to shadow function return values.

set_opts

set_opts <on|off>

The set_opts function checks if the shell options are supported and sets them. If they are not supported, it considers the shell unsupported. This causes it to print a fatal error message and return exit code 1.

This function is called with the on argument in the very early stages after the application just started in order to enable these options for the remainder of execution. It may then be used again to perform operations that require these options to be unset.

For example:

set_opts off # disable options
echo "$potentially_unset_variable" 
set_opts on # re-enable options

It modifies the following options:

• errexit: Exit after any non-zero return¹
• nounset: Exit when trying to expand an unset variable

To enable them, use set_opts on. To disable them, use set_opts off.

print_help

print_help

The print_help function is responsible for printing help text when the user runs tori help, tori -h or tori --help. It takes no arguments.

juno@akuma:~/tori $ tori -h

  tori: configuration managent and system replication tool

    Options:

        check           compare configuration to system state
        cache           refresh the local package cache

        version         print current version with release date
        help            show this help text

  See 'man tori' or https://tori.jutty.dev/docs for more help

The help text contains a brief description of what tori is, what commands are available and where to find more help via the man page or this documentation website.

prepare_directories

prepare_directories

The prepare_directories function runs at the very start of execution in order to verify that critical directories exist. It takes no arguments.

The directories are:

• CONFIG_ROOT, default ~/.config/tori: application exits with a fatal error and exit code 1 if not found
• TMP_ROOT, default /tmp/tori: created if not found
• CACHE_ROOT, default ~/.cache/tori: created if not found
• BACKUP_ROOT, default ~/.local/state/tori/backup: created if not found
  • with subdirectories canonical and ephemeral

ask

ask <prompt> <option>[,option2,option3[, ...]]

The ask function will prompt the user for one choice among the comma-separated list of alternatives.

It will also include an “Exit” option that does not need to be provided in the list of alternatives.

This function will print a number to standard output corresponding to the chosen option starting from 1, and 0 if the “Exit” option is chosen.

tildify

tildify <path>

The tildify function takes a file path and replaces the first occurrence of the user’s home directory with a ~ (tilde) character. The home directory string to replace is determined by reading the value of the presently set $HOME environment variable.

tori determines what is the running operating system through the /etc/os-release file and the output of uname -s.

If an /etc/os-release file is present, it takes precedence over the output of uname. Both the NAME and ID values will be looked at. This is aimed at helping to disambiguate between different variants of the same operating system.

The NAME value may be the only queried value if for the given supported operating system it is enough to disambiguate between the variants tori needs to be aware of.

In case there is no /etc/os-release file found, the output of uname is the next value considered.

If a supported operating system is not detected on neither of these, tori will exit with an error.

check

The check option performs two tasks through the configuration processing functions available at src/configuration.sh

1. Traversing the configuration’s base directory to assemble a file list containing the equivalent full paths starting from the actual filesystem’s root (/), checking if these paths correspond to actual files and if they differ from their counterparts in the base directory
2. Comparing the installed packages with the packages file at the root of the configuration directory

The first task is currently accomplished by resorting to find. While this allows for cleaner code, it relies on a utility with variable behavior across operating systems. Given the simplicity of the query, a more portable yet less readable option might be using a POSIX-compliant wildcard such as .[!.]* ..?* * to match the files directly (e.g., in a for loop). Another option that may provide both readability and portability is repeating the match, once for hidden file and once for non-hidden files.

The second task is accomplished by resorting to the package management functions available at src/package.sh. The package_manager function abstracts the actual package manager in the underlying system and provides an OS-independent way to query manually installed packages.

By parsing the packages configuration file at the root of the configuration directory (~/.config/tori/packages by default), a final package list is obtained. The package manager is also queried for its list of manually installed packages. Both lists are then sorted and deduplicated before they can be filtered by each other using grep inverted matching.

This allows obtaining two-way differences and displaying them to the user. If no conflict resolution strategy has been configured or passed through the command line interface, several options are given:

1. Install/uninstall all
2. Enter packages to install/uninstall
3. Add all to/remove all from configuration
4. Enter packages to add to/remove from configuration
5. Decide in editor

cache

To determine if a package really exists, there must be a quick way to query a list of all available packages that does not mean individually making requests with each package name.

To solve this, tori will ask the package manager for a list of all available packages and cache it. It then uses this file to determine if package names are valid.

By default, the cache is updated when the current date (day, month and year) differs from the date of modification of the cache file. Note that this is not the same as 24 hours after the last refresh.

The user may configure a different frequency and can also force a cache refresh through the command line interface using tori cache.

User data

User data is organized in two main categories: configuration and backups.

Configuration

The configuration directory defaults to ~/.config/tori. See the usage documentation for what are its contents.

tori will look only for regular files inside the configuration directory and currently will ignore symbolic link and any other filetype when scanning it.

Backups

The backups directory defaults to ~/.local/state/tori/backup. It contains copies of files made prior to tori modifying them. Its structure is also documented in the usage documentation, on the backups page.

This directory is not essential to the functioning of tori. It is provided as a safety net for the user and may be deleted at any time. It will be automatically recreated.

User files

tori can track if a given file in the configuration directory is present and matches the content of the corresponding file on the system level.

However, if a file changed on the system level is not in the configuration directory, tori can only alert users of that if the operating system provides some way to compare the present state of the system to the original one prior to user intervention.

One example would be FreeBSD’s intrusion detection system, which provides a way to know which files have been changed.

Another way would be by relying on a file system that provides the ability to compare differences between snapshots, such as ZFS.

A third way would be if the operating system’s package manager provides a command line interface to read the contents of packages and has packages that correspond to the core system, such as Void Linux’s package manager, xbps.

The resource-intensiveness of each of these methods will vary greatly and therefore checking the whole system for changes can be time-consuming or provide overwhelming output.

For this reason, tori by default operates in a more minimal fashion where users take responsibility for adding the files they would like to track to their configuration and only get warnings of untracked files when explicitly asked.

User packages

In order to take advantage of all the features offered by tori for package management, the operating system’s package manager must provide a way to:

• install and uninstall packages from a command-line interface
• query manually installed packages

For information on how the application determines differences between the configuration package list and installed packages, see check.

validate_input_packages

View source

validate_input_packages <package>[ <package>[ ...]]

This function takes a list of package names separated by spaces and verifies that:

• It only contains valid characters
• Each value is a valid package name

Character validation

An OS-specific pattern will be matched against the package name to decide which characters are valid or invalid by the naming standards its package repository uses.

For example, if the OS is FreeBSD, it will set the following pattern:

invalid_characters_pattern='[^A-Za-z0-9\+_\.-]'

If no OS-specific documentation is found on what are the allowed characters in package names, the obtained package list containing all available packages in the operating system’s main repository should be analyzed to determine what is the current range of characters it uses.

The following allowed character groups have been determined so far:

• FreeBSD (pkg)
  • uppercase and lowercase letters
  • numbers
  • dashes
  • underscores
  • dots
  • plus signs

Package name validation

Package name validation is performed by checking if each package name is found in the package cache. This list is retrieved by the package manager and should represent the full range of valid packages available in the queried repository.

Outcome

If there is any error in validation, the function will return 1. In the check command, for instance, this will cause the dialog to prompt the user again for valid package names.

If all package names are found in the cache, the function will have a clean exit with code 0, allowing the calling function to continue and perform any actual operations on these packages.

backup_paths

View source

This function takes a newline-separated list of absolute paths and creates canonical or ephemeral backups as needed.

Paths are assembled by taking the value of $BACKUP_ROOT and concatenating to it /canonical or /ephemeral followed by the absolute path. For ephemeral backups, a timestamp is also appended to the end of the filename.

Checks are performed for permission to read the files being copied. If permission is not granted to the user running tori, the command set in $AUTHORIZE_COMMAND will be used to escalate privilege.

Release process

This document specifies the process leading to a version being ready for release.

Overall steps

1. Update changelog
2. Update documentation and README
3. Update the --version output
4. Update the --help output
5. Create an update on the website if appropriate
6. Tag the last commit of main and web repositories
7. Push tags
8. Create a pull request
9. Merge
10. Update git mirrors
11. Create release
12. Push web changes
13. Announce on relevant channels