About

A collection of tools Standard Ebooks uses to produce its ebooks, including basic setup of ebooks, text processing, and build tools.

Installing the toolset makes the se command line executable available. Its various commands are described below, or you can use se help to list them.

Installation

The toolset requires Python >= 3.10.12.

To install the toolset locally for development and debugging, see Installation for toolset developers.

Optionally, install Ace and the se build --check command will automatically run it as part of the checking process.

Ubuntu 24.04 (Noble) users

# Install some pre-flight dependencies.
sudo apt install -y calibre default-jre git python3-dev python3-pip python3-venv pipx

# Install the toolset.
pipx install standardebooks

Ubuntu 20.04 (Trusty) users

# Install some pre-flight dependencies.
sudo apt install -y calibre default-jre git python3-dev python3-pip python3-venv

# Install pipx.
python3 -m pip install --user pipx
python3 -m pipx ensurepath

# Install the toolset.
pipx install --python=3.12 --fetch-missing-python standardebooks

Optional: Install shell completions

# Install ZSH completions.
sudo ln -s $(pipx environment --value PIPX_LOCAL_VENVS)/standardebooks/lib/python3.*/site-packages/se/completions/zsh/_se /usr/share/zsh/vendor-completions/_se && hash -rf && compinit

# Install Bash completions.
mkdir -p $HOME/.local/share/bash-completion/completions/
ln -s $(pipx environment --value PIPX_LOCAL_VENVS)/standardebooks/lib/python3.*/site-packages/se/completions/bash/se $HOME/.local/share/bash-completion/completions/se

# Install Fish completions.
ln -s $(pipx environment --value PIPX_LOCAL_VENVS)/standardebooks/lib/python3.*/site-packages/se/completions/fish/se.fish $HOME/.config/fish/completions/

Fedora 43 users

# Install some pre-flight dependencies.
sudo dnf install pipx python3-devel gcc libxslt-devel calibre git java-25-openjdk-headless

# Ensure $PATH environment variable is correctly set up for pipx.
pipx ensurepath

# Install the toolset.
pipx install standardebooks

Optional: Install shell completions

# Install ZSH completions.
sudo ln -s $(pipx environment --value PIPX_LOCAL_VENVS)/standardebooks/lib/python3.*/site-packages/se/completions/zsh/_se /usr/share/zsh/vendor-completions/_se && hash -rf && compinit

# Install Bash completions.
mkdir -p $HOME/.local/share/bash-completion/completions/
ln -s $(pipx environment --value PIPX_LOCAL_VENVS)/standardebooks/lib/python3.*/site-packages/se/completions/bash/se $HOME/.local/share/bash-completion/completions/se
# Install Fish completions.
ln -s $(pipx environment --value PIPX_LOCAL_VENVS)/standardebooks/lib/python3.*/site-packages/se/completions/fish/se $HOME/.config/fish/completions/se.fish

macOS users

1. Install the Homebrew package manager. Or, if you already have it installed, make sure it’s up-to-date:

   brew update
   

2. Install the tools with Homebrew:

   brew install standardebooks
   

pipx

Install the tools with pipx:

# Install some pre-flight dependencies.
brew install cairo calibre git openjdk pipx python@3.12
pipx ensurepath
sudo ln -sfn $(brew --prefix)/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk

# Install the toolset.
pipx install --python "$(brew --prefix)"/bin/python3.12 standardebooks

# Optional: Bash users who have set up bash-completion via brew can install tab completion.
ln -s $(pipx environment --value PIPX_LOCAL_VENVS)/standardebooks/lib/python3.*/site-packages/se/completions/bash/se $(brew --prefix)/etc/bash_completion.d/se

# Optional: Fish users can install tab completion.
ln -s $(pipx environment --value PIPX_LOCAL_VENVS)/standardebooks/lib/python3.*/site-packages/se/completions/fish/se $HOME/.config/fish/completions/se.fish

OpenBSD 6.6 Users

These instructions were tested on OpenBSD 6.6, but may also work on the 6.5 release as well.

1. Create a text file to feed into pkg_add called ~/standard-ebooks-packages. It should contain the following:

   py3-pip--
   py3-virtualenv--
   py3-gitdb--
   jdk--%11
   calibre--
   git--
   

2. Install dependencies using doas pkg_add -ivl ~/standard-ebooks-packages. Follow linking instructions provided by pkg_add to save keystrokes, unless you want to have multiple python versions and pip versions. In my case, I ran doas ln -sf /usr/local/bin/pip3.7 /usr/local/bin/pip.

3. Add ~/.local/bin to your path.

4. Run pip install --user pipx

5. If you’re using KSH from base and have already added ~/.local/bin, you can skip pipx ensurepath because this step is for Bash users.

6. The rest of the process is similar to that used on other platforms:

   # Install the toolset.
   pipx install standardebooks
   

Installation for toolset developers

If you want to work on the toolset source, it’s helpful to tell pipx to install the package in “editable” mode. This will allow you to edit the source of the package live and see changes immediately, without having to uninstall and re-install the package.

To do that, follow the general installation instructions above; but instead of doing pipx install standardebooks, do the following:

git clone https://github.com/standardebooks/tools.git
pipx install --editable ./tools

Now the se binary is in your path, and any edits you make to source files in the tools/ directory are immediately reflected when executing the binary.

Running commands on the entire corpus

As a developer, it’s often useful to run an se command like se lint or se build on the entire corpus for testing purposes. This can be very time-consuming in a regular invocation (like se lint /path/to/ebook/repos/*), because each argument is processed sequentially. Instead of waiting for a single invocation to process all of its arguments sequentially, use GNU Parallel to start multiple invocations in parallel, with each one processing a single argument. For example:

# Slow: Each argument is processed in sequence.
se lint /path/to/ebook/repos/*

# Fast: Multiple invocations each process a single argument in parallel.
export COLUMNS; parallel --keep-order se lint ::: /path/to/ebook/repos/*

The toolset tries to detect when it’s being invoked from parallel, and it adjusts its output to accommodate.

We export COLUMNS because se lint needs to know the width of the terminal so that it can format its tabular output correctly. We pass the --keep-order flag to output results in the order we passed them in, which is useful if comparing the results of multiple runs.

Linting with pylint and pyright

Before we can use pylint or pyright on the toolset source, we have to inject them (and additional typings) into the venv pipx created for the standardebooks package:

pipx inject standardebooks pylint==4.0.5 pyright==1.1.408 types-requests==2.32.4.20250913 types-setuptools==80.9.0.20250822 types-lxml==2026.2.16

Then make sure to call the pylint and pyright binaries that pipx installed in the standardebooks venv, not any other globally-installed binaries:

cd /path/to/tools/repo
$HOME/.local/share/pipx/venvs/standardebooks/bin/pylint .

# Specify the venv path to `pyright` so that it knows where to look for libraries.
$HOME/.local/share/pipx/venvs/standardebooks/bin/pyright --venvpath=$HOME/.local/share/pipx/venvs/ .

Testing with pytest

Instructions are found in the testing README.

Code style

• In general, we follow a relaxed version of PEP 8. In particular, we use tabs instead of spaces, and there is no line length limit.

• Always use the regex module instead of the re module.

• Complete type hints are required at all times. A run of pylint using the provided configuration should finish with zero output.

Help wanted

We need volunteers to take the lead on the following goals:

• Add more test cases to the test framework.

• When a test fails, the output is usually inscrutable. Can we get better, more readable error messages when a test fails?

• Writing installation instructions for Bash and ZSH completions for macOS.

• Currently, the toolset requires the whole Calibre package, which is very big, but it’s only used to convert epub to azw3. Can we inline Calibre’s azw3 conversion code into our ./vendor/ directory, to avoid having to install the entire Calibre package as a big dependency? If so, how do we keep it updated as Calibre evolves?

• The se lint tool uses Python logic and XPath to help validate various S.E. style requirements in an ebook metadata file, content.opf. Can this be done instead using an XML Schema?

Tool descriptions

• se add-file

  Add an SE template file and any accompanying CSS.

• se british2american

  Try to convert British quote style to American quote style in DIRECTORY/src/epub/text/.

  Quotes must already be typogrified using the se typogrify tool.

  This script isn’t perfect; proofreading is required, especially near closing quotes near to em-dashes.

• se build

  Build an ebook from a Standard Ebook source directory.

• se build-ids

  Change ID attributes for non-sectioning content to their expected values across the entire ebook. IDs must be globally unique and correctly referenced, and the ebook spine must be complete.

• se build-images

  Generate ebook cover and titlepages for Standard Ebooks ebooks, and then build ebook covers and titlepages, placing the output in DIRECTORY/src/epub/images/.

• se build-manifest

  Generate the <manifest> element for the given Standard Ebooks source directory and write it to the ebook’s metadata file.

• `se