This is a success story of running a medium-sized rust project (>25k lines) for roughly two years in production and developer environments. Included are the objectives, challenges faced, and general learnings and observations.

But first, some context and motivation. I often say that writing code is easy part. It's running/maintaining the code that's hard. When dealing with developer tools, I will add that distributing code is hard. I once worked on a team that maintained a single tool used by developers, the CI system, and by production/staging machines. This was a python executable distributed via pex files. This brought with it several problems. First, the pex executables were dependent on the version of python installed on whatever system you were trying to run the tool on. This necessitated having a consistent version of python everywhere the tool was distributed, which was relatively easy on production hosts, but somewhat difficult on developer machines. This problem was exacerbated by the fact that. Developers were running on osx, while the rest of the machines in our infrastructure were linux-based. While we could distribute the necessary python packages with the pex files, we had to build two different versions because of library dependencies (postgres, etc.).

Second, because the tool was a monolith - a kitchen-sink of functionality - we had to ensure cross-platform compatibility for some components that would never be used by a developer and, likewise, some components that would never be used by a production machine. Paired with the first problem, this meant we spent time fixing compatibility issues in code in areas where it didn't matter because that functionality would never be used on the system it was incompatible with.

With the size and scope of the tool came the third problem: maintainability. Since the tool did so much and in so many different contexts, it became a web of poorly documented, lightly tested, fragile code. Error messages were unhelpful and often relied on stack traces to yield context into what exactly went wrong. This wasn't entirely the fault of the tool, as python, especially then, offered few tools to enforce interfaces and type safety outside of pedantic unit tests and developers remembering to include type-hints.

Lastly, the compressed pex format meant that just running the tool was slow, taking upwards of a second or more just to unpack the various files and load the interpreter.

Goals for the new project.

When it came time to write a similar tool at my next job, I set the following goals to address the issues we'd encountered with the python tool.

  • Distribute as self-contained binaries with minimal external dependencies.*
  • Extensible interface that allows for adding only the components needed in a given context (we don't need the same functionality in CI as we do locally and vice versa).
  • Implemented in a language that enables compile-time checking of interfaces and types.
  • Any error message the user sees is one the developer knew could be raised, and it was handled accordingly.

* third-party tools as dependencies like docker and kubectl are exceptions.

Additional goals would include

  • Self update functionality, with a way to specify the additional functionality required by the user (which extensions to install/update).
  • An interface that adheres to the principle of least astonishment.

Several languages were candidates for this project, but I settled on rust for the following reasons.

  • Since I would be almost the sole contributor, I wanted something familiar and safe.
  • The compiler and borrow-checker would do much of the heavy lifting, reducing the number of "dumb mistakes" that I (or someone new to the code) could make.
  • The error-handling mechanism would ensure that no unanticipated errors could be shown to the user.
  • Compiled targets with few, if any external dependencies on linux via musl targets, and working bins on osx (both ARM and x86).

Extensibility

Tools like git, cargo, and others allow for user-defined "external" subcommands by providing a discovery and execution mechanism that treats any executable in $PATH prefixed by the tool name and an extension of the tool itself (cargo-watch, git-flow). Implementing this mechanism has the benefit of allowing the tool to be split up according to context, while presenting a consistent interface and/or entry point to the end user. In the event that a dependency required by a CI or production workflow could not be compiled on osx, that workflow could be relegated to an extension that would not be distributed to developers, solving the issue faced by my previous company of requiring that the entire tool be compatible on all supported operating systems. Because the discovery mechanism is just looking for properly-named executables, it also means extensions could be written in any language.

Architecture

The tools are distributed as a set of binaries which are backed by a subset of libraries. The libraries are broken up as follows:

hum-api-wrapper # library for working with arbitrary 3rd-party apis
hum-common-cli  # library for shared cli arg structures
hum-core        # shared functionality available to all apps that import it
hum-derives     # this is a proc macro crate needed by hum-core
hum-gitlib      # library for working with git
hum-resources   # library exposing resources like app, container, etc.

And the distributed binaries:

hum             # the top-level hum app (contains auth, debs, list)
hum-db          # database connection app
hum-dev         # local development utilities (and used by ci)
hum-jenkins     # interacting with jenkins runs
hum-new         # creating new projects like libraries and services
hum-notify      # sending notifications via slack and other means
hum-ops         # containing utilities to aid in devops tasks
hum-poetry      # wrapper for containerized poetry with codeartifact
hum-stack       # manipulate local stacks

To further reduce duplication, hum-core exposes a procedural macro for handling static dispatch of subcommands and external commands in the context of clap applications. This CliDispatch macro handles the majority of the boilerplate, including passing around the top-level Settings object and determining the prefix for external subcommand discovery. In conjunction with the CliSubcommand and AsyncCliSubcommand traits, the macro only requires users to define enum variants and implement the required trait in order to have a functioning command or subcommand.

// hum-ops/src/cli/app/mod.rs
#[derive(Debug, Subcommand, CliDispatch)] // derive CliDispatch
#[cli_meta(Settings)]         // indicate type of metadata object
#[cli_async]                  // indicate async command
pub enum Commands {
    Deploy(deploy::Deploy),
    ImageBuild(image_build::ImageBuild),
    PrunePrs(prune_prs::PrunePrs),
}

// hum-ops/src/cli/app/deploy.rs
#[async_trait]
impl AsyncCliSubcommand for Deploy {
    type Metadata = Settings;

    async fn run(&self, settings: &Self::Metadata) -> Result<()> {
        // ...
    }
}

hum-core also provides helpers for a consistent UI experience via its ui module, which includes styled versions of prompts, success/fail/warn messages, and generic rendering of tabular data. Also included is settings file parsing (with a mechanism to allow external commands to specify subsets of settings they care about), as well as a file/directory discovery trait that allows for discovering things at or below a specified path (used for discovering apps, configurations, and libraries in our monorepo).

The main entry point is the hum binary, which contains the core functionality of the self-updater, extension installer (for external subcommands), and the auth subcommand for authenticating users to AWS and other providers. This is the only binary required to work on all platforms.

Specifying the following the settings file instructs the extension installer to install the specified extensions.

# an example of the subset requested by a engineer on one of the backend teams
[bootstrap.extern_cmds]
db = "*"
dev = "*"
new = "*"
poetry = "*"
stack = "*"

Users and machines in the infrastructure can then request a minimal set of components in order to complete a given task.

Effects, adoption, and outcome

For over a year and a half, these tools have been used by our CI and developers with nearly zero issues. The few that have come up were due to third-party interface changes (docker, jenkins api, etc.). Considering in the same time there have been nearly 700 commits, some including major refactors, it is impressive that we were able to push so much code with so few issues. Part of this is due to the tests put in place to build and verify the tools on various architectures and operating systems, but most of this is due to the safety guarantees rust provides us paired with type-driven design patterns that eliminate the possibility of mixing up arguments and misusing interfaces.

Previously, operations like building an app or running tests were defined as makefile targets, with different targets for CI and local development. Because care was taken to make the new tools context-aware, a developer can see a failed command invocation in CI and then execute the same command locally without having to figure out which additional arguments they need to make something work on their machine. This has reduced duplication and confusion regarding which target to use in which context.

There is currently a 100% adoption rate, not because these tools are the only way to do something, but because they're far easier than the workflows they abstract. The automatic discovery and context-aware features means that developers do not have to know exactly where something lives in the repo in order to work with it. The ui and common-cli libraries enable a consistent interface experience across the various tools (failure messages always look the same, warnings are consistent, etc.). Installation and updates are easy: simply put this binary somewhere in your $PATH (and there's an installer script to make that even easier).

Lessons learned

The good

  • Rust will force you to slow down and think more carefully about data ownership and propagation, but this is ultimately beneficial in the long term when it comes to refactoring and adding guard-rails for new contributors.
  • Implementing new features, and changing existing ones, was faster because radical changes could be made without fear of introducing uncaught errors because the compiler would catch most mistakes.
  • The error handling paired with context from the anyhow crate allows for never showing an unhandled exception to the user, making for a much nicer interface.
  • While the problem did not crop up, we have the ability to ignore osx compatibility for functionality not used by developers because of the extension mechanism.
  • Ease of adoption because developers do not have to worry about their version of some dependency (or their python version) from preventing them from installing the tools.

The meh:

  • Compilation times for --release are still a bit of an issue, but breaking up the libraries into smaller crates enables a bit more parallelism during builds.
  • Cross-compilation from linux to osx targets is perhaps possible but probably not worth the effort. We currently build our aarch64 and x86 darwin targets on a mac mini, which also gets us around Apple's gatekeeper BS.
  • Workspace crate versions can be a little of a problem in terms of incompatible dependencies in separate workspace crates. There were a few times when 'hyperx' dependencies prevented upgrading unrelated crates.
    • A private cargo registry may have allowed us to have some tools outside of the workspace, but we did not and do not run one.

Overall, the decisions made at the outset of this project were mostly the correct ones. I would change very little if faced with a similar project in the future. The confidence that rust allows in terms of making new releases knowing you haven't broken anything is well worth any initial productivity impacts opting for a stricter language may bring. Rust may not be the best choice for every problem, but in terms of command line, cross-platform tooling, I'm finding it hard to beat.