Cargo, Docker and mtime

François Best • 25 January 2021 • 4 min read

I wanted to package a web application backend written in Rust (using Rocket as a web server) in a Docker image, for a more portable deployment solution than piping to shell or distro-specific package managers.

Multi-staged builds#

Since Rust compiles to an executable binary, there is no need to bring the whole compiler toolchain into the runtime image, we can leverage the multi-staged build feature introduced in Docker 17.05:

Depending on the dependencies used, some externally-linked C libraries might be needed at runtime, so bare-metal base images like scratch or busybox may not work. I chose to go for the good old debian:stretch1.

I won't go into the details of the naive approach (build all the dependencies and the app code in one step) vs leveraging the cache by first building dependencies, then the app code, it's explained in an article by Isaac Whitfield.

I would like instead to tell the story of a 3:00 am bug that really scratched my head.

Tweaking the Dockerfile#

I was replicating the steps that Isaac took to build his Dockerfile to understand what they did and why, when this couple of commands came up:

RUN rm src/*.rs

COPY ./src ./src

Premature-optimisation brain kicked in and said something like:

Hey, we can totally optimise this, no need to delete the sources, they will be replaced anyway.

It worked. But after a couple of builds, things started going weird.

Instead of running my application, the container would prompt Hello, world!, and die instantly. I put some logs into the build process to see if the cache was acting up, restarted Docker and the host machine, still the problem persisted.

Following the wisdom of the 5 whys, it turned out one cause of the problem was that cargo was not actually rebuilding the app source code. My initial assumtion when "optimising" the Dockerfile had been:

The sources changed, therefore cargo will see it and rebuild them.

Cargo and mtime#

Cargo does not use a hash-based mechanism to check for modified source files, but keeps track of file modification times instead. This is noted in issues #6529 and #2426.

Docker COPY did not change the mtime of main.rs when overwriting the empty shell used for building only the dependencies with the actual app code. At least not consistently, as it worked a few times initially. And there was our root problem.

In Isaac's Dockerfile, this was mitigated by deleting the contents of ./src before copying over the app code, which took care of updating the mtime of main.rs (and all other files).

Conclusion#

Here's the final Dockerfile for reference:

Dockerfile
FROM rustlang/rust:nightly as builder

# Create a shell to build the dependencies
RUN USER=root cargo init --bin factory
WORKDIR /factory

# Build only the dependencies (leverage Docker cache)
COPY Cargo.toml Cargo.lock ./
RUN cargo build --release

# Copy the project sources & build the project
COPY . .

# Sometimes cargo does not see that main.rs changed,
# use `touch` to change the modification date.
# See https://github.com/rust-lang/cargo/issues/6529
# and https://github.com/rust-lang/cargo/issues/2426
RUN touch ./src/main.rs && cargo build --release

# --

FROM debian:stretch

WORKDIR /usr/bin

COPY --from=builder /factory/target/release/stravels .

EXPOSE 8000

ENV                       \
  ROCKET_ENV=production   \
  ROCKET_PORT=8000

ENTRYPOINT [ "stravels" ]

Using touch ./src/main.rs seems to do the trick, since main.rs is the only file that is common to the empty shell and the app code, all other files are new.

Hopefully in the future Cargo will be able to use cryptographic hashes to see the content of files did actually change, but the blame can equally be placed onto Docker not changing the modification date when overwriting a file.

But most importantly:

Premature optimisation is the root of all evil.

Donald Knuth
  1. It would be possible to build Rust on top of musl and use Alpine to save even more space, but at the time of writing there is not a maintained Alpine base image for the Rust compiler. It also depends on what other base images you have in your deployment pipeline.

François Best

Freelance developer & founder

Edit this page on GitHubDiscuss on Twitter