macOS Docker Volume Performance

Introduction

With widespread macOS adoption for developers at reecetech, it became important for us to solve a well-known issue with Docker bind volumes performance when using Docker for Mac.

Docker for Mac uses a Linux VM to perform Docker operations, acting as a shim between Docker and the macOS kernel. This raises an issue with performance when trying to bind mount a volume to Docker from the local Mac environment.

File system performance for bind mounts is poor when running Docker for Mac. This can be seen with a compile of a program locally compared with the server-side build time. The build process loops over many files to build out the required methods to inject into the compiled binary. This process is over 10x slower when running local with Docker when compared to running on one of our build servers.

Even with more CPU cores on local Mac with faster clock speeds, the speed to compile is much slower locally when compared to the build sever. This is a pain for us as we try to empower our developers to get fast feedback locally and rely less on specific infrastructure dependencies when building their code.

Comparing build times

Using the basic time command to compare compilation times shows the discrepancy clearly.

Server-side build

real	0m35.25s
user	0m20.28s
sys	0m10.35s

Local Mac build

real	6m9.693s
user	0m20.390s
sys	0m41.267s

Complaints started to come back to my team after we tried to push local Docker compiling for the development teams. The complaints were understandable, we were trying to decrease the time developers spent trying to build code, not make it a lot worse.

Attempts to solve the performance issues

Searching around I was able to find some information to change the type of mount Docker uses to speed up performance by limiting the synchronisation with the local OS.

This would be a risk of data loss compared to speed improvements. Our build process would not suffer from issues with data synchronisation, so I believed this path would be a simple fix to the performance issue.

Docker Docs had a whole section dedicated to this topic due to the pervasive nature of the issues seen on macOS.

Volume types

Eagerly reading through the documentation to figure out where I should spend my time, it became obvious that changing volume type was very easy. I decided to try all types getting results I thought I would be happy with.

Bind mount volumes in Docker can be broken down into three types:

  1. Consistent (default)
  2. Cached
  3. Delegated

Each type lowers the consistency with the local operating systems respectively.

Consistent (default)

The consistent configuration Forces the container volume to be synchronised with the local OS directory, lowering performance to increase data protection. While running Docker on Linux I had never changed this value, Linux using native hooks to handle volumes means this type is rarely deviated from when running native Linux.

Cached

The cached configuration as the name suggests keeps data read from the OS volume in a cache to be used on repeat operations. This one seemed good for our use case as we loop over the same files for the build process.

Delegated

The delegated configuration as its name suggests delegates authorisation of the writes to the container view, meaning writes are not written to OS disk immediately and losing the container could result in data loss.

Testing volume types

With the learning from the Docker documentation, I started working through the different volume types hoping to see some healthy performance gains.

Starting with delegated which seemed the most promising, the times did improve, but nowhere near where we needed to be in terms of performance.

docker run -i -t -v $(pwd):/build:delegated builder

real	5m15.195s
user	0m18.804s
sys	0m34.197s

Next up was to try the cached type to see the performance benefits. Once again the performance was improved on the default type, but not where we wanted it, and slower than the delegated volume type. I tried a second run to see if the caching performance would speed up the reads but the results did not really change.

Below shows time taken for compile with two runs for the same program.

docker run -i -t -v $(pwd):/build:cached builder

real	5m56.335s
user	0m20.621s
sys	0m39.985s

real	5m55.458s
user	0m20.949s
sys	0m44.650s

Alternative solution

The results of changing volume type were disappointing, I was hoping for a simple change to the instructions and a happy developer community. With some hopeful searching I came across EugenMayer's docker-sync project.

Docker-sync is a neat solution to this frustrating problem, simply using a Docker container running on the local macOS instance which syncs data from a Docker named volume to the local OS directory.

A named volume in Docker lives inside the Linux VM running docker and has fast filesystem access speeds. Docker sync provides a simple listener for the macOS filesystem to pick up changes and sync them over to the named volume, providing a significant performance boast.

Installing docker-sync

Getting docker-sync running is fairly simple. The below lists how to install docker-sync locally on macOS and get a local directory mounted up as a named volume to use with Docker containers.

Install

Installing docker-sync is done as a ruby gem and uses a few macOS tricks to sync data. The below shows the installation steps to get basic dependencies required.

#> gem install --user-install docker-sync

#> brew install unison

#> brew install eugenmayer/dockersync/unox

Allow docker-sync to run

After installation of packages, you will need to add below to .bashrc in your local home directory then source to have applicable in current terminal session

#> vim ~/.bashrc

if which ruby >/dev/null && which gem >/dev/null; then
  PATH="$(ruby -r rubygems -e 'puts Gem.user_dir')/bin:$PATH"
fi

#> source ~/.bashrc

Running docker-sync

Docker sync requires a valid configuration file (docker-sync.yaml), the below file creates a named volume for Docker called osx-sync and mounts the local macOS directory ~/code-repo/ into the docker-sync running container.

Another important note is the unison sync_strategy used in the example, I have found this works better for Docker Desktop than the default macOS strategy.

#> cat docker-sync.yaml
version: "2"

options:
  verbose: true
syncs:
  osx-sync: # tip: add -sync and you keep consistent names as a convention
    src: '~/code-repo/'
    sync_strategy: 'unison'
    sync_excludes: ['ignored_folder', '.ignored_dot_folder']

Starting docker-sync with the configuration is done with the below command from the same directory the configuration is in, you can pass in a relative or absolute path for the config file.

#> docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

#> docker-sync start -c docker-sync.yaml

#> docker ps
CONTAINER ID   IMAGE                                   COMMAND                  CREATED              STATUS              PORTS                       NAMES
649ad0dfef6a   eugenmayer/unison:2.51.3-4.12.0-AMD64   "/entrypoint.sh supeā€¦"   About a minute ago   Up About a minute   127.0.0.1:63627->5000/tcp   osx-sync

#> docker volume ls | grep osx
local     osx-sync 

Using docker-sync volume

The next step is to use the volume by invoking with a docker run command as shown below

#> docker run -i -t -v osx-sync:/build builder

[user@30d92933]# ls /build/
code build-data other-files

docker-sync results

Using the docker-sync container to bridge the local macOS directory with a named volume in Docker the results were excellent. Building local using docker-sync performs better than on the build server.

real	0m33.297s
user	0m16.771s
sys	0m15.234s

Conclusion

Working with bind mounts in Docker on macOS leads to poor performance of disk operations. This is due to Docker for Mac running inside a Linux VM inside the macOS Kernel.

Docker provides some options to improve performance using Docker volume types, but these did not make a big enough impact for the workloads we were trying to run in Docker.

Using the simple third-party tool docker-sync I was able to get speeds I wanted with little overhead of additional complexity.

Some other projects have taken on this challenge to improve Docker volumes in macOS. I am happy to promote docker-sync as it works well, seems to be maintained and have low complexity. You may find a better experience looking into mutagen as an alternative.

Useful resources

https://docker-docs.netlify.app/docker-for-mac/osxfs-caching/#semantics https://github.com/EugenMayer/docker-sync https://mutagen.io/documentation/transports/docker