on
macOS Docker Volume Performance
Introduction
With widespread macOS adoption for developers at reecetech, it became important for us to solve a well-known issue with Docker bind volumes performance when using Docker for Mac.
Docker for Mac uses a Linux VM to perform Docker operations, acting as a shim between Docker and the macOS kernel. This raises an issue with performance when trying to bind mount a volume to Docker from the local Mac environment.
File system performance for bind mounts is poor when running Docker for Mac. This can be seen with a compile of a program locally compared with the server-side build time. The build process loops over many files to build out the required methods to inject into the compiled binary. This process is over 10x slower when running local with Docker when compared to running on one of our build servers.
Even with more CPU cores on local Mac with faster clock speeds, the speed to compile is much slower locally when compared to the build sever. This is a pain for us as we try to empower our developers to get fast feedback locally and rely less on specific infrastructure dependencies when building their code.
Comparing build times
Using the basic time
command to compare compilation times shows the discrepancy clearly.
Server-side build
real 0m35.25s
user 0m20.28s
sys 0m10.35s
Local Mac build
real 6m9.693s
user 0m20.390s
sys 0m41.267s
Complaints started to come back to my team after we tried to push local Docker compiling for the development teams. The complaints were understandable, we were trying to decrease
the time developers spent trying to build code, not make it a lot worse.
Attempts to solve the performance issues
Searching around I was able to find some information to change the type of mount Docker uses to speed up performance by limiting the synchronisation with the local OS.
This would be a risk of data loss compared to speed improvements. Our build process would not suffer from issues with data synchronisation, so I believed this path would be a simple fix to the performance issue.
Docker Docs had a whole section dedicated to this topic due to the pervasive nature of the issues seen on macOS.
Volume types
Eagerly reading through the documentation to figure out where I should spend my time, it became obvious that changing volume type was very easy. I decided to try all types getting results I thought I would be happy with.
Bind mount volumes in Docker can be broken down into three types:
- Consistent (default)
- Cached
- Delegated
Each type lowers the consistency with the local operating systems respectively.
Consistent (default)
The consistent
configuration Forces the container volume to be synchronised with the local OS directory, lowering performance to increase data protection. While running Docker on Linux I had never changed this value, Linux using native hooks to handle volumes means this type is rarely deviated from when running native Linux.
Cached
The cached
configuration as the name suggests keeps data read
from the OS volume in a cache to be used on repeat operations. This one seemed good for our use case as we loop over the same files for the build process.
Delegated
The delegated
configuration as its name suggests delegates authorisation of the writes to the container view, meaning writes are not written to OS disk immediately and losing the container could result in data loss.
Testing volume types
With the learning from the Docker documentation, I started working through the different volume types hoping to see some healthy performance gains.
Starting with delegated
which seemed the most promising, the times did improve, but nowhere near where we needed to be in terms of performance.
docker run -i -t -v $(pwd):/build:delegated builder
real 5m15.195s
user 0m18.804s
sys 0m34.197s
Next up was to try the cached
type to see the performance benefits. Once again the performance was improved on the default
type, but not where we wanted it, and slower than the delegated
volume type. I tried a second run to see if the caching performance would speed up the reads but the results did not really change.
Below shows time taken for compile with two runs for the same program.
docker run -i -t -v $(pwd):/build:cached builder
real 5m56.335s
user 0m20.621s
sys 0m39.985s
real 5m55.458s
user 0m20.949s
sys 0m44.650s
Alternative solution
The results of changing volume type were disappointing, I was hoping for a simple change to the instructions and a happy developer community. With some hopeful searching I came across EugenMayer's
docker-sync project.
Docker-sync is a neat solution to this frustrating problem, simply using a Docker container running on the local macOS instance which syncs data from a Docker named
volume to the local OS directory.
A named volume in Docker lives inside the Linux VM running docker and has fast filesystem access speeds. Docker sync provides a simple listener for the macOS filesystem to pick up changes and sync them over to the named volume, providing a significant performance boast.
Installing docker-sync
Getting docker-sync running is fairly simple. The below lists how to install docker-sync locally on macOS and get a local directory mounted up as a named volume to use with Docker containers.
Install
Installing docker-sync is done as a ruby gem and uses a few macOS tricks to sync data. The below shows the installation steps to get basic dependencies required.
#> gem install --user-install docker-sync
#> brew install unison
#> brew install eugenmayer/dockersync/unox
Allow docker-sync to run
After installation of packages, you will need to add below to .bashrc in your local home directory then source to have applicable in current terminal session
#> vim ~/.bashrc
if which ruby >/dev/null && which gem >/dev/null; then
PATH="$(ruby -r rubygems -e 'puts Gem.user_dir')/bin:$PATH"
fi
#> source ~/.bashrc
Running docker-sync
Docker sync requires a valid configuration file (docker-sync.yaml), the below file creates a named volume for Docker called osx-sync and mounts the local macOS directory ~/code-repo/
into the docker-sync
running container.
Another important note is the unison sync_strategy used in the example, I have found this works better for Docker Desktop than the default macOS strategy.
#> cat docker-sync.yaml
version: "2"
options:
verbose: true
syncs:
osx-sync: # tip: add -sync and you keep consistent names as a convention
src: '~/code-repo/'
sync_strategy: 'unison'
sync_excludes: ['ignored_folder', '.ignored_dot_folder']
Starting docker-sync
with the configuration is done with the below command from the same directory the configuration is in, you can pass in a relative or absolute path for the config file.
#> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
#> docker-sync start -c docker-sync.yaml
#> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
649ad0dfef6a eugenmayer/unison:2.51.3-4.12.0-AMD64 "/entrypoint.sh supeā¦" About a minute ago Up About a minute 127.0.0.1:63627->5000/tcp osx-sync
#> docker volume ls | grep osx
local osx-sync
Using docker-sync volume
The next step is to use the volume by invoking with a docker run command as shown below
#> docker run -i -t -v osx-sync:/build builder
[user@30d92933]# ls /build/
code build-data other-files
docker-sync results
Using the docker-sync container to bridge the local macOS directory with a named volume
in Docker the results were excellent. Building local using docker-sync
performs better than on the build server.
real 0m33.297s
user 0m16.771s
sys 0m15.234s
Conclusion
Working with bind mounts in Docker on macOS leads to poor performance of disk operations. This is due to Docker for Mac running inside a Linux VM inside the macOS Kernel.
Docker provides some options to improve performance using Docker volume types, but these did not make a big enough impact for the workloads we were trying to run in Docker.
Using the simple third-party tool docker-sync
I was able to get speeds I wanted with little overhead of additional complexity.
Some other projects have taken on this challenge to improve Docker volumes in macOS. I am happy to promote docker-sync
as it works well, seems to be maintained and have low complexity. You may find a better experience looking into mutagen as an alternative.
Useful resources
https://docker-docs.netlify.app/docker-for-mac/osxfs-caching/#semantics https://github.com/EugenMayer/docker-sync https://mutagen.io/documentation/transports/docker