2017-11-27 - cleaning up docker-registry

it has been 2 months since last post and over 5 months since anything more verbose. Code::Dive is now over and looks like i'll have a bit of time to catch up blogging some ideas and implementations i've being doing around, but had no time to post about. today we'll start with… docker-registry.

when setting up a new service, one must typically accept a bit of maintenance as well. :) there are 3 things you need to cover on your own, that might not come very obvious, if you have not set up a docker-registry instance before, namely:

1. HTTPS access
2. authentication/authorization
3. removal of (old?) images

first one is simple – just put nginx as a reverse-proxy, to terminate HTTPS, and use HTTP internally to contact to the instance.

second one is not very complex either - for many cases it should be enough to configure the very same nginx instance you're using for HTTPS, to enforce basic-auth on each and every path to reverse-proxied service.

the third thing however is FAR from obvious. more over there is very little help on the internet. some ppl write a lot of voodoo scripts, that manipulate registry's internals – this is definitely not a thing i wanted to have on my instance. it turns out that registry's 2.0 version of API defined a REST call to untag images, BUT it was not stated clearly how to do this! more over only the most recent versions on registry (since 2.5 i think) actually have this call implemented.

but how to use it? i've spent way more time i'd like to admit, to solve this issue. if you hit this page with the same problem – how to remove an image from a private registry, here's the procedure.

first thing, that is very easy to miss, is configuration. by default docker-registry does NOT allow any explicit removals! even if you set up everything else correctly, w/o this option the feature will just not work. what you need to do is edit /etc/docker/registry/config.yml file and add the following section:

storage:
  delete:
    enabled: true

the API looks very simple – just send a DELETE request to http://myregistry:5000/v2/<image>/manifest/<blob-id>. cool – now how do i get the blob-id thing? while most API calls do accept either ID, or just a name… delete does not. id is the only way to go.

in order to get the ID, the tricky part is that special header must be set in the GET query, namely Accept: application/vnd.docker.distribution.manifest.v2+json. if you won't set it, you'll get a lot of different pieces of information, aside from the single one you're actually looking for. :/ to make things even funnier - the response also comes in a header form…

so the example call via curl, that gets the blob-id for a given image and a tag pair, can look like this:

curl -q -v \
  -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  -X GET \
  http://myregistry:5000/v2/<image>/manifests/<tag> 2>&1 | grep Docker-Content-Digest | awk '{ print $3 }'

the output will look similar to this: sha256:40506065be119bfc51093a65bf0123147cd2af72c767168dafdcb2d131f72e4d.

now that we finally have the blob-id, we can delete it, from the registry:

curl -v -X DELETE http://myregistry:5000/v2/<image>/manifests/sha256:40506065be119bfc51093a65bf0123147cd2af72c767168dafdcb2d131f72e4d

if you made it this far, you'll notice that while you no longer can docker pull the removed image, your disk space usage have not decreased either! the procedure above only removes a tag of the image, but does not remove data blobs, to free disk space!

to actually wipe the data from the disk, garbage collector must be called, inside the docker-registry image, with a configuration file as a parameter. the call looks like this:

docker exec -it my-registry-container \
  registry garbage-collect /etc/docker/registry/config.yml

…and wait. for bigger registry this can take some minutes to complete and will flood your screen with messages.

when the command will exit, your cleanup has just finished.

oh – and btw: don't run this whole procedure in parallel with adding stuff to your registry. even though it only uses the official API and provided commands, it still tends to leave registry in an inconsistent state! :/

despite the procedure being from hell, pretty much undocumented (well – there is a documentation for all of it, but left on your own it is anything but trivial to glue the above procedure up!) and problematic in terms of stability, it is not the end of the story.

when you use registry to store CI-build microservices images (which seems very natural), it is very common to have cleanup procedure of old images. explicit deleting some images is only a one step of a bigger run in there. of course getting list of images ordered by creation time is tricky as well, since tags do not have a creation date – only images have it. and all of the sudden automation of the cleanup becomes even bigger pain.

long story short – as of writing this post, the procedure is terrible and barely usable. it think there is huge opportunity for improvements here and i hope we'll see better API in the future. in particular it would be nice to have a registry-operating API embedded directly into docker client application. push and pull are there, but removal is unfortunately still missing.