I was playing around with CRDs today when I wondered “what would happen if I tried to overwrite one of Kubernetes’ core object types? The results weren’t what I expected, so thought it was worth writing it down, in case it comes as a surprise to others :) N.B. There’s an update at the bottom which sheds more light on what happens when you do this
What’s a CRD?
Kubernetes Custom Resources are a way to extend the core Kubernetes API with custom types. They’re heavily used by various programs which extend Kubernetes functionality, and a common feature of most clusters. In general, Custom Resources have their own group so, for example if you use Cilium for your kubernetes networking, you’ll have CRDs like
ciliumendpoints.cilium.io which is the ciliumendpoint object in the cilium.io group.
Overwriting core Kubernetes CRDs
So an obvious question for a curious goose is, what happens if I try to create a CRD with the
k8s.io group, which is used by a lot of core Kubernetes objects. Well, let’s find out !
If we write a basic CRD which starts like this
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: networkpolicies.networking.k8s.io
we get an error when we create it. The error handily tells us that we need an annotation added to create a CRD in this group
The CustomResourceDefinition "networkpolicies.networking.k8s.io" is invalid: metadata.annotations[api-approved.kubernetes.io]: Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see https://github.com/kubernetes/enhancements/pull/1111
There’s no validation on the contents of this annotation, so I just took the sample one from the GH issue. After I did that, it works! The first time I created a CRD overwriting the Network Policy object, I had it as a namespace scoped object and the results were unexpected.
kubectl get crds shows that it created ok (The one I created didn’t have an egress field)
kubectl get crds NAME CREATED AT networkpolicies.networking.k8s.io 2021-11-01T19:28:04Z
but if I use
kubectl explain on the object, the definition I get back is the original core object.
kubectl explain networkpolicies.networking.k8s.io.spec KIND: NetworkPolicy VERSION: networking.k8s.io/v1 RESOURCE: spec <Object> DESCRIPTION: Specification of the desired behavior for this NetworkPolicy. NetworkPolicySpec provides the specification of a NetworkPolicy FIELDS: egress <Object> List of egress rules to be applied to the selected pods. Outgoing traffic is allowed if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic matches at least one egress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy limits all outgoing traffic (and serves solely to ensure that the pods it selects are isolated by default). This field is beta-level in 1.8
Also if I try to create a sample network policy which uses the original object type, it works ok. So the CRD is there, but it’s not working.
What happens if you change the scope in the CRD from namespaced to cluster? Well now, it shows in the CRD list and it’s overwritten the object definition retrieved by
kubectl explain networkpolicies.networking.k8s.io.spec KIND: NetworkPolicy VERSION: networking.k8s.io/v1 RESOURCE: spec <Object> DESCRIPTION: <empty> FIELDS: name <string>
and if I try to create a Network Policy, I get an error showing that it doesn’t understand those field, so this has overwritten the core object.
error: error validating "sample-netpol.yaml": error validating data: [ValidationError(NetworkPolicy.spec): unknown field "egress" in io.k8s.networking.v1.NetworkPolicy.spec, ValidationError(NetworkPolicy.spec): unknown field "ingress" in io.k8s.networking.v1.NetworkPolicy.spec, ValidationError(NetworkPolicy.spec): unknown field "podSelector" in io.k8s.networking.v1.NetworkPolicy.spec, ValidationError(NetworkPolicy.spec): unknown field "policyTypes" in io.k8s.networking.v1.NetworkPolicy.spec]; if you choose to ignore these errors, turn validation off with --validate=false
There’s more prodding at this to be done, but a fun additional one I noticed is that if you try to create a CRD with a name of
users.authentication.k8s.io it creates, but the result isn’t visible in
kubectl api-resources and you can’t use
kubectl explain on it, even though it’s not overwriting a visible Object type.
From a security standpoint, the takeaway here is, don’t let anyone create CRDs in your cluster unless you really really trust them,
as they can modify Kubernetes core behaviour. Also if you’re planning to appear on Rawkode’s klustered show, you could try this and hope the cluster fixers don’t read my blog :)
Curious to get some details on was going on behind the scenes and possible security implications, I asked on k8s slack (a great source of information), and the inimitable Jordan Liggitt was able to explain it. It seems like this doesn’t actually overwrite the core Kubernetes types, but what it does do is modify OpenAPI schemes used by clients like Kubectl to validate data being passed to the API server. So, in effect this could break tooling or produce odd behaviour but isn’t modifying the core Kubernetes API. There’s an issue in the Kubernetes Github repo with more info.
To me, it’s still interesting, but in a slightly different way as it illuminates how some of the different components in Kubernetes work together and some of the possible complexity when unusual or unexpected changes are made.