Kubernetes Namespace creation. Chicken and egg.

What came first, the namespace or the permissions to use it? We can give User privileges to Create a Namespace but we can’t give them privileges to manage objects inside the Namespace if it doesn’t exist!

Like the famous “chicken and egg” dilemma creation of Namespaces in Kubernetes does not have a simple answer.

In a typical organisation, a Cluster Admin will create the Namespace and necessary RoleBindings in the Namespace for Developers and CI/CD Tools. It gives only the necessary minimal privileges to the Dev Team and puts them within the boundaries of their namespace.

This model looks good from the security perspective but it slows down CI/CD Pipelines and creates an extra dependency on Admin.

Ideally, we would like to enable Developers to:

• create/delete their own Namespaces with correct names, labels, quotas etc…
• get necessary privileges to the Namespaces they create allowing them to deploy software

and at the same time:

• not have access to System Namespaces or Namespaces of different Development Team
• not be able to compromise the entire Cluster by accident

Unfortunately, the problem can’t be solved by the Kubernetes RBAC Model. Two Cluster-level privileges are required for Development Team to become self-sufficient:

• Creation/Deletion of a namespace
• Creation of RoleBinding in a namespace

If the creation of a Namespace is relatively harmless and could be given to a Dev Team, Deletion of a Namespace or Creation of RoleBinding will compromise the Security of a Cluster since both permissions should be given cluster-wide.

Even though this is quite a common problem there is no single best-practices solution exists.

In this post, I tried to consolidate all solutions we came across.

Option 1. Administrative solution.

Technically it’s not a solution at all, but still, I’d like to mention it. Every time when Developers need a namespace they request it from DevOps via ticket, chat or email.

It is a static, but workable approach.

Option 2. Self-Service Web Portal

Building some Web Portal just for this feature might be overkill, but if the organisation already has some “self-care” service – adding an extra feature for namespace management might not be such a big task. There are also several ready tools and products offering this functionality. Products like https://www.tremolosecurity.com include namespace self-service in their portals. Some organisations open-sourced their own implementations https://github.com/loft-sh/kiosk .

Option 3. Use of NamespaceAutoProvision Admission Controller

Admission Controller is more a Kubernetes-like way of addressing the problem. Enabling a NamespaceAutoProvision AdmissionController does allow users to create resources in the non-existing namespaces. Namespaces will be created for them automatically. Unfortunately, this specific controller does not solve the original problem. Users still should have Cluster-level Roles allowing them to create Resources in any namespaces. Since we are trying to separate Cluster Admin privileges and Developer Namespaced privileges this approach doesn’t work.

Option 4. Create your own Admission Controller

Similar to NamespaceAutoProvision it is possible to create your own Admission Controller which will do the necessary work for you, such as creating RoleBindings, namespace labels etc. Even though it sounds complicated the task is quite achievable. There are very good guides available to walk you step by step through the process.

Alternatively, a similar implementation can be found, for example, https://github.com/blakelead/nsinjector

Option 5. Create Namespace management operator

Another Kubernetes-style option is the creation of an operator to manage namespaces within the cluster. The operator pattern has gotten very popular among vendors and significantly simplifies software management.

Similar to the Controllers Operator, this can be developed from scratch(there are many guides available on how to do it) or an existing solution can be used.

The example I have found here is useful: https://github.com/redhat-cop/namespace-configuration-operator#NamespaceConfig

Option 6. Self-cranked Namespace management

The last option is overarching for all other techniques of removing Cluster-level privileges from developers. In previous solutions, these Admin actions were performed by WebPortal, AdmissionControllers or Operators.

Alternatively, the same task can be delegated to CI/CD, Lambda, Azure function, Kubernetes Jobs etc… Sometimes a combination of multiple tools can be used.

For example:

• Kubernetes Job can be used to remove unused namespaces
• Lambda can create namespaces and enforce all necessary resources. This can be useful if along with the namespace some additional resources are required outside the Cluster
• CI/CD can temporary escalate privileges to do the necessary cluster-wide operations before deployment

and so on…