Install Kubernetes Dashboard, access it outside the cluster & secure it with RBAC to allow access per namespace

A detailed, step-by-step guide on setting up a bare-metal Kubernetes Dashboard, accessing the dashboard outside the cluster & then securing the dashboard using RBAC (role-based access control) so that users can access the dashboard & interact with the resources in their authorized namespaces only.

By the time you complete this article, you will…

• be able to access your dashboard outside the cluster from your host.
• have an admin user who can log in to the dashboard with full access to all the resources.
• have 2 test namespaces to test RBAC.
• have 4 Service Accounts with admin & read-only access on the above 2 namespaces (2 accounts per namespace).

Introduction

In my older articles, I had explained how to set up a bare-metal Kubernetes cluster, provision the same cluster automatically using an Ansible Playbook & use Ingress/Ingress Controller to access the deployed applications.

In this article, I will show how we can deploy a Kubernetes Dashboard on bare-metal & then access the dashboard outside the cluster. I will also demonstrate how we can set up RBAC so that users can access the dashboard using their respective tokens & interact with the resources only in their authorized namespaces.

Requirements

• A bare-metal Kubernetes cluster. Minikube can also work, except that the dashboard is already installed there. You need to disable the dashboard addon.

shashanksrivastava@MBP ~> minikube addons disable dashboard
🌑  "The 'dashboard' addon is disabled

• Firefox browser. Chrome refused to load the dashboard certificates, so I loaded the dashboard in Firefox.
• SSH access to the master node.

Steps to perform

All the commands mentioned will be executed on the master node only.

1. Deploy the Kubernetes Dashboard.

We will start with deploying the latest version (at the time of writing this article) of the Kubernetes dashboard.

[shashank@k8s-master ~]$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

The above command will deploy the dashboard in the kubernetes-dashboard namespace. It will also create a few resources like secrets, role, clusterrole, service, etc. You will need to wait until the dashboard pods are up & running.

[shashank@k8s-master ~]$ kubectl get pods --all-namespaces
NAMESPACE              NAME                                         READY   STATUS      RESTARTS   AGE
default                itcalc-648b49c4cf-5bm8h                      1/1     Running     4          14d
default                mymusicstats-66bc75d78-v7wxw                 1/1     Running     2          13d
ingress-nginx          ingress-nginx-admission-create-kr4nz         0/1     Completed   5          14d
ingress-nginx          ingress-nginx-admission-patch-n7cmw          0/1     Completed   5          14d
ingress-nginx          ingress-nginx-controller-67897c9494-2tc4t    1/1     Running     27         14d
kube-system            coredns-74ff55c5b-8cltm                      1/1     Running     5          15d
kube-system            coredns-74ff55c5b-z54gp                      1/1     Running     5          15d
kube-system            etcd-k8s-master                              1/1     Running     7          15d
kube-system            kube-apiserver-k8s-master                    1/1     Running     35         15d
kube-system            kube-controller-manager-k8s-master           1/1     Running     70         15d
kube-system            kube-flannel-ds-5gnbz                        1/1     Running     5          14d
kube-system            kube-flannel-ds-dgzt6                        1/1     Running     4          14d
kube-system            kube-proxy-745j2                             1/1     Running     7          15d
kube-system            kube-proxy-tkhht                             1/1     Running     7          15d
kube-system            kube-scheduler-k8s-master                    1/1     Running     69         15d
kubernetes-dashboard   dashboard-metrics-scraper-79c5968bdc-m2r6q   1/1     Running     0          93m
kubernetes-dashboard   kubernetes-dashboard-9f9799597-ffwqg         1/1     Running     0          7s

2. Access the Kubernetes Dashboard.

If you execute kubectl proxy command, it will let you access the dashboard at http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/. URL.

But, this dashboard can only be accessed from the server on which the command is executed. Since my cluster is running on VirtualBox VMs & I want to access the dashboard from the host (MacBook Pro), the above solution won’t work.

So, how do we access the dashboard outside the cluster? Read on.

Before we do anything else, let’s first understand the basics of a Kubernetes dashboard. The Kubernetes dashboard is deployed as an application that eventually runs on a pod (or pods) & has a service. When you hit the dashboard URL in your browser, you are basically hitting the Kubernetes service which exposes this application. The default deployment sets the service type as ClusterIP which means the dashboard can only be accessed from within the cluster.

So, if we want to access our dashboard, we need to edit the service & set its type to NodePort.

Edit the service.

[shashank@k8s-master ~]$ kubectl -n kubernetes-dashboard edit service kubernetes-dashboard

Running this command will open the default text editor. Locate the line which says type: ClusterIP & change it to NodePort. It should be most probably the third last line. Write your changes & quit out of it.

Find out the port.

To find out the port, enter the below command. Look for the NodePort section.

[shashank@k8s-master ~]$ kubectl describe services kubernetes-dashboard --namespace=kubernetes-dashboard
Name:                     kubernetes-dashboard
Namespace:                kubernetes-dashboard
Labels:                   k8s-app=kubernetes-dashboard
Annotations:              <none>
Selector:                 k8s-app=kubernetes-dashboard
Type:                     NodePort
IP Families:              <none>
IP:                       10.111.49.78
IPs:                      10.111.49.78
Port:                     <unset>  443/TCP
TargetPort:               8443/TCP
NodePort:                 <unset>  32129/TCP
Endpoints:                10.244.3.20:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Restart the pod.

Now that we’ve edited the service & found the port on which the dashboard application is running, it's time to restart the pod so that we can access the dashboard using that port. To do this, just delete the running pod.

[shashank@k8s-master ~]$ kubectl delete pod kubernetes-dashboard-9f9799597-tgnx9 -n kubernetes-dashboard
pod "kubernetes-dashboard-9f9799597-tgnx9" deleted

Replace kubernetes-dashboard-9f9799597-tgnx9 with the name of your own pod.

It should now create a new pod with a different service.

Find out the node.

Let’s see which worker node is running the dashboard pod. For this, enter the below command. The output should be similar to this (I’ve truncated the output).

[shashank@k8s-master ~]$ kubectl describe pods kubernetes-dashboard-5d9c5bb99d-2f8lm -n kubernetes-dashboard
Name:         kubernetes-dashboard-5d9c5bb99d-2f8lm
Namespace:    kubernetes-dashboard
Priority:     0
Node:         k8s-worker/10.128.0.28

Line #4 tells me that it’s running on 10.128.0.28.

Edit the hosts file (note that this step is NOT mandatory).

If you’re like me, you’d want to use the hostname instead of the IP address to access your Kubernetes dashboard. I edited my /etc/hosts file on the host running the VMs (MacBook Pro) & added the below line.

10.128.0.28 kubernetes-dashboard

It means I can enter https://kubernetes-dashboard:32129 in my browser to be able to get to the dashboard. The only problem is that Google Chrome refuses to load the certificate & hence blocks the access.

Google Chrome blocking the Kubernetes dashboard certificates.

I tried to regenerate the certificates, secrets & re-deploy the dashboard but Chrome wouldn’t accept it. So, I tried to load it in Firefox.

That’s why I mentioned it in the Requirements section. Firefox browser is required for this tutorial.

Kubernetes dashboard fully loaded in Firefox.

Now that we’re able to get to our dashboard, it’s time to Sign-in to it. But, for that, we’d require a Token. This token will allow us to log in & interact with the cluster.

Let’s see how to obtain the token.

P.S — We’re going to create a service account with full access to the cluster. This account will give it admin level access on the Kubernetes Dashboard. I’ll explain how to enable RBAC for various service accounts so that they only have access to their authorized namespaces.

3. Create a Service Account.

It will create a service account admin-user inside kubernetes-dashboard namespace.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
EOF

4. Create a ClusterRoleBinding.

It will create a Cluster Role Binding called admin-user which references a pre-defined Cluster Role called cluster-admin. cluster-admin Cluster Role has admin access by default.

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

5. Extract the token.

Now we will extract the token of the admin-user Service Account. This token will enable us to log in to our dashboard & grant us full admin access. It means admin-user can view/create/delete any resource in the cluster via Kubernetes dashboard.

[shashank@k8s-master ~]$ kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"

This command will give you a long string of text. Copy it.

6. Sign in to the Kubernetes Dashboard.

Now navigate to the URL (https://worker-node:NodePort) of your dashboard & paste the token. Hit the Sign In button & you should be able to see everything that you have in your cluster.

Kubernetes dashboard loaded in Firefox with admin user token.

Congrats! You’ve made it to this point. You now have admin access on your dashboard & you should be able to interact with the cluster resources across all the namespaces.

But, as I mentioned earlier that you simply can’t give away the token to just anyone. You’d want to keep this token with you or the admins only. We will now create a couple of Service Accounts & grant them access only to the namespaces where they belong.

To demonstrate how we can restrict Kubernetes Dashboard access to the users according to their authorized namespaces, I created 2 namespaces dev & devops and 4 Service Accounts dev-admin-user, dev-read-only-user, devops-admin-user & devops-read-only-user.

The idea here is that dev-admin-user will have full access on dev namespace & devops-admin-user will have full access on devops namespace. dev-admin-user will not be able to see or create resources in devops namespace & vice-versa. Similarly dev-read-only-user will only have read-only access on dev namespace.

7. Create Namespaces —dev & devops.

[shashank@k8s-master]$ kubectl create ns dev
[shashank@k8s-master]$ kubectl create ns devops

8. Create Service Accounts — dev-admin-user, dev-read-only-user, devops-admin-user & devops-read-only-user.

For this, issue the below commands one by one.

ServiceAccount — dev-admin-user

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dev-admin-user
  namespace: dev
EOF

ServiceAccount — dev-read-only-user

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dev-read-only-user
  namespace: dev
EOF

ServiceAccount — devops-admin-user

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: devops-admin-user
  namespace: devops
EOF

ServiceAccount — devops-read-only-user

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: devops-read-only-user
  namespace: devops
EOF

Notice the namespaces.

9. Create Roles — dev-admin, devops-admin, dev-read-only, devops-read-only.

Roles in Kubernetes define what a member of that role can do & what resources they can access. Since we want full access for dev-admin-user & devops-admin-user on dev & devops namespaces respectively, we will create 2 admin roles.

dev-admin role.

cat <<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: dev
  name: dev-admin
rules:
- apiGroups: ["", "extensions", "apps", "batch", "networking.k8s.io"]
  resources: ["*"]
  verbs: ["*"]
EOF

devops-admin role.

cat <<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: devops
  name: devops-admin
rules:
- apiGroups: ["", "extensions", "apps", "batch", "networking.k8s.io"]
  resources: ["*"]
  verbs: ["*"]
EOF

* means everything. networking.k8s.io allows access on Ingresses & Services.

Similarly, we will create a couple more roles for read-only access.

dev-read-only role.

cat <<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: dev
  name: dev-read-only
rules:
- apiGroups: ["", "extensions", "apps", "batch", "networking.k8s.io"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
EOF

devops-read-only role.

cat <<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: devops
  name: devops-read-only
rules:
- apiGroups: ["", "extensions", "apps", "batch", "networking.k8s.io"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
EOF

Notice the verbs in the last 2 roles.

10. Create RoleBindings.

Once we have Service Accounts & Roles created, we will create the RoleBindings. RoleBinding in Kubernetes assigns a pre-existing role to a given user, group, or service account.

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-admin-user
  namespace: dev
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: dev-admin
subjects:
- kind: ServiceAccount
  name: dev-admin-user
  namespace: dev
EOF

Notice the roleRef section above. Here we’re telling Kubernetes to look for a role called dev-admin & bind it to the service account called dev-admin-user inside dev namespace.

Remember we had created the Service Accounts & Roles in steps 8 & 9?

Let’s create the remaining 3 RoleBindings.

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-read-only-user
  namespace: dev
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: dev-read-only
subjects:
- kind: ServiceAccount
  name: dev-read-only-user
  namespace: dev
EOF

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: devops-admin-user
  namespace: devops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: devops-admin
subjects:
- kind: ServiceAccount
  name: devops-admin-user
  namespace: devops
EOF

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: devops-read-only-user
  namespace: devops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: devops-read-only
subjects:
- kind: ServiceAccount
  name: devops-read-only-user
  namespace: devops
EOF

11. Extract Tokens of Service Accounts.

Now that we’ve created everything that we need to implement Kubernetes Dashboard RBAC, we will now need the tokens of the respective Service Accounts for logging in to the dashboard.

[shashank@k8s-master yaml-resources]$ kubectl -n dev get secret $(kubectl -n dev get sa/dev-admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"

[shashank@k8s-master yaml-resources]$ kubectl -n dev get secret $(kubectl -n dev get sa/dev-read-only-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"

[shashank@k8s-master yaml-resources]$ kubectl -n devops get secret $(kubectl -n devops get sa/devops-admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"

[shashank@k8s-master yaml-resources]$ kubectl -n devops get secret $(kubectl -n devops get sa/devops-read-only-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"

Copy the token of dev-admin-user & log out of your dashboard if you have not already (or open a New Private Window). Now log in using this token.

It will take you to the default namespace but you will not be able to see anything as dev-admin-user has full access on dev namespace only.

Access denied for dev-admin-user.

Now change the namespace by replacing default with dev in the browser address bar — https://kubernetes-dashboard:32129/#/workloads?namespace=dev

Of course, if you haven’t deployed anything in dev namespace, you won't see anything under Deployments. But if you scroll down to Secrets, you’ll see a few secrets.

Now try switching to devops namespace by replacing dev with devops in the address bar. You won’t see any secrets (or anything) as you don’t have any access on devops namespace.

Let’s try to test our admin access in dev namespace by trying to delete a secret. Switch back to dev namespace & delete a secret.

Trying to delete a secret.

The secret should be gone.

Let’s log in using dev-read-only-user. Copy the token (obtained in step #11) & log out of your dashboard. Now log back in using this token.

On your master node, enter the following commands to create 2 text files for our secret.

echo -n 'admin' > username.txt
echo -n '1f2d1e2e67df' > password.txt

Go back to the browser & click on the + button in the right corner. In the text-box, paste the below text & hit the Upload button.

kind: Secret
apiVersion: v1
metadata:
  name: db-user-pass
  namespace: dev
  managedFields:
    - manager: kubectl-create
      operation: Update
      apiVersion: v1
      fieldsType: FieldsV1
      fieldsV1:
        f:data:
          .: {}
          f:password.txt: {}
          f:username.txt: {}
        f:type: {}
data:
  password.txt: MWYyZDFlMmU2N2Rm
  username.txt: YWRtaW4=
type: Opaque

Notice the error message below. It won’t allow you to create a secret as you only have read-only access.

Log out & log back in as dev-admin-user & repeat the same exercise. Your secret should be created now.

And that’s it. We’ve successfully deployed our Kubernetes dashboard & we’re able to access it from the host on which the cluster is running. We’ve also successfully implemented RBAC & restricted access according to namespaces.

I hope you liked this article.

Thanks for reading!