Kustomize your way to MongoDB ReplicaSet

A replica set in MongoDB is a group of mongod processes that maintain the same data set. Replica sets provide redundancy and high availability and are the basis for production deployments.

In this post we will set up a MongoDB replica set with the abilities to be a production-ready environment. We are going to use a Kubernetes cluster, using kustomize to set up the whole system.

There are some capabilities that a Kubernetes system provides us; the biggest one is to maintain HA. If any pod/statefulset goes down Kubernetes will try to bring it up again, maintaining the number of replicas in the system that are configured by the user. Hence, while setting up the MongoDB replica set we also want to use the capabilities of Kubernetes to maintain high availability.

We want a capability to autoscale MongoDB ReplicaSet with the number of replicas of MongoDB StatefulSet. In order to do so we will set up a sidecar that will keep on monitoring the pod (replicas). If any new MongoDB StatefulSet’s replica comes up the sidecar will add it to the MongoDB ReplicaSet, and if the MongoDB StatefulSet replica scales down it will remove it from the existing MongoDB ReplicaSet. The code for sidecar you can check out on GitHub.

In our example we want our MongoDB to be set up with security features, so we will run MongoDB in a secure mode. We will make the communication between the mongo nodes secure with a mongo key; additionally we will start mongo with the “auth” option and will configure an “admin” user. We will run some scripts to set up the system according to our application needs and those scripts run only one time in a lifespan of MongoDB (only at start), with the help of docker-entrypoint.sh provided by MongoDB's docker image. We will load all those scripts into the filesystem with the help of ConfigMaps. We will supply passwords via K8s Secrets. Moreover, we don’t want to maintain a local copy of the image. Instead, we will be using the mongo image from hub.docker.com so that we can easily update mongo to the latest version.

Requirements

  1. A K8s cluster.

  2. Kustomize Installed.

  3. Internet Accessibility.

We will be using the docker image provided by MongoDB and a sidecar image built from mongodb-k8s-sidecar.

Let's get started

I am using a GKE cluster but you can run the same on your local Kubernetes cluster too.

The MongoDB docker image has the ability to configure some scripts to run only when the first time MongoDB is started. We will take benefit of this feature and will set a root user password and will create one more user just to demo that we can design the system in such a way that all those scripts run only once in their life span.

Then we will make communication between the nodes of the replica set secure with a mongo key. This too we will demo in a way where you don't need anything except a YAML file.

Create Scripts and Secrets

First, let's create a script that we want to run the first time. We will load the scripts using configMap.

  apiVersion: v1
  kind: ConfigMap
  metadata:
  name: mongo-init
  data:
  mongo-user.sh: |
    mongo admin -u ${MONGO_INITDB_ROOT_USERNAME} -p ${MONGO_INITDB_ROOT_PASSWORD} <<EOF
        use unatnahs_db
        db.createUser({user: "unatanhs", pwd: "${SECOND_USER_DB_PASSWORD}", roles: [
            { role: "readWrite", db: "unatnahs_db" }
        ]});
    EOF

All these env variables we will fill with the help of mongo secrets:

 apiVersion: v1
 kind: Secret
 type: Opaque
 metadata:
   name: mongosecret
 data:
   mongoRootPassword: c2hhbnRhbnViYW5zYWw=
   unatnahsDbPassword: Y2hhbmdldGhlc2VwYXNzd29yZHM=

“mongoRootPassword” will be the password for the admin user named “root” and “unatnahsDbPassword” will be the password for the “unatnahs” user mentioned in the mongo-user.sh script in configMap.

Now we will create a mongo-key which will help to secure inter-node communication:

 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: mongo-key

 data:
   mongo.key: |
     ahaksdnqsakdqnajhvckqaafnxasxaxaxmaskdadadsasfsdsdfsf
     schcacnctcacncuadasdadadfbsasddfbadadwsioweewvaas
     dfasasakjsvnaa

Change Script Permissions

When the Mongo runs we need to set specific permissions for mongo-key. Since we will be loading everything into the filesystem of the mongo container with the help of ConfigMap we may face an issue for the permissions as the mongo pod will start with the “mongo” user, not the “root” user. We first have to change permissions of the scripts on the fly and then start Mongo. To do that we will temporarily load the configMap of mongo-key into a temp location and then we will copy to a more obvious location. The reason for doing this is that the Container loads configMap as a symlink in the filesystem and that will not allow us to change the user and permission of the script. So first we will copy the file to another location and then we will change the user and the permission of the file. To do so, we have another script.

  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: mongo-scripts
  data:
    mongo-data-dir-permission.sh: |
      chown -R mongodb:mongodb ${MONGO_DATA_DIR}
      cp -r /var/lib/mongoKeyTemp /var/lib/mongoKey
      chown -R mongodb:mongodb /var/lib/mongoKey
      chmod 400 /var/lib/mongoKey/mongo.key
      chown -R mongodb:mongodb /var/lib/mongoKey/mongo.key

Create K8s Service

Now we will create a Service which should not load balance.

  apiVersion: v1
  kind: Service
  metadata:
    name: mongo
    labels:
      name: mongo
  spec:
    ports:
      - port: 27017
        targetPort: 27017
    clusterIP: None
    selector:
      role: mongo

Create Service Accounts

Now we will create a service account and cluster role binding. Our sidecar needs permission to watch the pod. You can control the permission. We need just a watch and list permission but the following gives a lot more permissions on the system.

  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: mongo-account
    namespace: mongodb-repl-system
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: mongo-role
  rules:
  - apiGroups: ["*"]
    resources: ["configmaps"]
    verbs: ["*"]
  - apiGroups: ["*"]
    resources: ["deployments"]
    verbs: ["list", "watch"]
  - apiGroups: ["*"]
    resources: ["services"]
    verbs: ["*"]
  - apiGroups: ["*"]
    resources: ["pods"]
    verbs: ["get","list", "watch"]
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: mongo_role_binding
  subjects:
  - kind: ServiceAccount
    name: mongo-account
    namespace: mongodb-repl-system
  roleRef:
    kind: ClusterRole
    name: mongo-role
    apiGroup: rbac.authorization.k8s.io

MongoDB StatefulSet

Now let's just go to the final state and create a StatefulSet with two containers - one of the actual mongo and the other one as the sidecar.

  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: mongo
  spec:
    podManagementPolicy: Parallel
    replicas: 1
    selector:
      matchLabels:
        role: mongo
    serviceName: mongo
    template:
      metadata:
        labels:
          role: mongo
      spec:
        serviceAccountName: mongo-account
        terminationGracePeriodSeconds: 30
        containers:
          - image: mongo:4.2
            name: mongo
            command: ["/bin/sh","-c"]
            args: ["/home/mongodb/mongo-data-dir-permission.sh && docker-entrypoint.sh mongod --replSet=rs0 --dbpath=/var/lib/mongodb --bind_ip=0.0.0.0 --keyFile=/var/lib/mongoKey/mongo.key"]
            env:
            - name: MONGO_INITDB_ROOT_USERNAME
              value: root
            - name: MONGO_DATA_DIR
              value: /var/lib/mongodb
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongosecret
                  key: mongoRootPassword
            - name: SECOND_USER_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongosecret
                  key: unatnahsDbPassword
            ports:
              - containerPort: 27017
            volumeMounts:
              - mountPath: /var/lib/mongodb
                name: mongo-data
              - name: mongoinit
                mountPath: /docker-entrypoint-initdb.d
              - name: mongopost
                mountPath: /home/mongodb
              - name: mongokey
                mountPath: /var/lib/mongoKeyTemp
          - name: mongo-sidecar
            image: cvallance/mongo-k8s-sidecar:latest
            env:
              - name: MONGO_SIDECAR_POD_LABELS
                value: "role=mongo"
              - name: KUBE_NAMESPACE
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
              - name: KUBERNETES_MONGO_SERVICE_NAME
                value: mongo
              - name: MONGODB_USERNAME
                value: root
              - name: MONGODB_DATABASE
                value: admin
              - name: MONGODB_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: mongosecret
                    key: mongoRootPassword
        volumes:
          - name: "mongoinit"
            configMap:
              name: "mongo-init"
              defaultMode: 0755
          - name: "mongopost"
            configMap:
              name: "mongo-scripts"
              defaultMode: 0755
          - name: "mongokey"
            configMap:
              name: "mongo-key"
              defaultMode: 0755


    volumeClaimTemplates:
      - metadata:
          name: mongo-data
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 20Gi

In this StatefulSet we have loaded all the ConfigMap as required on specific locations. The StatefulSet first will run the permission change script, then the actual mongo script with required arguments.

VOILA!!!

We have set up a complete mongo replica set! Now you can just increase or decrease the ReplicaSet count with a simple:

  kubectl scale --replicas=3 statefulset mongo -n mongodb-repl-system

Now you can connect to the mongo with the user mentioned above on:

  mongo-0.mongo, mongo-1.mongo,mongo-2.mongo

Port forward to access mongo on local:

  kubectl -n mongodb-repl-system port-forward svc/mongo 27017

Conclusion

We set up a secure MongoDB replica set on the K8s cluster using kustomize with the help of mongo’s own docker image.

PS: You can check out the code on GitHub. There either you can use a single manifest file or the properly segregated files for better readability.