Using Kustomize to Kubernetes your way to MongoDB ReplicaSet
Kustomize your way to MongoDB ReplicaSet
A replica set in MongoDB is a group of mongod processes that maintain the same data set. Replica sets provide redundancy and high availability and are the basis for production deployments.
In this post we will set up a MongoDB replica set with the abilities to be a production-ready environment. We are going to use a Kubernetes cluster, using kustomize to set up the whole system.
There are some capabilities that a Kubernetes system provides us; the biggest one is to maintain HA. If any pod/statefulset goes down Kubernetes will try to bring it up again, maintaining the number of replicas in the system that are configured by the user. Hence, while setting up the MongoDB replica set we also want to use the capabilities of Kubernetes to maintain high availability.
We want a capability to autoscale MongoDB ReplicaSet with the number of replicas of MongoDB StatefulSet. In order to do so we will set up a sidecar that will keep on monitoring the pod (replicas). If any new MongoDB StatefulSet’s replica comes up the sidecar will add it to the MongoDB ReplicaSet, and if the MongoDB StatefulSet replica scales down it will remove it from the existing MongoDB ReplicaSet. The code for sidecar you can check out on GitHub.
In our example we want our MongoDB to be set up with security features, so we will run MongoDB in a secure mode. We will make the communication between the mongo nodes secure with a mongo key; additionally we will start mongo with the “auth” option and will configure an “admin” user. We will run some scripts to set up the system according to our application needs and those scripts run only one time in a lifespan of MongoDB (only at start), with the help of docker-entrypoint.sh provided by MongoDB's docker image. We will load all those scripts into the filesystem with the help of ConfigMaps. We will supply passwords via K8s Secrets. Moreover, we don’t want to maintain a local copy of the image. Instead, we will be using the mongo image from hub.docker.com so that we can easily update mongo to the latest version.
Requirements
A K8s cluster.
Kustomize Installed.
Internet Accessibility.
We will be using the docker image provided by MongoDB and a sidecar image built from mongodb-k8s-sidecar.
Let's get started
I am using a GKE cluster but you can run the same on your local Kubernetes cluster too.
The MongoDB docker image has the ability to configure some scripts to run only when the first time MongoDB is started. We will take benefit of this feature and will set a root user password and will create one more user just to demo that we can design the system in such a way that all those scripts run only once in their life span.
Then we will make communication between the nodes of the replica set secure with a mongo key. This too we will demo in a way where you don't need anything except a YAML file.
Create Scripts and Secrets
First, let's create a script that we want to run the first time. We will load the scripts using configMap.
apiVersion: v1
kind: ConfigMap
metadata:
name: mongo-init
data:
mongo-user.sh: |
mongo admin -u ${MONGO_INITDB_ROOT_USERNAME} -p ${MONGO_INITDB_ROOT_PASSWORD} <<EOF
use unatnahs_db
db.createUser({user: "unatanhs", pwd: "${SECOND_USER_DB_PASSWORD}", roles: [
{ role: "readWrite", db: "unatnahs_db" }
]});
EOFAll these env variables we will fill with the help of mongo secrets:
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: mongosecret
data:
mongoRootPassword: c2hhbnRhbnViYW5zYWw=
unatnahsDbPassword: Y2hhbmdldGhlc2VwYXNzd29yZHM=“mongoRootPassword” will be the password for the admin user named “root” and “unatnahsDbPassword” will be the password for the “unatnahs” user mentioned in the mongo-user.sh script in configMap.
Now we will create a mongo-key which will help to secure inter-node communication:
apiVersion: v1
kind: ConfigMap
metadata:
name: mongo-key
data:
mongo.key: |
ahaksdnqsakdqnajhvckqaafnxasxaxaxmaskdadadsasfsdsdfsf
schcacnctcacncuadasdadadfbsasddfbadadwsioweewvaas
dfasasakjsvnaaChange Script Permissions
When the Mongo runs we need to set specific permissions for mongo-key. Since we will be loading everything into the filesystem of the mongo container with the help of ConfigMap we may face an issue for the permissions as the mongo pod will start with the “mongo” user, not the “root” user. We first have to change permissions of the scripts on the fly and then start Mongo. To do that we will temporarily load the configMap of mongo-key into a temp location and then we will copy to a more obvious location. The reason for doing this is that the Container loads configMap as a symlink in the filesystem and that will not allow us to change the user and permission of the script. So first we will copy the file to another location and then we will change the user and the permission of the file. To do so, we have another script.
apiVersion: v1
kind: ConfigMap
metadata:
name: mongo-scripts
data:
mongo-data-dir-permission.sh: |
chown -R mongodb:mongodb ${MONGO_DATA_DIR}
cp -r /var/lib/mongoKeyTemp /var/lib/mongoKey
chown -R mongodb:mongodb /var/lib/mongoKey
chmod 400 /var/lib/mongoKey/mongo.key
chown -R mongodb:mongodb /var/lib/mongoKey/mongo.keyCreate K8s Service
Now we will create a Service which should not load balance.
apiVersion: v1
kind: Service
metadata:
name: mongo
labels:
name: mongo
spec:
ports:
- port: 27017
targetPort: 27017
clusterIP: None
selector:
role: mongoCreate Service Accounts
Now we will create a service account and cluster role binding. Our sidecar needs permission to watch the pod. You can control the permission. We need just a watch and list permission but the following gives a lot more permissions on the system.
apiVersion: v1
kind: ServiceAccount
metadata:
name: mongo-account
namespace: mongodb-repl-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: mongo-role
rules:
- apiGroups: ["*"]
resources: ["configmaps"]
verbs: ["*"]
- apiGroups: ["*"]
resources: ["deployments"]
verbs: ["list", "watch"]
- apiGroups: ["*"]
resources: ["services"]
verbs: ["*"]
- apiGroups: ["*"]
resources: ["pods"]
verbs: ["get","list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: mongo_role_binding
subjects:
- kind: ServiceAccount
name: mongo-account
namespace: mongodb-repl-system
roleRef:
kind: ClusterRole
name: mongo-role
apiGroup: rbac.authorization.k8s.ioMongoDB StatefulSet
Now let's just go to the final state and create a StatefulSet with two containers - one of the actual mongo and the other one as the sidecar.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
spec:
podManagementPolicy: Parallel
replicas: 1
selector:
matchLabels:
role: mongo
serviceName: mongo
template:
metadata:
labels:
role: mongo
spec:
serviceAccountName: mongo-account
terminationGracePeriodSeconds: 30
containers:
- image: mongo:4.2
name: mongo
command: ["/bin/sh", "-c"]
args:
[
"/home/mongodb/mongo-data-dir-permission.sh && docker-entrypoint.sh mongod --replSet=rs0 --dbpath=/var/lib/mongodb --bind_ip=0.0.0.0 --keyFile=/var/lib/mongoKey/mongo.key",
]
env:
- name: MONGO_INITDB_ROOT_USERNAME
value: root
- name: MONGO_DATA_DIR
value: /var/lib/mongodb
- name: MONGO_INITDB_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mongosecret
key: mongoRootPassword
- name: SECOND_USER_DB_PASSWORD
valueFrom:
secretKeyRef:
name: mongosecret
key: unatnahsDbPassword
ports:
- containerPort: 27017
volumeMounts:
- mountPath: /var/lib/mongodb
name: mongo-data
- name: mongoinit
mountPath: /docker-entrypoint-initdb.d
- name: mongopost
mountPath: /home/mongodb
- name: mongokey
mountPath: /var/lib/mongoKeyTemp
- name: mongo-sidecar
image: cvallance/mongo-k8s-sidecar:latest
env:
- name: MONGO_SIDECAR_POD_LABELS
value: "role=mongo"
- name: KUBE_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KUBERNETES_MONGO_SERVICE_NAME
value: mongo
- name: MONGODB_USERNAME
value: root
- name: MONGODB_DATABASE
value: admin
- name: MONGODB_PASSWORD
valueFrom:
secretKeyRef:
name: mongosecret
key: mongoRootPassword
volumes:
- name: "mongoinit"
configMap:
name: "mongo-init"
defaultMode: 0755
- name: "mongopost"
configMap:
name: "mongo-scripts"
defaultMode: 0755
- name: "mongokey"
configMap:
name: "mongo-key"
defaultMode: 0755
volumeClaimTemplates:
- metadata:
name: mongo-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20GiIn this StatefulSet we have loaded all the ConfigMap as required on specific locations. The StatefulSet first will run the permission change script, then the actual mongo script with required arguments.
VOILA!!!
We have set up a complete mongo replica set! Now you can just increase or decrease the ReplicaSet count with a simple:
kubectl scale --replicas=3 statefulset mongo -n mongodb-repl-system
Now you can connect to the mongo with the user mentioned above on:
mongo-0.mongo, mongo-1.mongo,mongo-2.mongoPort forward to access mongo on local:
kubectl -n mongodb-repl-system port-forward svc/mongo 27017Conclusion
We set up a secure MongoDB replica set on the K8s cluster using kustomize with the help of mongo’s own docker image.
PS: You can check out the code on GitHub. There either you can use a single manifest file or the properly segregated files for better readability.
