How to set up Kubeflow for ML on K8s
In this blog we want to present a quick tutorial on how to set up the latest version of Kubeflow, the open source package for managing Machine Learning on Kubernetes. Please follow these instructions on how to get Kubeflow 1.5 started:
- Launch a cluster on EKS (we recommend 4 worker nodes with at least t3.2xlarge to give training enough juice).
- We recommend using Spectro Cloud Palette for ease of management :) Here’s a quick link on how to launch an EKS cluster on our platform. But you may do this any way you like.
- Install “kustomize” if not done already (version 3.2.0) (download link).
- have “kubectl” handy (link to install instructions).
Setting it up using command lines:
Let’s download from the source.
- Clone the manifest from the official repo and switch to v1.5 branch.
git clone https://github.com/kubeflow/manifests.git
cd manifests
git checkout v1.5-branch- Download kubeconfig from Palette and ensure the kubectl command knows where to find it (e.g.
export KUBECONFIG=<my.kubeconfig>). If you need a quick way to context switch KUBECONFIG we recommend a good trick “kctx” mentioned on our blog post. - Now it is time to apply the required resource files — there are quite a few! Please be aware that this may take up to 20 minutes because of the number of resources required.
while ! kustomize build example | kubectl apply -f -; do echo “Retrying to apply resources”; sleep 10; doneWhen this is done let’s setup port forwarding:
export NAMESPACE=istio-system && kubectl port-forward -n ${NAMESPACE} svc/istio-ingressgateway 8080:80And access Kubeflow dashboard as follows with default credentials:
URL: http://localhost:8080/
> username : user@example.com
> password: 12341234
Voila! You now have a kubeflow cluster at your disposal for your ML projects
Let’s try an example:
Notebook example
Text Classification with Movie Reviews
Using the same specifications above for the notebook server, we can run an example of NLP notebook as illustrated below
This notebook classifies movie reviews as positive or negative using the text of the review. This is an example of binary classification, an important and widely applicable kind of machine learning problem.
We’ll use the IMDB dataset that contains the text of 50,000 movie reviews from the Internet Movie Database. These are split into 25,000 reviews for training and 25,000 reviews for testing.
We download the dataset and we build a model based on a model from TensorFlow Hub called google/nnlm-en-dim50/2
Next step is the model training, depending on your server specifications, this part might take some time.
Graph of accuracy and loss over time:
As always, if you have any questions or want to discuss anything described in this tutorial, ping us at developers@spectrocloud.com.
