Published
April 22, 2022

How to set up Kubeflow for ML on K8s

Spectro Cloud Team
Spectro Cloud Team
Spectro Cloud

In this blog we want to present a quick tutorial on how to set up the latest version of Kubeflow, the open source package for managing Machine Learning on Kubernetes. Please follow these instructions on how to get Kubeflow 1.5 started:

  1. Launch a cluster on EKS (we recommend 4 worker nodes with at least t3.2xlarge to give training enough juice).
  2. We recommend using Spectro Cloud Palette for ease of management :) Here’s a quick link on how to launch an EKS cluster on our platform. But you may do this any way you like.
  3. Install “kustomize” if not done already (version 3.2.0) (download link).
  4. have “kubectl” handy (link to install instructions).

Setting it up using command lines:

Let’s download from the source.

  1. Clone the manifest from the official repo and switch to v1.5 branch.
 
git clone https://github.com/kubeflow/manifests.git
cd manifests
git checkout v1.5-branch
	

     2.Download kubeconfig from Palette and ensure the kubectl command knows where to find it e.g.

 
export KUBECONFIG=
	

     If you need a quick way to context switch KUBECONFIG we recommend a good trick “kctx” mentioned on our blog post.

     3.Now it is time to apply the required resource files — there are quite a few! Please be aware that this may take up to 20 minutes because of the number of resources       required.

 
while ! kustomize build example | kubectl apply -f -; do echo “Retrying to apply resources”; sleep 10; done
	

When this is done let’s setup port forwarding:

 
export NAMESPACE=istio-system && kubectl port-forward -n ${NAMESPACE} svc/istio-ingressgateway 8080:80
	

And access Kubeflow dashboard as follows with default credentials:

 URL: http://localhost:8080/
> username : user@example.com
> password: 12341234

Voila! You now have a kubeflow cluster at your disposal for your ML projects

Let’s try an example:

Notebook example

Text Classification with Movie Reviews

Using the same specifications above for the notebook server, we can run an example of NLP notebook as illustrated below

This notebook classifies movie reviews as positive or negative using the text of the review. This is an example of binary classification, an important and widely applicable kind of machine learning problem.

We’ll use the IMDB dataset that contains the text of 50,000 movie reviews from the Internet Movie Database. These are split into 25,000 reviews for training and 25,000 reviews for testing.

text classification with move reviews

We download the dataset and we build a model based on a model from TensorFlow Hub called google/nnlm-en-dim50/2

download dataset and build a model based on a model from TensorFlow Hub

Next step is the model training, depending on your server specifications, this part might take some time.

model training, depending on your server specifications

Graph of accuracy and loss over time:

Graph of accuracy and loss over time

As always, if you have any questions or want to discuss anything described in this tutorial, ping us at developers@spectrocloud.com.

Tags:
How to
Concepts
Subscribe to our newsletter
By signing up, you agree with our Terms of Service and our Privacy Policy