Published 2022-04-22

How to set up Kubeflow for ML on K8s

Director of Product Management @ Spectro Cloud

In this blog we want to present a quick tutorial on how to set up the latest version of Kubeflow, the open source package for managing Machine Learning on Kubernetes. Please follow these instructions on how to get Kubeflow 1.5 started:

  1. Launch a cluster on EKS (we recommend 4 worker nodes with at least t3.2xlarge to give training enough juice).
  2. Install “kustomize” if not done already (version 3.2.0) (download link).
  3. have “kubectl” handy (link to install instructions).

Setting it up using command lines:

Let’s download from the source.

  1. Clone the manifest from the official repo and switch to v1.5 branch.
git clone https://github.com/kubeflow/manifests.git cd manifests git checkout v1.5-branch
  1. Download kubeconfig from Palette and ensure the kubectl command knows where to find it (e.g. export KUBECONFIG=<my.kubeconfig>). If you need a quick way to context switch KUBECONFIG we recommend a good trick “kctx” mentioned on our blog post.
  2. Now it is time to apply the required resource files — there are quite a few! Please be aware that this may take up to 20 minutes because of the number of resources required.
while ! kustomize build example | kubectl apply -f -; do echo “Retrying to apply resources”; sleep 10; done

When this is done let’s setup port forwarding:

export NAMESPACE=istio-system && kubectl port-forward -n ${NAMESPACE} svc/istio-ingressgateway 8080:80

And access Kubeflow dashboard as follows with default credentials:

URL: http://localhost:8080/
> username : user@example.com
> password: 12341234

Voila! You now have a kubeflow cluster at your disposal for your ML projects

Let’s try an example:


Notebook example

Text Classification with Movie Reviews

Using the same specifications above for the notebook server, we can run an example of NLP notebook as illustrated below

This notebook classifies movie reviews as positive or negative using the text of the review. This is an example of binary classification, an important and widely applicable kind of machine learning problem.

We’ll use the IMDB dataset that contains the text of 50,000 movie reviews from the Internet Movie Database. These are split into 25,000 reviews for training and 25,000 reviews for testing.

img0

We download the dataset and we build a model based on a model from TensorFlow Hub called google/nnlm-en-dim50/2

img1

Next step is the model training, depending on your server specifications, this part might take some time.

img2

Graph of accuracy and loss over time:

img3

As always, if you have any questions or want to discuss anything described in this tutorial, ping us at developers@spectrocloud.com.

Author Bio
Director of Product Management @ Spectro Cloud

Related Articles

  • Blog Posts

    How to keep your Kubernetes secrets… secret

    Read our article
  • Blog Posts

    Kubernetes for dummies tutorial

    Read our article
  • Blog Posts

    Enterprise challenges for containers and Kubernetes

    Read our article
  • Blog Posts

    Why we Invested in Spectro Cloud

    Read our article
Be the first to receive the latest on
K8s, Palette, our upcoming webinar, events, and much more!

We are using the information you provide to us to send you our montly newsletter. You may unsubscribe at any time.
For more information, please see our Privacy Policy.

Spectro Cloud uniquely enables organizations to manage Kubernetes in production, at scale. Our Palette management platform gives effortless control of the full Kubernetes lifecycle, across clouds, data centers, bare metal and edge environments.
Connect with us
Connect with us

© 2022 Spectro Cloud®. All rights reserved.