How to Run Argo Workflows in K3s
March 29, 2022
K3s is a great way to deploy Argo Workflows to a K8s cluster locally. We'll implement the main components of an Argo Workflows deployment (Argo Server, Controller, UI, CLI, and artifact repo) in K3s, and also will run a couple basic workflows.
Argo Workflows is a powerful Kubernetes-native workflow engine used to create, manage, and orchestrate parallel jobs in your Kubernetes clusters.
Argo also works well with K3s, a lightweight, simplified version of Kubernetes designed for IoT and edge computing. In this tutorial, you will learn how to deploy Argo Workflows to a Kubernetes cluster in K3s.
About Argo Workflows
Argo Workflows offers a number of key features:
- Web-based UI
- Native artifact support (MinIO, S3, Artifactory, HDFS, OSS, HTTP, Git, GCS, raw)
- Templating and cron workflows
- Workflow Archive
- REST API, Argo CLI
The `Workflow` resource is used to define the execution of a workflow as well as its storage state. Workflows consist of instructions that operate like functions, known as templates in Argo. Templates detail the steps of execution in the workflow.
The `spec` is the most important part of the `Workflow` manifest file. It contains two properties:
- templates: This defines the types of templates that you want to use in your workflow.
- entrypoint: This determines which template will be executed first.
There are multiple types of templates available, such as:
- Container: The most common template type, this schedules a container. Its spec is identical to that of a Kubernetes container spec.
- Script: This is a convenience wrapper around the container. The spec is the same as the container, but it has a `source` field that allows you to define a script. The script is saved to a file and executed from there.
- Resource: This performs create, read, update, and delete (CRUD) operations directly on resources in the cluster.
- Suspend: This suspends the execution of a workflow either for a specified duration or indefinitely.
- DAG: This allows you to define the tasks in a workflow as a graph of dependencies.
- Steps: This allows you to define the tasks in your workflow as sequential steps. It consists of inner and outer lists; inner lists run in parallel, while outer lists run one after the other.
In order to follow this tutorial, you’ll need the following:
Getting Started with K3s
First download the installation script and start up the Kubernetes server with the below command:
You can check the status of your single-node cluster by running `sudo k3s kubectl get node`. By default, the kubeconfig file is written to `/etc/rancher/k3s/k3s.yaml` with privileged permissions. However, you can use kubectl to communicate with your cluster instead by either copying the content of the default `k3s.yml` configuration file to your main kubeconfig file, which is typically located at `~/.kube/config`, or updating the KUBECONFIG environment variable path.
K9s can also be used as a complementary tool for cluster visibility:
Installing Argo CLI
The next step will be to install the Argo CLI. You can use either the latest version or select a previous one from the Argo CLI releases GitHub page. The commands you run will vary depending on your operating system. The two code blocks below are for Mac and Linux, respectively. If you’re using Windows, you can download the relevant executable from the Assets section on the Argo releases GitHub page.
Installing Argo Controller and UI
Before installing the Argo Workflows resources, you need to create an `argo` namespace:
When the Workflow Controller and the Argo Server have successfully rolled out, you can follow the steps below to access the Argo Workflows UI.
You can access the UI in multiple ways, but for this tutorial, use the port-forwarding method:
Open your browser and go to https://127.0.0.1:2746. You will be redirected to a page for authentication.
To log in to the Argo Workflows server, you will need to generate a `ServiceAccount` access token that you will use to manage your workflows. For this, you need to create a `Role` and a `RoleBinding`, and then a `ServiceAccount`.
Role: The `Role` resource is used to determine a set of permitted operations on certain Kubernetes resources in a given namespace.
Below is an example of a `Role` manifest:
RoleBinding:`RoleBinding` is used to determine which users or `ServiceAccounts` are authorized to carry out specific operations on certain resources in a given namespace. The details of the permissions are outlined in the `Role` that the `RoleBinding` is attached to.
Below is an example of a `RoleBinding` manifest:
ServiceAccount: A `ServiceAccount` is used to authenticate machine-level processes to gain access to your Kubernetes cluster. The API server in the control plane manages authentication for the processes running in the Pod.
Below is an example of a `ServiceAccount` manifest:
You can create a manifest file for each of these resources using the code snippets detailed above. Alternatively, you can create the resources directly using the kubectl CLI.
Create resource with kubectl and manifest files:
Create resources with kubectl CLI and parameter options:
Next create an access token and store it in an environment variable (such as `ARGO_TOKEN`):
Copy the printed Bearer token from your terminal and paste it in the text area of the Argo Workflow UI login.
Now you can explore the sidebar menu and select Workflows.
In this section, you will create both a basic and an advanced workflow example.
Secret Template Example
This type of workflow template supports the same secret syntax and mechanisms as the Kubernetes pod specification. You can access a secret that serves as an environment variable or volume by using this template.
First you'll deploy a secret workflow template, which works the same way and supports the same syntax as the Pod specification accessing a secret in Kubernetes. You can access the secret as either an environment variable or a volume.
Start by creating a secret in your cluster with the following command:
Next you'll create a manifest file for the workflow that will access this secret.
Now you can use the Argo CLI tool to create the workflow. Argo uses the existing kubeconfig configurations to know which cluster to interact with. To create your first workflow, run the following command:
You can review it in the Argo Workflow UI.
Next you’ll deploy a workflow that is used to manage volumes. In the example below, a volume will be dynamically created and used in a multi-step workflow.
Create the following manifest file:
As you did before, create this workflow using the Argo CLI:
Once the workflow has completed, you can review the results in the UI and delve into the logs to see the results of the generated message in the mounted volume.
In this tutorial, you learned the fundamental concepts around Argo Workflows and how to use it with a single-node K3s cluster. As demonstrated, this workflow engine operates well even in a lightweight version of Kubernetes.
In an enterprise context, however, your Kubernetes jobs could become more complex. Pipekit allows you to manage these jobs at scale. The control plane configures Argo Workflows for you in your infrastructure, enabling you to optimize multi-cluster workloads while reducing your cloud spend. To see what else Pipekit can do for you, consult the Pipekit documentation.
Subscribe for Pipekit updates.
Get the latest articles on all things Pipekit & data orchestration delivered straight to your inbox.