End to End Test Workflow
This documentation covers the details of Akri's testing workflow, which runs Akri on several Kubernetes distributions and versions each time a commit is pushed to a PR or a PR is merged.
A GitHub workflow that:

Tests

Name
File
Documentation
end-to-end
/test/run-end-to-end.py
TBD

Versions

Distro K3s Version 1.16 creates Device Plugins sockets at /var/lib/rancher/k3s/agent/kubelet/device-plugins whereas Kubernetes expects these sockets to be created at /var/lib/kubelet/device-plugins.
The fix for K3s 1.16 is to create a symbolic link from the K3s location to the Kubernetes-expected location. This is added as an exception to the workflow step for K3s:
1
if [ "${{ matrix.kube.runtime }}" == "K3s-1.16" ]; then
2
mkdir -p /var/lib/kubelet
3
if [ -d /var/lib/kubelet/device-plugins ]; then
4
sudo rm -rf /var/lib/kubelet/device-plugins
5
fi
6
sudo ln -s /var/lib/rancher/k3s/agent/kubelet/device-plugins /var/lib/kubelet/device-plugins
7
fi
Copied!
This issue was addressed in K3s version 1.17.

Jobs|Steps

The workflow comprises two jobs (build-containers and test-cases).

build-containers

build-containers builds container images for Akri 'controller' and 'agent' based upon the commit that triggers the workflow. Once build, these iamges are shared across the test-cases job, using GitHub Action upload-artifact.

test-cases

test-cases uses a GitHub strategy to run its steps across the different Kubernetes distros and versions summarized at the top of this document.
New Kubernetes distro versions may be added to the job by adding entries to jobs.test-cases.strategy.matrix.kube. Each array entry must include:
Property
Description
runtime
A unique identifier for this distro-version pair
version
A distro-specific unique identifier for the Kubernetes version
crictl
A reference to the release of cri-tools including crictl that will be used
Notes:
  • runtime is used by subsequent steps as a way to determine the distro, e.g. startsWith(matrix.kube.runtime, 'K3s')
  • version is used by each distro to determine which binary, snap etc. to install. Refer to each distro's documentation to determine the value required
  • crictl is used by K3s and MicroK8s to determine which version of crictl (sic.) is must be installed. Kubeadm includes crictl and so this variable is left as UNUSED for this distro.

Distro installation and Akri container images

Each distro has an installation step and a step to import the Akri controller and agent images created by the build-containers job.
The installation steps are identified by:
1
if: startsWith(matrix.kube.runtime, ${DISTRO})
Copied!
The installation instructions map closely with the installation instructions provided for the distro. For K3s and MicroK8s, the step includes installation of cri-tools so that crictl is available.
The container image import steps are identified by:
1
if: (startsWith(github.event_name, 'pull_request')) && (startsWith(matrix.kube.runtime, ${DISTRO}))
Copied!

Helm and state

In order to pass state between the workflow and the Python end-to-end test scripts, temporary (/tmp) files are used:
File
Description
agent_log.txt
Filename used by workflow to persist the Agent's log
controller_log.txt
Filename used by workflow to persist the Controller's log
cri_args_to_test.txt
crictcl configuration that is passed to helm install command
extra_helm_args.txt
Additionl configuration that is passed to helm install command
helm_chart_location.txt
Path to the Helm Chart
kubeconfig_path_to_test.txt
Path to kubectl cluster configuration file
runtime_cmd_to_test
Location of kubectl binary
sleep_duration.txt
Optional: contains the number of seconds to pause
version_to_test.txt
Akri version to test
If you review /test/run-end-to-end.py and /test/shared_test_code.py, you will see these files referenced.
1
AGENT_LOG_PATH = "/tmp/agent_log.txt"
2
CONTROLLER_LOG_PATH = "/tmp/controller_log.txt"
3
KUBE_CONFIG_PATH_FILE = "/tmp/kubeconfig_path_to_test.txt"
4
RUNTIME_COMMAND_FILE = "/tmp/runtime_cmd_to_test.txt"
5
HELM_CRI_ARGS_FILE = "/tmp/cri_args_to_test.txt"
6
VERSION_FILE = "/tmp/version_to_test.txt"
7
SLEEP_DURATION_FILE = "/tmp/sleep_duration.txt"
8
EXTRA_HELM_ARGS_FILE = "/tmp/extra_helm_args.txt"
9
HELM_CHART_LOCATION = "/tmp/helm_chart_location.txt"
Copied!

Tests

Of all the steps, only one is needed to run the Python end-to-end script.
stdout|stderr from the script can be logged to the workflow.

Upload

Once the end-to-end script is complete, the workflow uses the GitHub Action upload-artifact again to upload /tmp/agent_log.txt and /tmp/controller_log.txt so that these remain available (for download) once the workflow completes.
Last modified 1mo ago