# Home ![](/files/-MgROtMZZJeXQppYtInv) ## What is Akri? Akri is hosted by the Cloud Native Computing Foundation (CNCF) as a [Sandbox project](https://www.cncf.io/sandbox-projects/). Akri is a Kubernetes Resource Interface that lets you easily expose heterogeneous leaf devices (such as IP cameras and USB devices) as resources in a Kubernetes cluster, while also supporting the exposure of embedded hardware resources such as GPUs and FPGAs. Akri continually detects nodes that have access to these devices and schedules workloads based on them. Simply put: you name it, Akri finds it, you use it. ## Why Akri? At the edge, there are a variety of sensors, controllers, and MCU class devices that are producing data and performing actions. For Kubernetes to be a viable edge computing solution, these heterogeneous “leaf devices” need to be easily utilized by Kubernetes clusters. However, many of these leaf devices are too small to run Kubernetes themselves. Akri is an open source project that exposes these leaf devices as resources in a Kubernetes cluster. It leverages and extends the Kubernetes [device plugin framework](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/), which was created with the cloud in mind and focuses on advertising static resources such as GPUs and other system hardware. Akri took this framework and applied it to the edge, where there is a diverse set of leaf devices with unique communication protocols and intermittent availability. Akri is made for the edge, **handling the dynamic appearance and disappearance of leaf devices**. Akri provides an abstraction layer similar to [CNI](https://github.com/containernetworking/cni), but instead of abstracting the underlying network details, it is removing the work of finding, utilizing, and monitoring the availability of the leaf device. An operator simply has to apply a Akri Configuration to a cluster, specifying the Discovery Handler (say ONVIF) that should be used to discover the devices and the Pod that should be deployed upon discovery (say a video frame server). Then, Akri does the rest. An operator can also allow multiple nodes to utilize a leaf device, thereby **providing high availability** in the case where a node goes offline. Furthermore, Akri will automatically create a Kubernetes service for each type of leaf device (or Akri Configuration), removing the need for an application to track the state of pods or nodes. Most importantly, Akri **was built to be extensible**. Akri currently supports ONVIF, udev, and OPC UA Discovery Handlers, but more can be easily added by community members like you. The more protocols Akri can support, the wider an array of leaf devices Akri can discover. We are excited to work with you to build a more connected edge. ## Documentation Akri's documentation is divided into six sections: 1. 📘 [User Guide](/user-guide/getting-started): Documentation for Akri users. 2. 🔎 [Discovery Handlers](/discovery-handlers/onvif): Documentation on how to configure Akri using Akri's currently supported Discovery Handlers 3. 🚀 [Demos](/demos/usb-camera-demo): End-to-End demos that demonstrate how Akri can discover and use devices. Contain sample brokers and end applications. 4. ⚙️ [Architecture](/architecture/architecture-overview): Documentation that details the design and implementation of Akri's components. 5. 💻 [Development](/development/development): Documentation for Akri developers or how to build, test, and extend Akri. 6. 🎉 [Community](/community/roadmap): Information on what's next for Akri and how to get involved! ## Trademark The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our [Trademark Usage page](https://www.linuxfoundation.org/legal/trademark-usage) # Getting Started To best understand the benefits of Akri and jump into using it, we recommend you start off by completing the [end to end demo](/demos/usb-camera-demo). In the demo, you will see Akri discover mock video cameras and a streaming app display the footage from those cameras. It includes instructions on K8s cluster setup. ## The Journey Begins To get started using Akri, you must first decide what you want to discover and whether Akri currently supports a Discovery Handler that can be used to discover resources of that type. Akri discovers devices via Discovery Handlers, which are often protocol implementations that understand filter information passed via an Akri Configuration. To see the list of currently supported Discovery Handlers, see our [roadmap](https://github.com/project-akri/akri/projects/3). ### Understanding Akri Helm charts Akri is most easily deployed with Helm charts. Helm charts provide convenient packaging and configuration. Starting in v0.0.36, an **akri-dev** Helm chart will be published for each build version. Each Akri build is verified with end-to-end tests on Kubernetes, K3s, and MicroK8s. These builds may be less stable than our Releases. You can deploy these versions of Akri with this command (note: **akri-dev**): ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri-dev ``` Starting in Release v0.0.44, an **akri** Helm chart will be published for each [Release](https://github.com/project-akri/akri/releases). Releases will generally reflect milestones and will have more rigorous testing. You can deploy Release versions of Akri with this command (note: **akri**): ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri ``` To use the latest containers of the Akri components, add `--set useLatestContainers=true` when installing Akri like so: ```bash helm install akri akri-helm-charts/akri \ --set useLatestContainers=true ``` Before v0.4.0, all of Akri's Discovery Handlers were embedded in the Agent. As more Discovery Handlers are added to Akri, this will become unsustainable and cause the Agent to have a larger footprint than oftentimes necessary (if only one of the many Discovery Handlers is being leveraged). Starting in v0.4.0, Akri is starting the transition to mainly supporting an Agent image without any embedded Discovery Handlers, which will be the image used by Akri's Helm chart by default. The required Discovery Handlers can be deployed as DaemonSets by setting `.discovery.enabled=true` when installing Akri, as explained in the [user flow](#installing-akri-flow). To instead use the previous strategy of an Agent image with embedded udev, OPC UA, and ONVIF Discovery Handlers, set `agent.full=true`. To see which version of the **akri** and **akri-dev** Helm charts are stored locally, run `helm inspect chart akri-helm-charts/akri` and `helm inspect chart akri-helm-charts/akri-dev`, respectively. To grab the latest Akri Helm charts, run `helm repo update`. ### Setting up your cluster Before deploying Akri, you must have a Kubernetes cluster (v1.33 or higher) running with `kubectl` and `Helm` installed. Reference our [cluster setup documentation](/user-guide/cluster-setup) to set up a cluster or adapt your currently existing cluster. Akri currently supports Linux Nodes on amd64, arm64v8, or arm32v7. ### Installing Akri Flow Akri is installed using its Helm Chart, which contains settings for deploying the Akri Agents, Controller, Discovery Handlers, and Configurations. All these can be installed in one command, in several different Helm installations, or via consecutive `helm upgrades`. This section will focus on the latter strategy, helping you construct your Akri installation command, assuming you have already decided what you want Akri to discover. Akri's Helm chart deploys the Akri Controller and Agent by default, so you only need to specify which Discovery Handlers and Configurations need to be deployed in your command. Akri discovers devices via Discovery Handlers, which are often protocol implementations. Akri currently supports three Discovery Handlers (udev, OPC UA and ONVIF); however, custom discovery handlers can be created and deployed as explained in Akri's [Discovery Handler development document](/development/handler-development). Akri is told what to discover via Akri Configurations, which specify the name of the Discovery Handler that should be used, any discovery details (such as filters) that need to be passed to the Discovery Handler, and optionally any broker Pods and services that should be created upon discovery. For example, the ONVIF Discovery Handler can receive requests to include or exclude cameras with certain IP addresses. Let's walk through building an Akri installation command: 1. Get Akri's Helm repo ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ ``` 2. Install Akri's Controller and Agent: ```bash helm install akri akri-helm-charts/akri ``` > Note: To use Akri's latest dev releases, specify `akri-helm-charts/akri` 3. Upgrade the installation to deploy the Discovery Handler you wish to use. Discovery Handlers are deployed as DaemonSets like the Agent when `.discovery.enabled` is set. ```bash helm upgrade akri akri-helm-charts/akri \ --set .discovery.enabled=true ``` > Note: To install a full Agent with embedded udev, OPC UA, and ONVIF Discovery Handlers, set `agent.full=true` instead of enabling the Discovery Handlers. Note, this we restart the Agent Pods. > > ```bash > helm upgrade akri akri-helm-charts/akri \ > --set agent.full=true > ``` 4. Upgrade the installation to apply a Configuration, which requests discovery of certain devices by a Discovery Handler. A Configuration is applied by setting `.configuration.enabled`. While some Configurations may not require any discovery details to be set, oftentimes setting details is preferable for narrowing the Discovery Handlers' search. These are set under `.configuration.discoveryDetails`. For example, udev rules are passed to the udev Discovery Handler to specify which devices in the Linux device file system it should search for by setting `udev.configuration.discoveryDetails.udevRules`. Akri can be instructed to automatically deploy workloads called "brokers" to each discovered device by setting a broker Pod image in a Configuration via `--set .configuration.brokerPod.image.repository=`. Learn more about creating brokers in the [broker development document](/development/broker-development). ```bash helm upgrade akri akri-helm-charts/akri \ --set .discovery.enabled=true \ --set .configuration.enabled=true \ # set any discovery details in the Configuration # specify any broker images in the Configuration ``` Installation could have been done in one step rather than a series of upgrades: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set .discovery.enabled=true \ --set .configuration.enabled=true \ # set any discovery details in the Configuration # specify any broker images in the Configuration ``` As a real example, Akri's Controller, Agents, udev Discovery Handlers, and a udev Configuration that specifies the discovery of only USB video devices and an Nginx broker Pod image are installed like so: ```bash helm install akri akri-helm-charts/akri \ --set udev.discovery.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.discoveryDetails.udevRules[0]='KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"' \ --set udev.configuration.brokerPod.image.repository=nginx ``` > Note: set `.brokerPod.image.tag` to specify an image tag (defaults to `latest`). A terminating BusyBox Job broker could have been specified instead by setting the image of the `brokerJob` instead of the `brokerPod`. ```bash helm install akri akri-helm-charts/akri \ --set udev.discovery.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.discoveryDetails.udevRules[0]='KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"' \ --set udev.configuration.brokerJob.image.repository=busybox ``` This installation can be expanded to install multiple Discovery Handlers and/or Configurations. See the documentation on [udev](/discovery-handlers/udev), [OPC UA](/discovery-handlers/opc-ua), and [ONVIF](/discovery-handlers/onvif) Configurations to learn more about setting the discovery details passed to their Discovery Handlers and more. See [modifying an Akri Installation](/user-guide/customizing-an-akri-installation) to learn about how to use Akri's Helm chart to install additional Configurations and Discovery Handlers. ### Inspecting an Akri Installation * Run `kubectl get crd`, and you should see Akri's two CRDs listed. * Run `kubectl get pods -o wide`, and you should see the Akri Controller, Agent, and (if specified) broker pods. * Run `kubectl get akric`, and you should see the Configuration for the protocol you specified. * If devices were discovered, the instances can be seen by running `kubectl get akrii` and further inspected by running `kubectl get akrii - -o yaml`. * List all that Akri has automatically created and deployed, namely the Akri Controller, Agents, Configurations, Instances (which are the Akri custom resource that represents each device), and if specified, broker Pods, a service for each broker Pod, and a service for all brokers. ```bash watch microk8s kubectl get pods,akric,akrii,services -o wide ``` For K3s and vanilla Kubernetes ```bash watch kubectl get pods,akric,akrii,services -o wide ``` **Deleting Akri Configurations** To tell Akri to stop discovering devices, simply delete the Configuration that initiated the discovery. Watch as all instances that represent the discovered devices are deleted. ```bash kubectl delete akric akri- kubectl get akrii ``` ### Deleting Akri 1. If you are done using Akri, it can be uninstalled via Helm. ```bash helm delete akri ``` 2. Delete Akri's CRDs. ```bash kubectl delete crd instances.akri.sh kubectl delete crd configurations.akri.sh ``` ### Customizing where the Controller runs By default the Controller can be deployed to any control plane or worker node. This can be changed by adding extra settings when installing Akri below. If you don't want the Controller to ever be scheduled to control plane nodes, add `--set controller.allowOnControlPlane=false` to your install command below. Conversely, if you only want the Controller to run on control plane nodes, add `--set controller.onlyOnControlPlane=true`. This will guarantee the Controller only runs on nodes with the label (key, value) of (`node-role.kubernetes.io/master`, ""), which is the default label for the control plane node for Kubernetes. However, control plane nodes on MicroK8s and K3s/RKE2 may not have this exact label by default, so you can add it by running `kubectl label node ${HOSTNAME,,} node-role.kubernetes.io/master=--overwrite=true`. Or alternatively, in K3s/RKE2, you can keep the default label value on the master and set `controller.nodeSelectors."node-role\.kubernetes\.io/master"=true`. # Kubernetes Cluster Setup Before deploying Akri, you must have a Kubernetes cluster (v1.33 or higher) running with `kubectl` and `Helm` installed. Akri is Kubernetes native, so it should run on most Kubernetes distributions. This document provides cluster setup instructions for the three Kubernetes distributions that all of our end-to-end tests run on. {% hint style="info" %} Note: All nodes must be Linux on amd64, arm64v8, or arm32v7. {% endhint %} ## Install Kubernetes Distribution {% tabs %} {% tab title="Kubernetes" %} 1. Reference [Kubernetes documentation](https://kubernetes.io/docs/tasks/tools/) for instructions on how to install Kubernetes. See Akri's [release notes](https://github.com/project-akri/akri/releases) to see what versions of Kubernetes Akri has been tested on. 2. Install Helm for deploying Akri. ```bash sudo apt install -y curl curl -L https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash ``` {% hint style="info" %} Note: To enable workloads on a single-node cluster, remove the master taint. ```bash kubectl taint nodes --all node-role.kubernetes.io/master- ``` {% endhint %} {% endtab %} {% tab title="K3s" %} 1. Install [K3s](https://k3s.io/). The following will install the latest K3s version. Reference Akri's [release notes](https://github.com/project-akri/akri/releases) to see what versions of K3s Akri has been tested on. ```bash curl -sfL https://get.k3s.io | sh - ``` > Note: Optionally specify a version with the `INSTALL_K3S_VERSION` env var as follows: `curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.21.5+k3s1 sh -` 2. Grant admin privilege to access kube config. ```bash sudo addgroup k3s-admin sudo adduser $USER k3s-admin sudo usermod -a -G k3s-admin $USER sudo chgrp k3s-admin /etc/rancher/k3s/k3s.yaml sudo chmod g+r /etc/rancher/k3s/k3s.yaml su - $USER ``` 3. Check K3s status. ```bash kubectl get node ``` 4. Install Helm. ```bash export KUBECONFIG=/etc/rancher/k3s/k3s.yaml sudo apt install -y curl curl -L https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash ``` 5. If desired, add nodes to your cluster by running the K3s installation script with the `K3S_URL` and `K3S_TOKEN` environment variables. See [K3s installation documentation](https://rancher.com/docs/k3s/latest/en/quick-start/#install-script) for more details. {% endtab %} {% tab title="MicroK8s" %} 1. Install [MicroK8s](https://microk8s.io/docs). The following will install the latest MicroK8s version. Add `--channel=$VERSION/stable` to specify as specific Kubernetes version. Reference Akri's [release notes](https://github.com/project-akri/akri/releases) to see what versions of MicroK8s Akri has been tested on. ```bash snap install microk8s --classic ``` 2. Grant admin privilege for running MicroK8s commands. ```bash sudo usermod -a -G microk8s $USER sudo chown -f -R $USER ~/.kube su - $USER ``` 3. Check MicroK8s status. ```bash microk8s status --wait-ready ``` 4. Enable CoreDNS, Helm and RBAC for MicroK8s. ```bash microk8s enable dns helm3 rbac ``` 5. If you don't have an existing `kubectl` and `helm` installations, add aliases. If you do not want to set an alias, add `microk8s` in front of all `kubectl` and `helm` commands. ```bash alias kubectl='microk8s kubectl' alias helm='microk8s helm3' ``` 6. By default, MicroK8s does not allow Pods to run in a privileged context. None of Akri's components run privileged; however, if your custom broker Pods do in order to access devices for example, enable privileged Pods like so: ```bash echo "--allow-privileged=true" >> /var/snap/microk8s/current/args/kube-apiserver microk8s.stop microk8s.start ``` 7. If desired, reference [MicroK8's documentation](https://microk8s.io/docs/clustering) to add additional nodes to the cluster. {% endtab %} {% endtabs %} # Customizing an Akri Installation The [ONVIF](/discovery-handlers/onvif), [udev](/discovery-handlers/udev), and [OPC UA](/discovery-handlers/opc-ua) Configurations documentation explains how to deploy Akri and utilize a specific Discovery Handler using Helm (more information about the Akri Helm charts can be found in the [user guide](/user-guide/getting-started#understanding-akri-helm-charts)). This documentation elaborates upon them, covering the following: 1. Starting Akri without any Configurations 2. Generating, modifying and applying a Configuration 3. Deploying multiple Configurations 4. Modifying a deployed Configuration 5. Adding another Configuration to a cluster 6. Modifying a broker 7. Deleting a Configuration from a cluster 8. Applying Discovery Handlers ## Starting Akri without any Configurations To install Akri without any protocol Configurations, run this: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri ``` This will deploy the Akri Controller and deploy Akri Agents. ## Generating, modifying and applying a Configuration Helm allows us to parametrize the commonly modified fields in our Configuration templates and we have provided many (to see them, run `helm inspect values akri-helm-charts/akri`). For more advanced Configuration changes that are not aided by our Helm chart, we suggest creating a Configuration file using Helm and then manually modifying it. For example, to create an ONVIF Configuration file, run the following. (To instead create a udev Configuration, substitute `onvif.configuration.enabled` with `udev.configuration.enabled` and add a udev rule. For OPC UA, substitute with `opcua.configuration.enabled`.) ```bash helm template akri akri-helm-charts/akri \ --set onvif.configuration.enabled=true \ --set onvif.configuration.brokerPod.image.repository=nginx \ --set rbac.enabled=false \ --set controller.enabled=false \ --set agent.enabled=false > configuration.yaml ``` Note, that for the broker pod image, nginx was specified. Insert your broker image instead or remove the broker pod image from the installation command to generate a Configuration without a broker PodSpec or ServiceSpecs. Once you have modified the yaml file, you can apply the new Configuration to the cluster with standard kubectl like this: ```bash kubectl apply -f configuration.yaml ``` {% hint style="info" %} When modifying the Configuration, do not remove the resource request and limit `{{PLACEHOLDER}}`. The Controller inserts the request for the discovered device/Instance here. {% endhint %} The following sections explain some of the ways the configuration.yaml could be modified to customize settings/fields that cannot be set with Akri's Helm Chart. #### Modifying the brokerPodSpec The `brokerPodSpec` property is a full [PodSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.33/#podspec-v1-core) and can be modified as such. For example, to allow the master Node to have a protocol broker Pod scheduled to it, modify the Configuration, ONVIF in this case, like so: ```yaml spec: brokerPodSpec: containers: - name: akri-onvif-video-broker image: "ghcr.io/project-akri/examples/onvif-video-broker:latest-dev" resources: limits: "{{PLACEHOLDER}}": "1" tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule ``` #### Modifying the brokerJobSpec The `brokerJobSpec` property is a full [JobSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.33/#jobspec-v1-batch) and can be modified as such. Akri's Helm chart enables modifying the `capacity`, `parallelism`, and `backoffLimit` fields of the JobSpec. Other fields of the JobSpec and the PodSpec within the JobSpec can be specified in a similar manner as described in the [modifying the PodSpec section](#Modifying-the-brokerPodSpec). #### Modifying instanceServiceSpec or configurationServiceSpec The `instanceServiceSpec` and `configurationServiceSpec` properties are full [ServiceSpecs](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.33/#servicespec-v1-core) and can be modified as such. The simplest reason to modify either might be to specify different ports (perhaps 8085 and 8086): ```yaml spec: instanceServiceSpec: ports: - name: grpc port: 8085 targetPort: 8083 configurationServiceSpec: ports: - name: grpc port: 8086 targetPort: 8083 ``` {% hint style="info" %} The simple properties of `instanceServiceSpec` and `configurationServiceSpec` (like name, port, targetPort, and protocol) can be set using Helm's `--set` command, e.g.`--set onvif.instanceService.targetPort=90`. {% endhint %} ## Deploying multiple Configurations using `helm install` If you want your end application to consume frames from both IP cameras and locally attached cameras, Akri can be installed from the start with both the ONVIF and udev Configurations like so: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set onvif.configuration.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.discoveryDetails.udevRules[0]='KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"' ``` {% hint style="info" %} You must specify a udev rule to successfully build the udev Configuration. {% endhint %} You can confirm that both a `akri-onvif` and `akri-udev` Configuration have been created by running: ```bash kubectl get akric ``` Each Configuration could also have been deployed via separate Helm installations: ```bash helm install udev-config akri-helm-charts/akri \ --set controller.enabled=false \ --set agent.enabled=false \ --set rbac.enabled=false \ --set udev.configuration.enabled=true \ --set udev.configuration.discoveryDetails.udevRules[0]='KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"' helm install onvif-config akri-helm-charts/akri \ --set controller.enabled=false \ --set agent.enabled=false \ --set rbac.enabled=false \ --set onvif.configuration.enabled=true ``` ## Modifying a deployed Configuration An already deployed Configuration can be modified in one of two ways: 1. Using the `helm upgrade` command 2. [Generating, modifying and applying a custom Configuration](#generating-modifying-and-applying-a-configuration) Note: Only the broker properties and capacity of an applied configuration should be modified, for any other modification, you need to delete and reapply the Configuration. ### Using `helm upgrade` A Configuration can be modified by using the `helm upgrade` command. It upgrades an existing release according to the values provided, only updating what has changed. Simply modify your `helm install` command to reflect the new **desired state** of Akri and replace `helm install` with `helm upgrade`. Using the ONVIF protocol implementation as an example, say you want to set the capacity of discovered cameras to 3: ```bash helm upgrade akri akri-helm-charts/akri \ --set onvif.configuration.enabled=true \ --set onvif.configuration.brokerPod.image.repository= \ --set onvif.configuration.brokerPod.image.tag= \ --set onvif.configuration.capacity=3 ``` Note that the command is not simply `helm upgrade --set onvif.configuration.capacity=3`; rather, it includes all the old settings along with the new one. Also, note that we assumed you specified a broker pod image in your original installation command, so that brokers were deployed to utilize discovered cameras. Helm will create a new ONVIF Configuration and apply it to the cluster. When the Agent sees that a Configuration has been updated, it updates all Instances associated with that new Configuration. ## Adding another Configuration to a cluster Another Configuration can be added to an existing Akri installation using `helm upgrade` or via a new Helm installation. ### Adding additional Configurations using `helm upgrade` Another Configuration can be added to the cluster by using `helm upgrade`. For example, if you originally installed just the ONVIF Configuration and now also want to discover local cameras via udev, as well, simply run the following: ```bash helm upgrade akri akri-helm-charts/akri \ --set onvif.enabled=true \ --set udev.enabled=true \ --set udev.udevRules[0]='KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"' ``` ### Adding additional Configurations via new Helm installations The udev Configuration could also have been applied via a new Helm installation like so: ```bash helm install udev-config akri-helm-charts/akri \ --set controller.enabled=false \ --set agent.enabled=false \ --set rbac.enabled=false \ --set udev.configuration.enabled=true \ --set udev.configuration.discoveryDetails.udevRules[0]='KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"' ``` ## Modifying a broker Currently, to modify a broker (be it a Job or Pod), you need to delete and re-apply the Configuration. ## Deleting a Configuration from a cluster If an operator no longer wants Akri to discover devices defined by a Configuration, they can delete the Configuration and all associated broker pods will automatically be brought down. This can be done with `helm upgrade`, `helm delete`, or kubectl. ### Deleting a Configuration using `helm upgrade` A Configuration can be deleted from a cluster using `helm upgrade`. For example, if both ONVIF and udev Configurations have been installed in a cluster, the udev Configuration can be deleted by only specifying the ONVIF Configuration in a `helm upgrade` command like the following: ```bash helm upgrade akri akri-helm-charts/akri \ --set onvif.enabled=true ``` ### Deleting a Configuration using `helm delete` If the Configuration was applied in its own Helm installation (named `udev-config` in this example), the Configuration can be deleted by deleting the installation. ```bash helm delete udev-config ``` ### Deleting a Configuration using kubectl A configuration can also be deleted using kubectl. To list all applied Configurations, run `kubectl get akric`. If both udev and ONVIF Configurations have been applied with capacities of 5. The output should look like the following: ```bash NAME CAPACITY AGE akri-onvif 5 3s akri-udev 5 16m ``` To delete the ONVIF Configuration and bring down all ONVIF broker pods, run: ```bash kubectl delete akric akri-onvif ``` ## Installing Discovery Handlers The Agent discovers devices via Discovery Handlers. Akri supports an Agent image that includes all supported Discovery Handlers. This Agent will be used if `agent.full=true`, like so: ```bash helm install akri akri-helm-charts/akri \ --set agent.full=true ``` By default, a slim Agent without any embedded Discovery Handlers is deployed and the required Discovery Handlers can be deployed as DaemonSets by specifying `.discovery.enabled=true` when installing Akri. For example, Akri is installed with the OPC UA and ONVIF Discovery Handlers like so: ```bash helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set onvif.discovery.enabled=true ``` # Requesting Akri Resources Akri discovers tiny devices, advertizes them as resources, and automatically deploys workloads to utilize those devices. The latter functionality is optional. You can use Akri solely to discover and advertize devices by omitting a broker pod image from a Configuration. Then, you can schedule your own pods, requesting the discovered Akri Instances (which represent each tiny device) as resource limits. Lets walk through how this works, using the ONVIF Discovery Handler as an example. Install Akri with the ONVIF Discovery Handler and Configuration, omitting a broker pod image. ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set onvif.discovery.enabled=true \ --set onvif.configuration.enabled=true ``` After installing Akri and your Configuration, list all discovered instances by running `kubectl get akrii`. Note `akrii` is a short name for Akri Instance. All the instances will be named in the format `-`, where `id` varies whether or not the device is sharable or visible by multiple nodes. 1. For unshared devices, `id` is a hash of a descriptor of the device and the name of the node that can see the device. For example, the `id` of an Instance representing a usb camera at devnode `/dev/video0` on a node named workerA would be `hash(/dev/video0workerA)`. 2. For shared devices, `id` is only a hash of the descriptor of the device. This way, all agents create or modify an Instance with the same name for the same device. For example, since IP cameras are sharable, the `id` for an IP camera would be `hash(uri)`. You can change the name of the Configuration and resultant Instances to be `onvif-camera` by adding `--set onvif.configuration.name=onvif-camera` to your installation command. Now, you can schedule pods that request these Instances as resources. Assuming the Configuration name has been set to `onvif-camera`, you can request the `onvif-camera-` Instance as a resource by adding the following to the PodSpec of your Deployment or Job: ```yaml resources: limits: akri.sh/onvif-camera-: "1" requests: akri.sh/onvif-camera-: "1" ``` As an example, a Deployment that would deploy an nginx broker to one of the devices discovered by the ONVIF Discovery Handler may look like this: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: onvif-camera-broker-deployment labels: app: onvif-camera-broker spec: replicas: 1 selector: matchLabels: app: onvif-camera-broker template: metadata: labels: app: onvif-camera-broker spec: containers: - name: onvif-camera-broker image: nginx resources: limits: akri.sh/onvif-camera-: "1" requests: akri.sh/onvif-camera-: "1" ``` Apply your Deployment to the cluster and watch the broker start to run. If you inspect the Instance of the resource you requested in your deployment, you will see one of the slots has now been reserved by the node that is currently running the broker. ```bash kubectl apply -f deployment-requesting-onvif-camera.yaml kubectl get akrii onvif-camera- -o yaml ``` ## Requesting resources at Configuration level Akri also exposes all discovered devices as resources at Configuration level. Configuration level resources can be referred by the name of Configuration. With Configuration-level resources, instead of needing to know the specific Instances id `onvif-camera-` to request, you can use Configuration name `` to request resources. Agent will behind the scenes do the work of selecting which Instances to reserve. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: onvif-camera-broker-deployment labels: app: onvif-camera-broker spec: replicas: 1 selector: matchLabels: app: onvif-camera-broker template: metadata: labels: app: onvif-camera-broker spec: containers: - name: onvif-camera-broker image: nginx resources: limits: akri.sh/onvif-camera: "1" requests: akri.sh/onvif-camera: "1" ``` # Monitoring with Prometheus To enable a deeper understanding of the state of an Akri deployment and Node resource usage by Akri containers, Akri exposes metrics with Prometheus. This document will cover: * Installing Prometheus * Enabling Prometheus with Akri * Visualizing metrics with Grafana * Akri's currently exposed metrics * Exposing metrics from an Akri Broker Pod ## Installing Prometheus In order to expose Akri's metrics, Prometheus must be deployed to your cluster. If you already have Prometheus running on your cluster, you can skip this step. Prometheus is comprised of many components. Instead of manually deploying all the components, the entire kube-prometheus stack can be deployed via its [Helm chart](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack). It includes the Prometheus operator, node exporter, built in Grafana support, and more. 1. Get the kube-prometheus stack Helm repo. ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update ``` 2. Install the chart, specifying what namespace you want Prometheus to run in. It does not have to be the same namespace in which you are running Akri. For example, it may be in a namespace called `monitoring` as in the command below. [By default](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusioscrape), Prometheus only discovers PodMonitors within its namespace. This should be disabled by setting`podMonitorSelectorNilUsesHelmValues` to `false` so that Akri's custom PodMonitors can be discovered. Additionally, the Grafana service can be exposed to the host by making it a NodePort service. It may take a minute or so to deploy all the components. ```bash helm install prometheus prometheus-community/kube-prometheus-stack \ --set grafana.service.type=NodePort \ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \ --namespace monitoring ``` > The Prometheus dashboard can also be exposed to the host by adding `--set prometheus.service.type=NodePort`. If intending to[ expose metrics](#exposing-metrics-from-an-akri-broker-pod) from a Broker Pod via a ServiceMonitor also set `serviceMonitorSelectorNilUsesHelmValues` to `false`. ## Enabling Prometheus in Akri The Akri Controller and Agent publish metrics to port 8080 at a `/metrics` endpoint. However, these cannot be accessed by Prometheus without creating PodMonitors, which are custom resources that tell Prometheus which Pods to monitor. These components can all be automatically created and deployed via Helm by setting `--set prometheus.enabled=true` when installing Akri. Install Akri and expose the Controller and Agent's metrics to Prometheus by running: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set prometheus.enabled=true ``` ## Visualizing metrics with Grafana Now that Akri's metrics are being exposed to Prometheus, they can be visualized in Grafana. 1. Determine the port that the Grafana Service is running on, specifying the namespace if necessary, and save it for the next step. ```bash kubectl get service/prometheus-grafana --namespace=monitoring --output=jsonpath='{.spec.ports[?(@.name=="service")].nodePort}' && echo ``` 2. SSH port forwarding can be used to access Grafana. Open a new terminal, and enter your ssh command to access the machine running Akri and Prometheus followed by the port forwarding request. The following command will use port 50000 on the host. Feel free to change it if it is not available. Be sure to replace `` with the port number outputted in the previous step. ```bash ssh someuser@ -L 50000:localhost: ``` 3. Navigate to `http://localhost:50000/` and enter Grafana's default username `admin` and password `prom-operator`. Once logged in, the username and password can be changed in account settings. Now, you can create a Dashboard to display the Akri metrics. ## Akri's currently exposed metrics Akri uses the [Rust Prometheus client library](https://github.com/tikv/rust-prometheus) to expose metrics. It exposes all the [default process metrics](https://prometheus.io/docs/instrumenting/writing_clientlibs/#process-metrics), such as Agent or Controller total CPU time usage (`process_cpu_seconds_total`) and RAM usage (`process_resident_memory_bytes`), along with the following custom metrics, all of which are prefixed with `akri`. | Metric Name | Metric Type | Metric Source | Buckets | | --------------------------------- | ------------- | ------------- | ------------------------------------------------------ | | akri\_instance\_count | IntGaugeVec | Agent | Configuration, shared | | akri\_discovery\_response\_result | IntCounterVec | Agent | Discovery Handler name, response result (Success/Fail) | | akri\_discovery\_response\_time | HistogramVec | Agent | Configuration | | akri\_broker\_pod\_count | IntGaugeVec | Controller | Configuration, Node | ## Exposing metrics from an Akri Broker Pod Metrics can also be published by Broker Pods and exposed to Prometheus. This workflow is not unique to Akri and is equivalent to exposing metrics from any deployment to Prometheus. Using the [appropriate Prometheus client library](https://prometheus.io/docs/instrumenting/clientlibs/) for your broker, expose some metrics. Then, deploy a Service to expose the metrics, specifying the name of the associated Akri Configuration as a selector (`akri.sh/configuration: `), since the Configuration name is added as a label to all the Broker Pods by the Akri Controller. Finally, deploy a ServiceMonitor that selects for the previously mentioned service. This tells Prometheus which service(s) to discover. ### Example: Exposing metrics from the udev video sample Broker As an example, an `akri_frame_count` metric has been created in the sample [udev-video-broker](https://github.com/project-akri/examples/tree/main/brokers/udev-video-broker). Like the Agent and Controller, it publishes both the default process metrics and the custom `akri_frame_count` metric to port 8080 at a `/metrics` endpoint. 1. Akri can be installed with the udev Configuration, filtering for only usb video cameras and specifying a Configuration name of `akri-udev-video`, by running: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set udev.enabled=true \ --set udev.name=akri-udev-video \ --set udev.udevRules[0]='KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"' \ --set udev.brokerPod.image.repository="ghcr.io/project-akri/examples/udev-video-broker" ``` > **Note**: To expose the Agent and Controller's Prometheus metrics, add `--set prometheus.enabled=true`. > **Note**: If Prometheus is running in a different namespace as Akri and was not enabled to discover ServiceMonitors in other namespaces when installed, upgrade your Prometheus Helm installation to set `prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues` to `false`. > > > ```bash > > helm upgrade prometheus prometheus-community/kube-prometheus-stack \ > > --set grafana.service.type=NodePort \ > > --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \ > > --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \ > > --namespace monitoring > > ``` 2. Then, create a Service for exposing these metrics, targeting all Pods labeled with the Configuration name `akri-udev-video`. ``` apiVersion: v1 kind: Service metadata: name: akri-udev-video-broker-metrics labels: app: akri-udev-video-broker-metrics spec: selector: akri.sh/configuration: akri-udev-video ports: - name: metrics port: 8080 type: ClusterIP ``` > The metrics also could have been exposed by adding the metrics port to the Configuration level service in the udev Configuration. 3. Apply the Service to your cluster. ``` kubectl apply -f akri-udev-video-broker-metrics-service.yaml ``` 4. Create the associated ServiceMonitor. Note how the selector matches the app name of the Service. ``` apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: akri-udev-video-broker-metrics labels: release: prometheus spec: selector: matchLabels: app: akri-udev-video-broker-metrics endpoints: - port: metrics ``` 5. Apply the ServiceMonitor to your cluster. ``` kubectl apply -f akri-udev-video-broker-metrics-service-monitor.yaml ``` 6. The frame count metric reports the number of video frames that have been requested by some application. It will remain at zero unless an application is deployed that utilizes the video Brokers. Deploy the Akri sample streaming application by running the following: ``` kubectl apply -f https://raw.githubusercontent.com/project-akri/akri/main/deployment/samples/akri-video-streaming-app.yaml watch kubectl get pods ``` # ONVIF for IP Cameras ## Background ONVIF is a standard used by many IP cameras and defines discovery and access for RTSP camera streams. Along with an ONVIF Discovery Handler, Akri has provided a generic ONVIF Configuration for initiating ONVIF discovery. Akri has also provided a sample broker (`akri-onvif-video-broker`), which acts as a frame server. Using Akri's default ONVIF Configuration to discover and utilize ONVIF cameras looks like the following: ![](/files/-MgSrsPDq3tGTYzDd-c8) 1. An operator applies the ONVIF Configuration to the cluster (by enabling ONVIF when installing the Akri Helm chart). They also specific a broker image -- `akri-onvif-video-broker` in the figure. 2. The Akri Agent uses the ONVIF Discovery Handler to discover the IP cameras and creates Instances for each discovered camera. 3. The Akri Controller sees the Instances and deploys `akri-onvif-video-broker` pods, which were specified in the Configuration. The Controller also creates a Kubernetes service for each ONVIF camera along with one service for all the ONVIF cameras. All of Akri's components can be deployed by specifying values in its Helm chart during an installation. This section will cover the values that should be set to (1) deploy the ONVIF Discovery Handlers and (2) apply a Configuration that tells Akri to discover devices using that Discovery Handler. ## Deploying the ONVIF Discovery Handler In order for the Agent to know how to discover IP cameras an ONVIF Discovery Handler must exist. Akri supports an Agent image that includes all supported Discovery Handlers. This Agent will be used if `agent.full=true`. By default, a slim Agent without any embedded Discovery Handlers is deployed and the required Discovery Handlers can be deployed as DaemonSets. This documentation will use that strategy, deploying ONVIF Discovery Handlers by specifying `onvif.discovery.enabled=true` when installing Akri. ## ONVIF Configuration Settings Instead of having to assemble your own ONVIF Configuration yaml, we have provided a [Helm template](https://github.com/project-akri/akri/blob/main/deployment/helm/templates/onvif-configuration.yaml). Helm allows us to parametrize the commonly modified fields in our configuration files, and we have provided many for ONVIF (to see them, run `helm inspect values akri-helm-charts/akri`). More information about the Akri Helm charts can be found in the [user guide](/user-guide/getting-started#understanding-akri-helm-charts). To apply the ONVIF Configuration to your cluster, simply set `onvif.configuration.enabled=true` along with any of the following additional Configuration settings when installing Akri. ### Discovery Handler Discovery Details Settings Discovery Handlers are passed discovery details that are set in a Configuration to determine what to discover, filter out of discovery, and so on. The ONVIF Discovery Handler, supports the following discovery details: | Helm Key | Value | Default | Description | | ------------------------------------------------------------ | ---------------------- | ------- | ------------------------------------------------------------------------------------------------------ | | onvif.configuration.discoveryDetails.ipAddresses.action | Include, Exclude | Exclude | filter action to take on a set of IP addresses | | onvif.configuration.discoveryDetails.ipAddresses.items | array of IP addresses | empty | IP addresses that the filter action acts upon | | onvif.configuration.discoveryDetails.macAddresses.action | Include, Exclude | Exclude | filter action to take on a set of mac addresses | | onvif.configuration.discoveryDetails.macAddresses.items | array of mac addresses | empty | mac addresses that the filter action acts upon | | onvif.configuration.discoveryDetails.scope.action | Include, Exclude | Exclude | filter action to take on a set of scopes | | onvif.configuration.discoveryDetails.scope.items | array of scopes | empty | scopes that the filter action acts upon | | onvif.configuration.discoveryDetails.uuids.action\* | Include, Exclude | Exclude | filter action to take on a set of device uuids | | onvif.configuration.discoveryDetails.uuids.items\* | array of UUIDs | empty | device uuids that the filter action acts upon | | onvif.configuration.discoveryDetails.discoveryTimeoutSeconds | number of seconds | 1 | max amount of time the Discovery Handler should search before reporting any (newly) discovered devices | \*Onvif device uuid: the address property of the Endpoint Reference \[ONVIF Core Specification 7.3.1 Endpoint reference] can be used as the device id to identify the device. The address property in Endpoint Reference is in the Uniform Resource Name: Universally Unique Identifier (URN:UUID) format. The same UUID can be retrieved by the `GetEndpointReference` command after a camera is discovered by Probe message. ### Broker Pod Settings If you would like non-terminating workloads ("broker" Pods) to be deployed automatically to discovered cameras, a broker image should be specified (under `brokerPod`) in the Configuration. Alternatively, if it meets your scenario, you could use the Akri frame server broker ("ghcr.io/project-akri/examples/onvif-video-broker"). If you would rather manually deploy pods to utilize the cameras advertized by Akri, don't specify a broker pod and see our documentation on [requesting resources advertized by Akri](/user-guide/requesting-akri-resources). > Note only a `brokerJob` OR `brokerPod` should be specified. | Helm Key | Value | Default | Description | | ----------------------------------------------------- | ------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | onvif.configuration.brokerPod.image.repository | image string | "" | image of broker Pod that should be deployed to discovered devices | | onvif.configuration.brokerPod.image.tag | tag string | "latest" | image tag of broker Pod that should be deployed to discovered devices | | onvif.configuration.brokerPod.resources.memoryRequest | string | "98Mi" | the minimum amount of RAM that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri ONVIF sample broker. Adjust to the size of your broker. | | onvif.configuration.brokerPod.resources.cpuRequest | string | "134m" | the minimum amount of CPU that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri ONVIF sample broker. Adjust to the size of your broker. | | onvif.configuration.brokerPod.resources.memoryLimit | string | "400Mi" | the maximum amount of RAM this Pod can consume. Default based on the Akri ONVIF sample broker. Adjust to the size of your broker. | | onvif.configuration.brokerPod.resources.cpuLimit | string | "2800m" | the maximum amount of CPU this Pod can consume. Default based on the Akri ONVIF sample broker. Adjust to the size of your broker. | ### Broker Job Settings If you would like terminating [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) to be deployed automatically to discovered cameras, a broker image should be specified (under `brokerJob`) in the Configuration. A Kubernetes Job deploys a set number of terminating Pods. > Note only a `brokerJob` OR `brokerPod` should be specified. | Helm Key | Value | Default | Description | | ----------------------------------------------------- | ------------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | onvif.configuration.brokerJob.image.repository | image string | "" | image of broker Job that should be deployed to discovered devices | | onvif.configuration.brokerJob.image.tag | tag string | "latest" | image tag of broker Job that should be deployed to discovered devices | | onvif.configuration.brokerJob.resources.memoryRequest | string | "98Mi" | the minimum amount of RAM that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Adjust to the size of your broker. | | onvif.configuration.brokerJob.resources.cpuRequest | string | "134m" | the minimum amount of CPU that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Adjust to the size of your broker. | | onvif.configuration.brokerJob.resources.memoryLimit | string | "400Mi" | the maximum amount of RAM this Pod can consume.Adjust to the size of your broker. | | onvif.configuration.brokerJob.resources.cpuLimit | string | "2800m" | the maximum amount of CPU this Pod can consume. Adjust to the size of your broker. | | onvif.configuration.brokerJob.command | string array | Empty | command to be executed in the Pod | | onvif.configuration.brokerJob.restartPolicy | string array | `OnFailure` | `RestartPolicy` for the Job. Can either be `OnFailure` or `Never` for Jobs. | | onvif.configuration.brokerJob.backoffLimit | number | 2 | defines the Kubernetes Job [backoff failure policy](https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy) | | onvif.configuration.brokerJob.parallelism | number | 1 | defines the Kubernetes Job [`parallelism`](https://kubernetes.io/docs/concepts/workloads/controllers/job/#parallel-jobs) | | onvif.configuration.brokerJob.completions | number | 1 | defines the Kubernetes Job [`completions`](https://kubernetes.io/docs/concepts/workloads/controllers/job) | ### Disabling Automatic Service Creation By default, if a broker Pod is specified, the generic ONVIF Configuration will create services for all the brokers of a specific Akri Instance and all the brokers of an Akri Configuration. The creation of these services can be disabled. | Helm Key | Value | Default | Description | | ---------------------------------------------- | ----------- | ------- | --------------------------------------------------------------------- | | onvif.configuration.createInstanceServices | true, false | true | a service should be automatically created for each broker Pod | | onvif.configuration.createConfigurationService | true, false | true | a single service should be created for all brokers of a Configuration | ### Capacity Setting By default, if a broker Pod is specified, a single broker Pod is deployed to each device. To modify the Configuration so that a camera is accessed by more or fewer nodes via broker Pods, update the `onvif.configuration.capacity` setting to reflect the correct number. For example, if your high availability needs are met by having 1 redundant pod, you can update the Configuration like this by setting `onvif.configuration.capacity=2`. | Helm Key | Value | Default | Description | | ---------------------------- | ------ | ------- | ------------------------------------------------------------------------------------- | | onvif.configuration.capacity | number | 1 | maximum number of brokers that can be deployed to utilize a device (up to 1 per Node) | ### Discovery Handler Discovery Properties Settings Agent read the content of `discoveryProperties` in Configuration and generate a string key-value pair list to Discovery Handler. The Onvif discovery handler leverage the `discoveryProperties` to read the credentials for authenticated discovery. There are two attributes required for Onvif discovery handler to perform authenticated discovery: 1. an id that can unique identify a camera 2. a credential (username/password) to authenticate the access to a camera Onvif discovery handler gets the device uuid when discovering Onvif camera devices, and use the id to look up for matching credential from the string key-value pair list passed by Agent. #### Organize Credentials in Akri Configuration and Kubernetes Secrets All secret information are kept in Kubernetes Secrets. In Configuration, we need to create a mapping for the secret information so Agent can read the secret information and pass it with the mapping to Onvif Discovery Handler. With the mapping and secret information, Onvif Discovery Handler can look up credential using device ids. There are 3 ways to organize secret information: 1. Device credential list 2. Device credential ref list 3. Device credential entry All three ways can be used in the same Configuration, the order above is the order of Onvif Discovery Handler processing the secret information. If there is any secret information duplication between different groups, the latter overwrites the prior entries. If there is any duplication within the same group, it’s up to the Onvif Discovery Handler to decide which one wins when processing the entries, and it’s not guaranteed the order is always the same. **Device credential list** Here is an example of Device credential list. In Configuration, an entry named “`device_credential_list`” is listed in discoveryProperties. The value contains an array of device secret lists. The device secret lists are entries that point to the actual Kubernetes Secret key. ```yaml discoveryProperties: - name: "device_credential_list" value: |+ [ "secret_list1", "secret_list2" ] - name: "secret_list1" valueFrom: secretKeyRef: name: "onvif-auth-secret" namespace: "onvif-auth-secret-namespace" key: "secret_list1" optional: false - name: "secret_list2" valueFrom: secretKeyRef: name: "onvif-auth-secret" namespace: "onvif-auth-secret-namespace" key: "secret_list2" optional: false ``` In Kubernetes Secret `onvif-auth-secret`, the `secret_list1` and `secret_list2` contain the actual secret information for a list of devices. The entry uses the device id as key and the value is a json object with username and password. The password can be optionally encoded with base64 (with “`base64encoded`” set to true). ```yaml apiVersion: v1 kind: Secret metadata: name: onvif-auth-secret namespace: onvif-auth-secret-namespace type: Opaque stringData: secret_list1: |+ { "6821dc67-8438-5588-1547-4d1349048438" : { "username" : "admin", "password" : "adminpassword" }, "6a67158b-42b1-400b-8afe-1bec9a5d7919" : { "username" : "user1", "password" : "SGFwcHlEYXk=", "base64encoded": true } } secret_list2: |+ { "5f5a69c2-e0ae-504f-829b-00fcdab169cc" : { "username" : "admin", "password" : "admin" } } ``` **Device credential ref list** Device credential ref list is similar to Device credential list except the device ids are listed and the credentials are references to another entries in the Akri `discoveryProperties`. The key name for device credential ref list is “`device_credential_ref_list`”. For example, the device credential ref list below contains an array of “device id”->”credential reference” objects. The credential of device id “5f5a69c2-e0ae-504f-829b-00fcdab169cc” is referred to (username-> device1\_username, password->device1\_password). The device1\_username and device1\_password are entries in Akri discoverProperties that point to the actual secret information in Kubernetes Secrets. Note different device ids may use the same secret reference. ```yaml - name: "device_credential_ref_list" value: |+ [ "secret_ref_list1", "secret_ref_list2" ] - name: "secret_ref_list1" value: |+ { "5f5a69c2-e0ae-504f-829b-00fcdab169cc" : { "username_ref" : "device1_username", "password_ref" : "device1_password" }, "6a67158b-42b1-400b-8afe-1bec9a5d7909": { "username_ref" : "device2_username", "password_ref" : "device2_password" } } - name: "secret_ref_list2" value: |+ { "7a67158b-42b1-400b-8afe-1bec9a5d790a": { "username_ref" : "device2_username", "password_ref" : "device2_password" } } - name: "device1_username" valueFrom: secretKeyRef: name: "onvif-auth-secret" namespace: "onvif-auth-secret-namespace" key: "device1_username" optional: false - name: "device1_password" valueFrom: secretKeyRef: name: "onvif-auth-secret" namespace: "onvif-auth-secret-namespace" key: "device1_password" optional: true - name: "device2_username" valueFrom: secretKeyRef: name: "onvif-auth-secret" namespace: "onvif-auth-secret-namespace" key: "device2_username" optional: false - name: "device2_password" valueFrom: secretKeyRef: name: "onvif-auth-secret" namespace: "onvif-auth-secret-namespace" key: "device2_password" optional: true ``` The actual secret information is in Kubernetes Secret `onvif-auth-secret` ```yaml apiVersion: v1 kind: Secret metadata: name: onvif-auth-secret namespace: onvif-auth-secret-namespace type: Opaque stringData: device1_username: "admin" device1_password: "admin" device2_username: "cam2_user" device2_password: "cam2_pwd" ``` **Device credential entry** Device credential entry is a direct mapping from device id to its credential, using "`username_`" and "`password_`" as key names, note that `device_id` is in uuid string format, need to convert to C\_IDENTIFIER format for use it in `discoveryProperties` key name. In addition to the "`username_`" and "`password_`" keys, Onvif Discovery Handler looks for two specific key names "`username_default`" and "`password_default`" that, if specified, Onvif Discovery Handler uses it as a fall back username/password value. If Onvif Discovery Handler cannot find a match credential by looking up the device id, and "`username_default`"/"`password_default`" are specified, Onvif Discovery Handler uses the default username/password to authenticate the device discovery. ```yaml discoveryProperties: - name: "username_6a67158b_42b1_400b_8afe_1bec9a5d7909" valueFrom: secretKeyRef: name: "onvif-auth-secret" namespace: "onvif-auth-secret-namespace" key: "username_6a67158b_42b1_400b_8afe_1bec9a5d7909" optional: false - name: "password_6a67158b_42b1_400b_8afe_1bec9a5d7909" valueFrom: secretKeyRef: name: "onvif-auth-secret" namespace: "onvif-auth-secret-namespace" key: "password_6a67158b_42b1_400b_8afe_1bec9a5d7909" optional: false ``` The actual secret information is in Kubernetes Secret `onvif-auth-secret` ```yaml apiVersion: v1 kind: Secret metadata: name: onvif-auth-secret namespace: onvif-auth-secret-namespace type: Opaque stringData: username_6a67158b_42b1_400b_8afe_1bec9a5d7909: "admin" password_6a67158b_42b1_400b_8afe_1bec9a5d7909: "admin" ``` ### Installing Akri with the ONVIF Configuration and Discovery Handler Leveraging the above settings, Akri can be installed with the ONVIF Discovery Handler and an ONVIF Configuration that specifies the Akri frame server broker: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set onvif.discovery.enabled=true \ --set onvif.configuration.enabled=true \ --set onvif.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/onvif-video-broker" \ --set onvif.configuration.brokerPod.image.tag="latest" ``` No discovery details were provided to filter the Discovery Handler's search so the Discovery Handler will find any ONVIF camera on the network and will deploy up to one broker pod to each camera, since `capacity` defaults to one. The brokers will supply the automatically created Instance Services and the Configuration Service with frames. The following installation examples have been given to show how to the ONVIF Configuration can be tailored to you cluster: * Filtering ONVIF cameras * Changing the discovery timeout #### Filtering ONVIF cameras The ONVIF Discovery Handler supports basic filter capabilities has been provided. Discovery details can be set in the Configuration that tell the Discovery Handler to either include or exclude specific IP addresses, MAC addresses, ONVIF scopes, or device uuids. For example, the following enables discovery of every camera that does not have an IP address of 10.0.0.1: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set onvif.discovery.enabled=true \ --set onvif.configuration.enabled=true \ --set onvif.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/onvif-video-broker" \ --set onvif.configuration.discoveryDetails.ipAddresses.action=Exclude \ --set onvif.configuration.discoveryDetails.ipAddresses.items[0]=10.0.0.1 ``` The following enables discovery of every camera that uuid is not `3fa1fe68-b915-4053-a3e1-ac15a21f5f91`: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set onvif.discovery.enabled=true \ --set onvif.configuration.enabled=true \ --set onvif.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/onvif-video-broker" \ --set onvif.configuration.discoveryDetails.uuids.action=Exclude \ --set onvif.configuration.discoveryDetails.uuids.items[0]="3fa1fe68-b915-4053-a3e1-ac15a21f5f91" ``` You can enable cluster access for every camera with a specific name, you can modify the Configuration like so: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set onvif.discovery.enabled=true \ --set onvif.configuration.enabled=true \ --set onvif.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/onvif-video-broker" \ --set onvif.configuration.discoveryDetails.scopes.action=Include \ --set onvif.configuration.discoveryDetails.scopes.items[0]="onvif://www.onvif.configuration.org/name/GreatONVIFCamera" \ --set onvif.configuration.discoveryDetails.scopes.items[1]="onvif://www.onvif.configuration.org/name/AwesomeONVIFCamera" ``` #### Changing the discovery timeout The ONVIF Discovery Handler will search for up to `discoveryTimeoutSeconds` for IP cameras. This timeout can be increased or decreased as desired, and defaults to 1 second if not configured. It can be set in the Configuration like this: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set onvif.discovery.enabled=true \ --set onvif.configuration.enabled=true \ --set onvif.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/onvif-video-broker" \ --set onvif.configuration.discoveryDetails.discoveryTimeoutSeconds=2 ``` ### Accessing Secret data in Broker The Onvif sample broker (`akri-onvif-video-broker`) can be configured to access Secret and ConfigMap data, if configured, it expects the Secret and ConfigMap data are mounted as files. The sample broker checks the environment variables `CREDENTIAL_DIRECTORY` for the directory that contains Secret data and `CREDENTIAL_CONFIGMAP_DIRECTORY` for directory contains configMap data. `CREDENTIAL_CONFIGMAP_DIRECTORY` is optional. When the sample broker launched, the uuids of discovered Onvif devices are set in the environment variables `ONVIF_DEVICE_UUID_{INSTANCE_HASH_ID}`, the sample broker picks the first one found from the environment variables `ONVIF_DEVICE_UUID_{INSTANCE_HASH_ID}` as device uuid and get match credential from files under `CREDENTIAL_DIRECTORY` and `CREDENTIAL_CONFIGMAP_DIRECTORY`. The schema of how the Secret/ConfigMap files are organized aligned to the schema that Onvif Discovery Handler used for passing the secret data, as follow: 1. The sample broker first looks for files under `CREDENTIAL_DIRECTORY` with file name "username\*\" for username and "password\*\" for password, where device\_uuid is the device uuid string that replaces all '-' with underscore '*'. If file "username\*\" exists, but "password\*\" doesn't exist, the sample broker uses empty string as password. The sample broker also looks for files "username\_default" and "password\_default", if found, the username and password is used as fallback username/password if a credential with matched device uuid can not be found. 2. If the sample broker can not find a matched credential from the username/password secret files directly, it looks for credentials from the device credential ref list `device_credential_ref_list`. The sample broker try to get a file name `device_credential_ref_list` from `CREDENTIAL_CONFIGMAP_DIRECTORY`, and if the file does not exists, it tries to get the same file name under `CREDENTIAL_DIRECTORY`. The credential ref list should contains reference entries to the actual credentals. The sample broker look up the device uuid from the list to get the credential reference and read the actual credential files from `CREDENTIAL_DIRECTORY`. Similar to the fallback credenial "username\_default" and "password\_default", a credential ref entry with key "default" indicates the fallback credental. 3. if the sample broker can not find a matched credental from credential ref list, it looks for credentials from crdential list `device_credential_list`. The sample broker try to get a file name `device_credential_list` from `CREDENTIAL_CONFIGMAP_DIRECTORY`, and if the file does not exist, it tries to get the same file name under `CREDENTIAL_DIRECTORY`. The credential list should contains actual credential entries. The sample broker look up the device uuid from the list to get the credential. Similarly, a credential entry with key "default" indicates the fallback credental. The following example shows how the credential information is organized in Secret and ConfigMap. There are 4 credentials specified in this example, credential for device id "5f5a69c2-e0ae-504f-829b-00fcdab169cc", "3fa1fe68-b915-4053-a3e1-ac15a21f5f91", "6a67158b-42b1-400b-8afe-1bec9a5d7909" and a fallback credential "username\_default"/"password\_default". ```yaml apiVersion: v1 kind: ConfigMap metadata: name: onvif-auth-configmap data: device_credential_ref_list: |+ [ "credential_ref_list1" ] credential_ref_list1: |+ { "5f5a69c2-e0ae-504f-829b-00fcdab169cc": { "username_ref" : "device2_username", "password_ref" : "device2_password" } } --- apiVersion: v1 kind: Secret metadata: name: onvif-auth-secret type: Opaque stringData: device_credential_list: |+ [ "credential_list" ] credential_list: |+ { "3fa1fe68-b915-4053-a3e1-ac15a21f5f91" : { "username" : "user1", "password" : "SGFwcHlEYXk=", "base64encoded": true } } username_6a67158b_42b1_400b_8afe_1bec9a5d7909: "admin" password_6a67158b_42b1_400b_8afe_1bec9a5d7909: "admin" device2_username: "user2" device2_password: "abcde" username_default: "user1" password_default: "12345" ``` ## Modifying a Configuration Akri has provided further documentation on [modifying the broker PodSpec](/user-guide/customizing-an-akri-installation#modifying-the-brokerpodspec), [instanceServiceSpec, or configurationServiceSpec](/user-guide/customizing-an-akri-installation#modifying-instanceservicespec-or-configurationservicespec) More information about how to modify an installed Configuration, add additional Configurations to a cluster, or delete a Configuration can be found in the [Customizing an Akri Installation document](/user-guide/customizing-an-akri-installation). ## Implementation details The ONVIF implementation can be understood by looking at several things: 1. [OnvifDiscoveryDetails](https://github.com/project-akri/akri/blob/main/discovery-handlers/onvif/src/discovery_handler.rs) defines the required properties 2. [OnvifDiscoveryHandler](https://github.com/project-akri/akri/blob/main/discovery-handlers/onvif/src/discovery_handler.rs) defines ONVIF camera discovery 3. [samples/brokers/onvif-video-broker](https://github.com/project-akri/examples/tree/main/brokers/onvif-video-broker) defines the ONVIF broker # OPC UA ## Background OPC UA (Open Platform Communications Unified Architecture) is a communication protocol for industrial automation. Akri has implemented a Discovery Handler for discovering OPC UA Servers that live at specified endpoints or are registered with specified Local Discovery Servers. Background on the OPC UA Discovery Handler implementation can be found in the [proposal](https://github.com/deislabs/akri-docs/blob/main/proposals/opcua.md). To try out using Akri to discover and utilize OPC UA servers, see the [OPC UA end-to-end demo](/demos/opc-thermometer-demo). All of Akri's components can be deployed by specifying values in its Helm chart during an installation. This section will cover the values that should be set to (1) deploy the OPC UA Discovery Handlers and (2) apply a Configuration that tells Akri to discover devices using that Discovery Handler. ## Deploying the OPC UA Discovery Handler In order for the Agent to know how to discover OPC UA servers an OPC UA Discovery Handler must exist. Akri supports an Agent image that includes all supported Discovery Handlers. This Agent will be used if `agent.full=true`. By default, a slim Agent without any embedded Discovery Handlers is deployed and the required Discovery Handlers can be deployed as DaemonSets. This documentation will use that strategy, deploying OPC UA Discovery Handlers by specifying `opcua.discovery.enabled=true` when installing Akri. ## OPC UA Configuration Settings Instead of having to assemble your own OPC UA Configuration yaml, we have provided a [Helm template](https://github.com/project-akri/akri/blob/main/deployment/helm/templates/opcua-configuration.yaml). Helm allows us to parametrize the commonly modified fields in our configuration files, and we have provided many for OPC UA (to see them, run `helm inspect values akri-helm-charts/akri`). More information about the Akri Helm charts can be found in the [user guide](/user-guide/getting-started#understanding-akri-helm-charts). To apply the OPC UA Configuration to your cluster, simply set `opcua.configuration.enabled=true` along with any of the following additional Configuration settings when installing Akri. ### Discovery Handler Discovery Details Settings Discovery Handlers are passed discovery details that are set in a Configuration to determine what to discover, filter out of discovery, and so on. The OPC UA Discovery Handler, requires a set of DiscoveryURLs to direct its search. Every OPC UA server/application has a DiscoveryEndpoint that Clients can access without establishing a session. The address for this endpoint is defined by a DiscoveryURL. A Local Discovery Server (LDS) is a unique type of OPC UA server which maintains a list of OPC UA servers that have registered with it. The generic OPC UA Configuration takes in a list of DiscoveryURLs, whether for LDSes or a specific servers and an optional list of application names to either include or exclude. By default, if no DiscoveryURLs are set, the Discovery Handler will attempt to reach out to the Local Discovery Server on its host at the default address [from OPC UA Specification 12](https://reference.opcfoundation.org/v104/Core/docs/Part6/7.6/) of `opc.tcp://localhost:4840/` and get the list of OPC UA servers registered with it. | Helm Key | Value | Default | Description | | ------------------------------------------------------------ | -------------------------- | ------------------------------- | ----------------------------------------------------------- | | opcua.configuration.discoveryDetails.discoveryUrls | array of DiscoveryURLs | \["opc.tcp\://localhost:4840/"] | DiscoveryURLs for OPC UA Servers or Local Discovery Servers | | opcua.configuration.discoveryDetails.applicationNames.action | Include, Exclude | Exclude | filter action to take on a set of OPC UA Applications | | opcua.configuration.discoveryDetails.applicationNames.items | array of application names | empty | application names that the filter action acts upon | ### Broker Pod Settings If you would like non-terminating workloads ("broker" Pods) to be deployed automatically to discovered devices, a broker image should be specified (under `brokerPod`) in the Configuration. Alternatively, if it meets your scenario, you could use the Akri frame server broker ("ghcr.io/project-akri/examples/opcua-monitoring-broker"). If you would rather manually deploy pods to utilize the devices advertized by Akri, don't specify a broker pod and see our documentation on [requesting resources advertized by Akri](/user-guide/requesting-akri-resources). > Note only a `brokerJob` OR `brokerPod` should be specified. | Helm Key | Value | Default | Description | | ----------------------------------------------------- | ------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | opcua.configuration.brokerPod.image.repository | image string | "" | image of broker Pod that should be deployed to discovered devices | | opcua.configuration.brokerPod.image.tag | tag string | "latest" | image tag of broker Pod that should be deployed to discovered devices | | opcua.configuration.brokerPod.resources.memoryRequest | string | "76Mi" | the minimum amount of RAM that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri OPC UA sample broker. Adjust to the size of your broker. | | opcua.configuration.brokerPod.resources.cpuRequest | string | "9m" | the minimum amount of CPU that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri OPC UA sample broker. Adjust to the size of your broker. | | opcua.configuration.brokerPod.resources.memoryLimit | string | "200Mi" | the maximum amount of RAM this Pod can consume. Default based on the Akri OPC UA sample broker. Adjust to the size of your broker. | | opcua.configuration.brokerPod.resources.cpuLimit | string | "30m" | the maximum amount of CPU this Pod can consume. Default based on the Akri OPC UA sample broker. Adjust to the size of your broker. | ### Broker Job Settings If you would like terminating [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) to be deployed automatically to discovered servers, a broker image should be specified (under `brokerJob`) in the Configuration. A Kubernetes Job deploys a set number of terminating Pods. > Note only a `brokerJob` OR `brokerPod` should be specified. | Helm Key | Value | Default | Description | | ----------------------------------------------------- | ------------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | opcua.configuration.brokerJob.image.repository | image string | "" | image of broker Job that should be deployed to discovered devices | | opcua.configuration.brokerJob.image.tag | tag string | "latest" | image tag of broker Job that should be deployed to discovered devices | | opcua.configuration.brokerJob.resources.memoryRequest | string | "76Mi" | the minimum amount of RAM that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri OPC UA sample broker. Adjust to the size of your broker. | | opcua.configuration.brokerJob.resources.cpuRequest | string | "9m" | the minimum amount of CPU that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri OPC UA sample broker. Adjust to the size of your broker. | | opcua.configuration.brokerJob.resources.memoryLimit | string | "200Mi" | the maximum amount of RAM this Pod can consume. Default based on the Akri OPC UA sample broker. Adjust to the size of your broker. | | opcua.configuration.brokerJob.resources.cpuLimit | string | "30m" | the maximum amount of CPU this Pod can consume. Default based on the Akri OPC UA sample broker. Adjust to the size of your broker. | | opcua.configuration.brokerJob.command | string array | Empty | command to be executed in the Pod | | opcua.configuration.brokerJob.restartPolicy | string array | `OnFailure` | `RestartPolicy` for the Job. Can either be `OnFailure` or `Never` for Jobs. | | opcua.configuration.brokerJob.backoffLimit | number | 2 | defines the Kubernetes Job [backoff failure policy](https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy) | | opcua.configuration.brokerJob.parallelism | number | 1 | defines the Kubernetes Job [`parallelism`](https://kubernetes.io/docs/concepts/workloads/controllers/job/#parallel-jobs) | | opcua.configuration.brokerJob.completions | number | 1 | defines the Kubernetes Job [`completions`](https://kubernetes.io/docs/concepts/workloads/controllers/job) | ### Mounting Credentials Settings See [Mounting OPC UA credentials to enable security](#mounting-opc-ua-credentials-to-enable-security) for more details on how to use this setting. | Helm Key | Value | Default | Description | | ------------------------------------- | ----------- | ------- | ------------------------------------------------------------------------------------------ | | opcua.configuration.mountCertificates | true, false | false | specify whether to mount a secret named `opcua-broker-credentials` into the OPC UA brokers | ### Disabling Automatic Service Creation By default, if a broker Pod is specified, the generic OPC UA Configuration will create services for all the brokers of a specific Akri Instance and all the brokers of an Akri Configuration. The creation of these services can be disabled. | Helm Key | Value | Default | Description | | ---------------------------------------------- | ----------- | ------- | --------------------------------------------------------------------- | | opcua.configuration.createInstanceServices | true, false | true | a service should be automatically created for each broker Pod | | opcua.configuration.createConfigurationService | true, false | true | a single service should be created for all brokers of a Configuration | ### Capacity Setting By default, if a broker Pod is specified, a single broker Pod is deployed to each device. To modify the Configuration so that an OPC UA server is accessed by more or fewer nodes via broker Pods, update the `opcua.configuration.capacity` setting to reflect the correct number. For example, if your high availability needs are met by having 1 redundant pod, you can update the Configuration like this by setting `opcua.configuration.capacity=2`. | Helm Key | Value | Default | Description | | ---------------------------- | ------ | ------- | ------------------------------------------------------------------------------------- | | opcua.configuration.capacity | number | 1 | maximum number of brokers that can be deployed to utilize a device (up to 1 per Node) | ### Installing Akri with the OPC UA Configuration and Discovery Handler Leveraging the above settings, Akri can be installed with the OPC UA Discovery Handler and an OPC UA Configuration that specifies discovery via the default LDS DiscoveryURL: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true ``` If you have a workload that you would like to automatically be deployed to each discovered server, specify the workload image when installing Akri. As an example, the installation below will deploy an empty nginx pod for each server. Instead, you should point to your image, say `ghcr.io//opcua-broker`. ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.brokerPod.image.repository=nginx ``` > Note: set `opcua.configuration.brokerPod.image.tag` to specify an image tag (defaults to `latest`). The following installation examples have been given to show how to the OPC UA Configuration can be tailored to you cluster: * Specifying the DiscoveryURLs for OPC UA Local Discovery Servers * Specifying the DiscoveryURLs for specific OPC UA servers * Specifying the DiscoveryURLs for both Local Discovery Servers and servers * Filtering the servers by application name * Mounting OPC UA credentials to enable security ### Specifying the DiscoveryURLs for OPC UA LocalDiscoveryServers If no DiscoveryURLs are passed as Helm values, the default DiscoveryURL for LocalDiscoveryServers is used. Instead of using the default `opc.tcp://localhost:4840/` LDS DiscoveryURL, an operator can specify the addresses of one or more Local Discovery Servers, like in the following example: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.discoveryDetails.discoveryUrls[0]="opc.tcp://10.1.2.3:4840/" \ --set opcua.configuration.discoveryDetails.discoveryUrls[1]="opc.tcp://10.1.3.4:4840/" ``` ### Specifying the DiscoveryURLs for specific OPC UA Servers If you know the DiscoveryURLs for the OPC UA Servers you want Akri to discover, manually list them when deploying Akri, like in the following: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.discoveryDetails.discoveryUrls[0]="opc.tcp://10.123.456.7:4855/" ``` ### Specifying the DiscoveryURLs for both LocalDiscoveryServers and Servers OPC UA discovery can also receive a list of both OPC UA LDS DiscoveryURLs and specific Server urls, as in the following. ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.discoveryDetails.discoveryUrls[0]="opc.tcp://10.1.2.3:4840/" \ --set opcua.configuration.discoveryDetails.discoveryUrls[1]="opc.tcp://10.1.3.4:4840/" \ --set opcua.configuration.discoveryDetails.discoveryUrls[2]="opc.tcp://10.123.456.7:4855/" ``` > **Note**: The Agent's OPC UA discovery method only supports tcp DiscoveryURLs, since the [Rust OPC UA library](https://github.com/locka99/opcua) has yet to support http(s). ### Filtering the Servers by application name Instead of discovering all servers registered with specified Local Discovery Servers, you can choose to include or exclude a list of application names (the `applicationName` property of a server's `ApplicationDescription` as specified by [OPC UA Specification](https://reference.opcfoundation.org/v104/Core/DataTypes/ApplicationDescription/)). For example, to discover all servers registered with the default LDS except for the server named "Duke", do the following. ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.discoveryDetails.applicationNames.action=Exclude \ --set opcua.configuration.discoveryDetails.applicationNames.items[0]="Duke" ``` Alternatively, to only discover the server named "Go Tar Heels!", do the following: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.discoveryDetails.applicationNames.action=Include \ --set opcua.configuration.discoveryDetails.applicationNames.items[0]="Go Tar Heels!" ``` ### Mounting OPC UA credentials to enable security For your broker pod to utilize a discovered OPC UA server, it will need to contain an OPC UA Client. OPC UA Clients and Servers can establish an insecure connection so long as the OPC UA Servers support a Security Policy of None. However, if you would like your broker's OPC UA Client to establish a secure connection with an OPC UA server, the Client and Server must trust each other's x509 v3 certificates. This can be done in one of the three ways explained in the [OPC UA proposal](https://github.com/deislabs/akri-docs/blob/main/proposals/opcua.md#giving-proper-credentials-to-the-akri-broker). The simplest method is to sign the OPC UA broker's certificate with the same Certificate Authority (CA) as the Server with which it wishes to connect. The certificates are passed to the broker via a Kubernetes Secret mounted as a volume to the directory `/etc/opcua-certs/client-pki`. It is the operator's responsibility to generate the certificates and securely create a Kubernetes Secret named `opcua-broker-credentials`, ideally using a KMS. More information about using Kubernetes Secrets securely can be found in the [credentials passing proposal](https://github.com/deislabs/akri-docs/blob/main/proposals/credentials-passing.md). The following is an example kubectl command to create the Kubernetes Secret, projecting each certificate/crl/private key with the expected key name (ie `client_certificate`, `client_key`, `ca_certificate`, and `ca_crl`). ```bash kubectl create secret generic opcua-broker-credentials \ --from-file=client_certificate=/path/to/AkriBroker.der \ --from-file=client_key=/path/to/AkriBroker.pfx \ --from-file=ca_certificate=/path/to/SomeCA.der \ --from-file=ca_crl=/path/to/SomeCA.crl ``` Certificates can be created and signed with a CA manually using openssl, or by using the OPC Foundation [certificate generator tool](https://github.com/OPCFoundation/Misc-Tools). Be sure that the certificates are in the format expected by your OPC UA Client. Finally, when mounting certificates is enabled with Helm via `--set opcua.configuration.mountCertificates='true'`, the secret named `opcua-broker-credentials` will be mounted into the OPC UA brokers. It is mounted to the volume `credentials` at the `mountPath` /etc/opcua-certs/client-pki, as shown in the [OPC UA Helm template](https://github.com/project-akri/akri/blob/main/deployment/helm/templates/opcua-configuration.yaml). This is the path where the broker expects to find the certificates. The following is an example how to enable security: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.mountCertificates='true' ``` > **Note**: If the Helm template for the OPC UA Configuration is too specific, you can [customize the Configuration yaml](/user-guide/customizing-an-akri-installation#generating-modifying-and-applying-a-custom-configuration) to suit your needs. ## Modifying a Configuration Akri has provided further documentation on [modifying the broker PodSpec](/user-guide/customizing-an-akri-installation#modifying-the-brokerpodspec), [instanceServiceSpec, or configurationServiceSpec](/user-guide/customizing-an-akri-installation#modifying-instanceservicespec-or-configurationservicespec) More information about how to modify an installed Configuration, add additional Configurations to a cluster, or delete a Configuration can be found in the [Customizing an Akri Installation document](/user-guide/customizing-an-akri-installation). ## Implementation details The OPC UA implementation can be understood by looking at several things: 1. [OpcuaDiscoveryDetails](https://github.com/project-akri/akri/blob/main/discovery-handlers/opcua/src/discovery_handler.rs) defines the required properties. 2. [OpcuaDiscoveryHandler](https://github.com/project-akri/akri/blob/main/discovery-handlers/opcua/src/discovery_handler.rs) defines OPC UA Server discovery. 3. [sample-brokers/opcua-monitoring-broker](https://github.com/project-akri/examples/tree/main/brokers/opcua-monitoring-broker) defines a sample OPC UA protocol broker that monitors an OPC UA Variable with a specific NodeID. # udev ## Background Udev is the device manager for Linux. It manages device nodes in the `/dev` directory, such as microphones, security chips, usb cameras, and so on. Udev can be used to find devices that are attached to or embedded in Linux nodes. All of Akri's components can be deployed by specifying values in its Helm chart during an installation. This document will cover the values that should be set to (1) deploy the udev Discovery Handlers and (2) apply a Configuration that tells Akri to discover devices using that Discovery Handler. ## Deploying the udev Discovery Handler In order for the Agent to discover udev devices, a udev Discovery Handler must exist. Akri supports an Agent image that includes all supported Discovery Handlers. This Agent will be used if `agent.full=true` is set. By default, a slim Agent without any embedded Discovery Handlers is deployed and the required Discovery Handlers can be deployed as DaemonSets. This documentation will use that strategy, deploying udev Discovery Handlers by specifying `udev.discovery.enabled=true` when installing Akri. ## udev Configuration Settings Instead of having to assemble your own udev Configuration yaml, we have provided a [Helm template](https://github.com/project-akri/akri/blob/main/deployment/helm/templates/udev-configuration.yaml). Helm allows us to parametrize the commonly modified fields in our configuration files, and we have provided many for udev (to see them, run `helm inspect values akri-helm-charts/akri`). To apply the udev Configuration to your cluster, simply set `udev.configuration.enabled=true` when installing Akri. Be sure to also **specify one or more udev rules** for the Configuration, as explained [below](#discovery-handler-discovery-details-settings). ### Discovery Handler Discovery Details Settings Discovery Handlers are passed discovery details that are set in a Configuration to determine what to discover, filter out of discovery, and so on. The udev Discovery Handler requires that one discovery detail to be provided: [udev rules](https://wiki.archlinux.org/index.php/Udev). | Helm Key | Value | Default | Description | | -------------------------------------------------- | ------------------- | ------- | ---------------------------------------------------------------------- | | udev.configuration.discoveryDetails.udevRules | array of udev rules | empty | udev rule [supported by the udev Discovery Handler](#udev-rule-format) | | udev.configuration.discoveryDetails.groupRecursive | boolean | false | If set to true, group devices with a matching parent | | udev.configuration.discoveryDetails.permissions | string | rwm | udev device mount permissions (**r**ead, **w**rite, **m**odify) | The udev Discovery Handler parses the udev rules listed in a Configuration, searches for them using udev, and returns a list of discovered device nodes (ie: /dev/video0). It parses the udev rules via a grammar [grammar](https://github.com/project-akri/akri/blob/main/discovery-handlers/udev/src/udev_rule_grammar.pest) Akri has created. It expects the udev rules to be formatted according to the [Linux Man pages](https://linux.die.net/man/7/udev). Additionally, the permissions for the device mount are defined as well and are set to `rwm` (read, write, modify) by default. #### Udev rule format While udev rules are normally used to both find devices and perform actions on devices, the Akri udev discovery handler is only interested in finding devices. Consequently, the discovery handler will throw an error if any of the rules contain an action operation ("=" , "+=" , "-=" , ":=") or action fields such as `IMPORT` in the udev rules. You should only use match operations ("==", "!=") and the following udev fields: `ATTRIBUTE`, `ATTRIBUTE`, `DEVPATH`, `DRIVER`, `DRIVERS`, `KERNEL`, `KERNELS`, `ENV`, `SUBSYSTEM`, `SUBSYSTEMS`, `TAG`, and `TAGS`. To see some examples, reference our example [supported rules](https://github.com/project-akri/akri/blob/main/test/example.rules) and [unsupported rules](https://github.com/project-akri/akri/blob/main/test/example-unsupported.rules) that we run some tests against. ### Broker Pod Settings If you would like non-terminating workloads ("broker" Pods) to be deployed automatically to discovered cameras, a broker image should be specified (under `brokerPod`) in the Configuration. Alternatively, if it meets your scenario, you could use the Akri frame server broker ("ghcr.io/project-akri/examples/udev-video-broker"). If you would rather manually deploy pods to utilize the cameras advertized by Akri, don't specify a broker pod and see our documentation on [requesting resources advertized by Akri](/user-guide/requesting-akri-resources). > Note only a `brokerJob` OR `brokerPod` should be specified. | Helm Key | Value | Default | Description | | ---------------------------------------------------- | ------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | udev.configuration.brokerPod.image.repository | image string | "" | image of broker Pod that should be deployed to discovered devices | | udev.configuration.brokerPod.image.tag | tag string | "latest" | image tag of broker Pod that should be deployed to discovered devices | | udev.configuration.brokerPod.resources.memoryRequest | string | "10Mi" | the minimum amount of RAM that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri udev sample broker. Adjust to the size of your broker. | | udev.configuration.brokerPod.resources.cpuRequest | string | "10m" | the minimum amount of CPU that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri udev sample broker. Adjust to the size of your broker. | | udev.configuration.brokerPod.resources.memoryLimit | string | "30Mi" | the maximum amount of RAM this Pod can consume. Default based on the Akri udev sample broker. Adjust to the size of your broker. | | udev.configuration.brokerPod.resources.cpuLimit | string | "29m" | the maximum amount of CPU this Pod can consume. Default based on the Akri udev sample broker. Adjust to the size of your broker. | ### Broker Job Settings If you would like terminating [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) to be deployed automatically to discovered devices, a broker image should be specified (under `brokerJob`) in the Configuration. A Kubernetes Job deploys a set number of terminating Pods. > Note only a `brokerJob` OR `brokerPod` should be specified. | Helm Key | Value | Default | Description | | ---------------------------------------------------- | ------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | udev.configuration.brokerJob.image.repository | image string | "" | image of broker Job that should be deployed to discovered devices | | udev.configuration.brokerJob.image.tag | tag string | "latest" | image tag of broker Job that should be deployed to discovered devices | | udev.configuration.brokerJob.resources.memoryRequest | string | "10Mi" | the minimum amount of RAM that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri udev sample broker. Adjust to the size of your broker. | | udev.configuration.brokerJob.resources.cpuRequest | string | "10m" | the minimum amount of CPU that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri udev sample broker. Adjust to the size of your broker. | | udev.configuration.brokerJob.resources.memoryLimit | string | "30Mi" | the maximum amount of RAM this Pod can consume. Default based on the Akri udev sample broker. Adjust to the size of your broker. | | udev.configuration.brokerJob.resources.cpuLimit | string | "29m" | the maximum amount of CPU this Pod can consume. Default based on the Akri udev sample broker. Adjust to the size of your broker. | | udev.configuration.brokerJob.command | string array | Empty | command to be executed in the Pod | | udev.configuration.brokerJob.restartPolicy | string array | `OnFailure` | `RestartPolicy` for the Job. Can either be `OnFailure` or `Never` for Jobs. | | udev.configuration.brokerJob.backoffLimit | number | 2 | defines the Kubernetes Job [backoff failure policy](https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy) | | udev.configuration.brokerJob.parallelism | number | 1 | defines the Kubernetes Job [`parallelism`](https://kubernetes.io/docs/concepts/workloads/controllers/job/#parallel-jobs) | | udev.configuration.brokerJob.completions | number | 1 | defines the Kubernetes Job [`completions`](https://kubernetes.io/docs/concepts/workloads/controllers/job) | ### Disabling Automatic Service Creation By default, if a broker Pod is specified, the generic udev Configuration will create services for all the brokers of a specific Akri Instance and all the brokers of an Akri Configuration. The creation of these services can be disabled. | Helm Key | Value | Default | Description | | --------------------------------------------- | ----------- | ------- | --------------------------------------------------------------------- | | udev.configuration.createInstanceServices | true, false | true | a service should be automatically created for each broker Pod | | udev.configuration.createConfigurationService | true, false | true | a single service should be created for all brokers of a Configuration | ### Capacity Setting By default, if a broker Pod is specified, a single broker Pod is deployed to each device. To modify the Configuration so that a device is accessed by more or fewer nodes via broker Pods, update the `udev.configuration.capacity` setting to reflect the correct number. For example, if your high availability needs are met by having 1 redundant pod, you can update the Configuration like this by setting `udev.configuration.capacity=2`. | Helm Key | Value | Default | Description | | --------------------------- | ------ | ------- | ------------------------------------------------------------------------------------- | | udev.configuration.capacity | number | 1 | maximum number of brokers that can be deployed to utilize a device (up to 1 per Node) | ## Choosing a udev rule To see what devices will be discovered on a specific node by a udev rule, you can use `udevadm`. For example, to find all devices in the sound subsystem, you could run: ```bash udevadm trigger --verbose --dry-run --type=devices --subsystem-match=sound ``` To see all the properties of a specific device discovered, you can use `udevadm info`: ```bash udevadm info --attribute-walk --path=$(udevadm info --query=path /sys/devices/pci0000:00/0000:00:1f.3/sound/card0) ``` Now, you can see a bunch of attributes you could use to narrow your udev rule. Maybe you decide you want to find all sound devices made by the vendor `Great Vendor`. You set the following udev rule under the udev Discovery Handler in your Configuration: ```yaml discoveryHandler: name: udev discoveryDetails: |+ udevRules: - 'SUBSYSTEM=="sound", ATTR{vendor}=="Great Vendor"' ``` ### Testing a udev rule To test which devices Akri will discover with a udev rule, you can run the rule locally adding a tag action to it. Then you can search for all devices with that tag, which will be the ones discovered by Akri. 1. Create a new rules file called `90-akri.rules` in the `/etc/udev/rules.d` directory, and add your udev rule(s) to it. For this example, we will be testing the rule `SUBSYSTEM=="sound", KERNEL=="card[0-9]*"`. Add `TAG+="akri_tag"` to the end of each rule. Note how 90 is the prefix to the file name. This makes sure these rules are run after the others in the default `70-snap.core.rules`, preventing them from being overwritten. Feel free to explore `70-snap.core.rules` to see numerous examples of udev rules. ```bash sudo echo 'SUBSYSTEM=="sound", KERNEL=="card[0-9]*", TAG+="akri_tag"' | sudo tee -a /etc/udev/rules.d/90-akri.rules ``` 1. Reload the udev rules and trigger them. ```bash sudo udevadm control --reload sudo udevadm trigger ``` 2. List the devices that have been tagged, which Akri will discover. Akri will only discover devices with device nodes (devices within the `/dev` directory). These device node paths will be mounted into broker Pods so the brokers can utilize the devices. ```bash udevadm trigger --verbose --dry-run --type=devices --tag-match=akri_tag | xargs -l bash -c 'if [ -e $0/dev ]; then echo $0/dev; fi' ``` 3. Explore the attributes of each device in order to decide how to refine your udev rule. ```bash udevadm trigger --verbose --dry-run --type=devices --tag-match=akri_tag | xargs -l bash -c 'if [ -e $0/dev ]; then echo $0; fi' | xargs -l bash -c 'udevadm info --path=$0 --attribute-walk' | less ``` 4. Modify the rule as needed, being sure to reload and trigger the rules each time. 5. Remove the tag from the devices -- note how `+=` turns to `-=` -- and reload and trigger the udev rules. Alternatively, if you are trying to discover devices with fields that Akri does not yet support, such as `ATTRS`, you could leave the tag and add it to the rule in your Configuration with `TAG=="akri_tag"`. ```bash sudo echo 'SUBSYSTEM=="sound", KERNEL=="card[0-9]*", TAG-="akri_tag"' | sudo tee -a /etc/udev/rules.d/90-akri.rules sudo udevadm control --reload sudo udevadm trigger ``` 6. Confirm that the tag has been removed and no devices are listed. ```bash udevadm trigger --verbose --dry-run --type=devices --tag-match=akri_tag ``` 7. Create an Akri Configuration with your udev rule! ## Installing Akri with a udev Configuration and Discovery Handler Leveraging the above settings, Akri can be installed with the udev Discovery Handler and a udev Configuration with our udev rule specified. ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set udev.discovery.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.discoveryDetails.udevRules[0]='SUBSYSTEM=="sound"\, ATTR{vendor}=="Great Vendor"' ``` The following installation examples have been given to show how to the udev Configuration can be tailored to you cluster: * Modifying the udev rule * Specifying a broker pod image For more advanced Configuration changes that are not aided by our Helm chart, we suggest creating a Configuration file using Helm and then manually modifying it. To do this, see our documentation on [Customizing an Akri Installation](/user-guide/customizing-an-akri-installation#generating-modifying-and-applying-a-custom-configuration) ## Modifying the udev rule The udev Discovery Handler will find all devices that are described by ANY of the udev rules. For example, to discover devices made by either Great Vendor or Awesome Vendor, you could add a second udev rule. ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set udev.discovery.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.discoveryDetails.udevRules[0]='SUBSYSTEM=="sound"\, ATTR{vendor}=="Great Vendor"' \ --set udev.configuration.discoveryDetails.udevRules[1]='SUBSYSTEM=="sound"\, ATTR{vendor}=="Awesome Vendor"' ``` Akri will now discover these devices and advertize them to the cluster as resources. Each discovered device is represented as an Akri Instance. To list them, run `kubectl get akrii`. Note `akrii` is a short name for Akri Instance. All the instances will be named in the format `-`. You could change the name of the Configuration and resultant Instances to be `sound-device` by adding `--set udev.configuration.name=sound-devices` to your installation command. Now, you can schedule pods that request these Instances as resources, as explained in the [requesting akri resources document](/user-guide/requesting-akri-resources). ## Specifying a broker pod image Instead of manually deploying Pods to resources advertized by Akri, you can add a broker image to the udev Configuration. Then, a broker will automatically be deployed to each discovered device. The controller will inject the information the broker needs to find its device as environment variables. Namely, it injects an environment variable named `UDEV_DEVPATH_{INSTANCE_HASH}` which contains the device's sysfs path (i.e. `/devices/pci0000:00/0000:00:1f.3/sound/card0/input4`). Additionally, if the devnode path is found, it also injects an environment variable named `UDEV_DEVNODE_{INSTANCE_HASH}` which contains the devnode path for that device (i.e. `/dev/snd/pcmC0D0c`). The broker can grab these environment variables and proceed to interact with the device. To add a broker to the udev configuration, set the `udev.configuration.brokerPod.image.repository` value to point to your image. As an example, the installation below will deploy an empty nginx pod for each instance. Instead, you can point to your image, say `ghcr.io//sound-broker`. ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set udev.discovery.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.discoveryDetails.udevRules[0]='SUBSYSTEM=="sound"\, ATTR{vendor}=="Great Vendor"' \ --set udev.configuration.brokerPod.image.repository=nginx ``` > Note: set `udev.configuration.brokerPod.image.tag` to specify an image tag (defaults to `latest`). Akri will automatically create a broker for each discovered device. It will also create a service for each broker and one for all brokers of the Configuration that applications can point to. See the [Customizing Akri Installation](/user-guide/customizing-an-akri-installation) to learn how to [modify the broker pod spec](/user-guide/customizing-an-akri-installation#modifying-the-brokerpodspec) and [service specs](/user-guide/customizing-an-akri-installation#modifying-instanceservicespec-or-configurationservicespec) in the Configuration. ### Setting the broker Pod security context By default in the generic udev Configuration, the udev broker is run in privileged security context. This container [security context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) can be customized via Helm. For example, to instead run all processes in the Pod with user ID 1000 and group 1000, do the following: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set udev.discovery.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.discoveryDetails.udevRules[0]='SUBSYSTEM=="sound"\, ATTR{vendor}=="Great Vendor"' \ --set udev.configuration.brokerPod.image.repository=nginx \ --set udev.configuration.brokerPod.securityContext.runAsUser=1000 \ --set udev.configuration.brokerPod.securityContext.runAsGroup=1000 ``` ## Modifying a Configuration Akri has provided further documentation on [modifying the broker PodSpec](/user-guide/customizing-an-akri-installation#modifying-the-brokerpodspec), [instanceServiceSpec, or configurationServiceSpec](/user-guide/customizing-an-akri-installation#modifying-instanceservicespec-or-configurationservicespec) More information about how to modify an installed Configuration, add additional Configurations to a cluster, or delete a Configuration can be found in the [Customizing an Akri Installation document](/user-guide/customizing-an-akri-installation). ## Grouping related device nodes Akri currently provides a way to group device nodes under the topmost matching node, this allows to handle a complex device with multiple device nodes as one Instance. For example with the following udev device tree and the rule `ENV{ID_SERIAL}=="Great Vendor Complex Camera"`: ``` root ├── P: /devices/root/device1 │ A: vendor=Great Vendor │ E: ID_SERIAL=Great Vendor Complex Camera │   ├── P: /devices/root/device1/video4linux/video0 │ │ A: vendor=Great Vendor │ │ E: ID_SERIAL=Great Vendor Complex Camera │ │ E: DEVNAME=/dev/video0 │   ├── P: /devices/root/device1/video4linux/video1 │ │ A: vendor=Great Vendor │ │ E: ID_SERIAL=Great Vendor Complex Camera │ │ E: DEVNAME=/dev/video1 │   └── P: /devices/root/device1/sound/card0/pcmC0D0c │ A: vendor=Great Vendor │ E: ID_SERIAL=Great Vendor Complex Camera │ E: DEVNAME=/dev/snd/pcmC0D0c └── P: /devices/root/device2 A: vendor=Another Vendor ``` This would result in a single instance grouping `video0`, `video1` and `pcmC0D0c`. All the device nodes will get mounted into the broker pod and will be listed in the environment variables with `UDEV_DEVNODE` prefix This behavior can be enabled by setting the `udev.configuration.discoveryDetails.groupRecursive` to `true`. ## Implementation details The udev implementation can be understood by looking at several things: 1. [UdevDiscoveryDetails](https://github.com/project-akri/akri/blob/main/discovery-handlers/udev/src/discovery_handler.rs) defines the required properties 2. [UdevDiscoveryHandler](https://github.com/project-akri/akri/blob/main/discovery-handlers/udev/src/discovery_handler.rs) defines udev discovery 3. [samples/brokers/udev-video-broker](https://github.com/project-akri/examples/blob/main/brokers/udev-video-broker) defines the udev broker 4. [udev\_rule\_grammar.pest](https://github.com/project-akri/akri/blob/main/discovery-handlers/udev/src/udev_rule_grammar.pest) defines the grammar for parsing udev rules and enumerate which fields are supported (such as `ATTR` and `TAG`), which are yet to be supported (`ATTRS` and `TAGS`), and which fields will never be supported, mainly due to be assignment rather than matching fields (such as `ACTION` and `GOTO`). # Discovering and Using USB Cameras In this guide, we will walk through using Akri to discover mock USB cameras attached to nodes in a Kubernetes cluster. You'll see how Akri automatically deploys workloads to pull frames from the cameras. We will then deploy a streaming application that will point to services automatically created by Akri to access the video frames from the workloads. The following will be covered in this demo: 1. Setting up mock udev video devices 2. Setting up a cluster 3. Installing Akri via Helm with settings to create your Akri udev Configuration 4. Inspecting Akri 5. Deploying a streaming application 6. Cleanup 7. Going beyond the demo ## Setting up mock udev video devices 1. Acquire an Ubuntu 20.04 LTS, 18.04 LTS or 16.04 LTS environment to run the commands. This demo assumes that the VM being used supports the proper kernel modules, which may not be the case if using a cloud-based VM which sometimes have been slimmed down to remove unnecessary modules such as for USB devices. For example, on an Ubuntu 20.04 VM in Azure, the following prerequisite step is needed to add the necessary kernel modules: ```sh sudo apt update sudo apt -y install linux-modules-extra-azure ``` > Note: There are also guides Akri's HackMD for running the demo on [DigitalOcean](https://hackmd.io/@akri/Hyz1GW1gY) and [Google Compute Engine](https://hackmd.io/@akri/rJHdQWJeF) (and you can skip the rest of the steps in this document). Note, these guides are unmaintained and may not be up to date. 2. To setup fake usb video devices, install the v4l2loopback kernel module and its prerequisites. Learn more about v4l2 loopback [here](https://github.com/umlaeute/v4l2loopback) ```bash sudo apt update sudo apt -y install linux-headers-$(uname -r) sudo apt -y install linux-modules-extra-$(uname -r) sudo apt -y install dkms curl http://deb.debian.org/debian/pool/main/v/v4l2loopback/v4l2loopback-dkms_0.12.5-1_all.deb -o v4l2loopback-dkms_0.12.5-1_all.deb sudo dpkg -i v4l2loopback-dkms_0.12.5-1_all.deb ``` > **Note** When running on Ubuntu 20.04 LTS, 18.04 LTS or 16.04 LTS, do NOT install v4l2loopback through `sudo apt install -y v4l2loopback-dkms`, you will get an older version (0.12.3). 0.12.5-1 is required for gstreamer to work properly. > **Note**: If not able to install the debian package of v4l2loopback due to using a different Linux kernel, you can clone the repo, build the module, and setup the module dependencies like so: > > ```bash > git clone https://github.com/umlaeute/v4l2loopback.git > cd v4l2loopback > make & sudo make install > sudo make install-utils > sudo depmod -a > ``` 3. "Plug-in" two cameras by inserting the kernel module. To create different number video devices modify the `video_nr` argument. ```bash sudo modprobe v4l2loopback exclusive_caps=1 video_nr=1,2 ``` 4. Confirm that two video device nodes (video1 and video2) have been created. ```bash ls /dev/video* ``` 5. Install the necessary Gstreamer packages. ```bash sudo apt-get install -y \ libgstreamer1.0-0 gstreamer1.0-tools gstreamer1.0-plugins-base \ gstreamer1.0-plugins-good gstreamer1.0-libav ``` 6. Now that our cameras are set up, lets use Gstreamer to pass fake video streams through them. ```bash mkdir camera-logs sudo gst-launch-1.0 -v videotestsrc pattern=ball ! "video/x-raw,width=640,height=480,framerate=10/1" ! avenc_mjpeg ! v4l2sink device=/dev/video1 > camera-logs/ball.log 2>&1 & sudo gst-launch-1.0 -v videotestsrc pattern=smpte horizontal-speed=1 ! "video/x-raw,width=640,height=480,framerate=10/1" ! avenc_mjpeg ! v4l2sink device=/dev/video2 > camera-logs/smpte.log 2>&1 & ``` > **Note**: If this generates an error, be sure that there are no existing video streams targeting the video device nodes by running the following and then re-running the previous command: > > ```bash > if pgrep gst-launch-1.0 > /dev/null; then > sudo pkill -9 gst-launch-1.0 > fi > ``` ## Setting up a cluster Reference our [cluster setup documentation](/user-guide/cluster-setup) to set up a cluster for this demo. For ease of setup, only create single-node cluster, so if installing K3s or MicroK8s, you can skip the last step of the installation instructions of adding additional nodes. If you have an existing cluster, feel free to leverage it for the demo. This documentation assumes you are using a single-node cluster; however, you can certainly use a multi-node cluster. You will see additional Akri Agents and Discovery Handlers deployed [when inspecting the Akri installation](#Inspecting-Akri). > Note, if using MicroK8s, enable privileged Pods, as the udev video broker pods run privileged to easily grant them access to video devices. More explicit device access could have been configured by setting the appropriate [security context](/discovery-handlers/udev#setting-the-broker-pod-security-context) in the broker PodSpec in the Configuration. ## Installing Akri You tell Akri what you want to find with an Akri Configuration, which is one of Akri's Kubernetes custom resources. The Akri Configuration is simply a `yaml` file that you apply to your cluster. Within it, you specify three things: 1. a Discovery Handler 2. any additional device filtering 3. an image for a Pod (that we call a "broker") that you want to be automatically deployed to utilize each discovered device For this demo, we will specify 1. Akri's udev Discovery Handler, which is used to discover devices in the Linux device file system. Akri's udev Discovery Handler supports 2. filtering by udev rules. We want to find all mock USB cameras in the Linux device file system, which can be specified with a simple udev rule `KERNEL=="video[0-9]*"`. It matches name of the mock USB cameras. > Note, when real USB cameras are used, the filtering udev rule can be more precise to avoid mistaken device match. For example, a better rule is `KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"` that adds a criteria on device capability. We may go further by adding criteria such as vendor name. An example is `KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"\, ENV{ID_VENDOR}=="Great Vendor"`. In order to write correct rule, check output of "udevadm" command for USB cameras. A example is "udevadm info --query=all --name=video1". 3. a broker Pod image, we will use a sample container that Akri has provided that pulls frames from the cameras and serves them over gRPC. All of Akri's components can be deployed by specifying values in its Helm chart during an installation. Instead of having to build a Configuration from scratch, Akri has provided [Helm templates](https://github.com/project-akri/akri/blob/main/deployment/helm/templates) for Configurations for each supported Discovery Handler. Lets customize the generic [udev Configuration Helm template](https://github.com/project-akri/akri/blob/main/deployment/helm/templates/udev-configuration.yaml) with our three specifications above. We can also set the name for the Configuration to be `akri-udev-video`. In order for the Agent to know how to discover video devices, the udev Discovery Handler must exist. Akri supports an Agent image that includes all supported Discovery Handlers. This Agent will be used if `agent.full=true` is set. By default, a slim Agent without any embedded Discovery Handlers is deployed and the required Discovery Handlers can be deployed as DaemonSets. This demo will use that strategy, deploying the udev Discovery Handlers by specifying `udev.discovery.enabled=true` when installing Akri. 1. Add the Akri Helm chart and run the install command, setting Helm values as described above. ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set udev.discovery.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.name=akri-udev-video \ --set udev.configuration.discoveryDetails.udevRules[0]='KERNEL=="video[0-9]*"' \ --set udev.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/udev-video-broker" ``` ## Inspecting Akri After installing Akri, since the /dev/video1 and /dev/video2 devices are running on this node, the Akri Agent will discover them and create an Instance for each camera. 1. List all that Akri has automatically created and deployed, namely Akri Configuration we created when installing Akri, two Instances (which are the Akri custom resource that represents each device), two broker Pods (one for each camera), a service for each broker Pod, a service for all brokers, the Controller Pod, Agent Pod, and the udev Discovery Handler Pod. ```bash watch microk8s kubectl get pods,akric,akrii,services -o wide ``` For K3s and vanilla Kubernetes ```bash watch kubectl get pods,akric,akrii,services -o wide ``` Look at the Configuration and Instances in more detail. 2. Inspect the Configuration that was created via the Akri udev Helm template and values that were set when installing Akri by running the following. ```bash kubectl get akric -o yaml ``` 3. Inspect the two Instances. Notice that in the `brokerProperties` of each instance, you can see the device nodes (`/dev/video1` or `/dev/video2`) that the Instance represents. The `brokerProperties` of an Instance are set as environment variables in the broker Pods that are utilizing the device the Instance represents. This told the broker which device to connect to. We can also see in the Instance a usage slot and that it was reserved for this node. Each Instance represents a device and its usage. ```bash kubectl get akrii -o yaml ``` If this was a shared device (such as an IP camera), you may have wanted to increase the number of nodes that could use the same device by specifying `capacity`. There is a `capacity` parameter for each Configuration, which defaults to `1`. Its value could have been increased when installing Akri (via `--set .configuration.capacity=2` to allow 2 nodes to use the same device) and more usage slots (the number of usage slots is equal to `capacity`) would have been created in the Instance. **Deploying a streaming application** 4. Deploy a video streaming web application that points to both the Configuration and Instance level services that were automatically created by Akri. ```bash kubectl apply -f https://raw.githubusercontent.com/project-akri/akri/main/deployment/samples/akri-video-streaming-app.yaml ``` For MicroK8s ```bash watch microk8s kubectl get pods ``` For K3s and vanilla Kubernetes ```bash watch kubectl get pods ``` 5. Determine which port the service is running on. Be sure to save this port number for the next step. ```bash kubectl get service/akri-video-streaming-app --output=jsonpath='{.spec.ports[?(@.name=="http")].nodePort}' && echo ``` 6. SSH port forwarding can be used to access the streaming application. In a new terminal, enter your ssh command to to access your VM followed by the port forwarding request. The following command will use port 50000 on the host. Feel free to change it if it is not available. Be sure to replace `` with the port number outputted in the previous step. ```bash ssh someuser@ -L 50000:localhost: ``` > **Note** we've noticed issues with port forwarding with WSL 2. Please use a different terminal. 7. Navigate to `http://localhost:50000/`. The large feed points to Configuration level service (`udev-camera-svc`), while the bottom feed points to the service for each Instance or camera (`udev-camera-svc-`). ## Cleanup 1. Bring down the streaming service. ```bash kubectl delete service akri-video-streaming-app kubectl delete deployment akri-video-streaming-app ``` For MicroK8s ```bash watch microk8s kubectl get pods ``` For K3s and vanilla Kubernetes ```bash watch kubectl get pods ``` 2. Delete the configuration, and watch the associated instances, pods, and services be deleted. ```bash kubectl delete akric akri-udev-video ``` For MicroK8s ```bash watch microk8s kubectl get pods,services,akric,akrii -o wide ``` For K3s and vanilla Kubernetes ```bash watch kubectl get pods,services,akric,akrii -o wide ``` 3. If you are done using Akri, it can be uninstalled via Helm. ```bash helm delete akri ``` 4. Delete Akri's CRDs. ```bash kubectl delete crd instances.akri.sh kubectl delete crd configurations.akri.sh ``` 5. Stop video streaming from the video devices. ```bash if pgrep gst-launch-1.0 > /dev/null; then sudo pkill -9 gst-launch-1.0 fi ``` 6. "Unplug" the fake video devices by removing the kernel module. ```bash sudo modprobe -r v4l2loopback ``` ## Going beyond the demo 1. Plug in real cameras! You can [pass environment variables](/development/broker-development#Specifying-additional-broker-environment-variables-in-a-Configuration) to the frame server broker to specify the format, resolution width/height, and frames per second of your cameras. 2. Apply the [ONVIF Configuration](/discovery-handlers/onvif) and make the streaming app display footage from both the local video devices and onvif cameras. To do this, modify the [video streaming yaml](https://github.com/project-akri/akri/blob/main/deployment/samples/akri-video-streaming-app.yaml) as described in the inline comments in order to create a larger service that aggregates the output from both the `udev-camera-svc` service and `onvif-camera-svc` service. 3. Add more nodes to the cluster. 4. Modify the udev rule to find a more specific subset of cameras Instead of finding all video4linux device nodes, the udev rule can be modified to exclude certain device nodes, find devices only made by a certain manufacturer, and more. For example, the rule can be narrowed by matching cameras with specific properties. To see the properties of a camera on a node, do `udevadm info --query=property --name /dev/video0`, passing in the proper devnode name. In this example, `ID_VENDOR=Microsoft` was one of the outputted properties. To only find cameras made by Microsoft, the rule can be modified like the following: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set udev.discovery.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.name=akri-udev-video \ --set udev.configuration.discoveryDetails.udevRules[0]='KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"\, ENV{ID_VENDOR}=="Microsoft"' \ --set udev.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/udev-video-broker" ``` 5. Discover other udev devices by creating a new udev configuration and broker. Learn more about the udev Discovery Handler Configuration [here](/discovery-handlers/udev). # Discovering and Using USB Cameras on Raspberry Pi 4 This will demonstrate how to get Akri working on a **Raspberry Pi 4** and walk through using Akri to discover mock USB cameras attached to nodes in a Kubernetes cluster. You'll see how Akri automatically deploys workloads to pull frames from the cameras. We will then deploy a streaming application that will point to services automatically created by Akri to access the video frames from the workloads. The following will be covered in this demo: 1. Setting up single node cluster on a Raspberry Pi 4 2. Setting up mock udev video devices 3. Installing Akri via Helm with settings to create your Akri udev Configuration 4. Inspecting Akri 5. Deploying a streaming application 6. Cleanup 7. Going beyond the demo ## Set up single node cluster on a Raspberry Pi 4 1. Using instructions found [here](https://ubuntu.com/download/raspberry-pi), download 64-bit Ubuntu:18.04 2. Using the instructions found [here](https://ubuntu.com/download/raspberry-pi/thank-you?version=18.04\&versionPatch=.4\&architecture=arm64+raspi3), apply the Ubuntu image to an SD card. 3. Plug in SD card and start Raspberry Pi 4. 4. Install docker. ```sh sudo apt install -y docker.io ``` 5. Install Helm. ```sh sudo apt install -y curl curl -L https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash ``` 6. Install Kubernetes. ```sh curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main" sudo apt install -y kubectl kubeadm kubelet ``` 7. Enable cgroup memory by appending `cgroup_enable=cpuset` and `cgroup_enable=memory cgroup_memory=1` to this file: `/boot/firmware/nobtcmd.txt` 8. Start master node ```sh sudo kubeadm init ``` You will then need to setup kubernetes config and environment variables using the commands below ```sh mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config export KUBECONFIG=$HOME/.kube/config ``` 9. To enable workloads on our single-node cluster, remove the master taint. ```sh kubectl taint nodes --all node-role.kubernetes.io/master- ``` 10. Apply a network provider to the cluster. ```sh kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" ``` ## Set up mock udev video devices 1. Open a new terminal and ssh into your ubuntu server that your cluster is running on. 2. To setup fake usb video devices, install the v4l2loopback kernel module and its prerequisites. Learn more about v4l2 loopback [here](https://github.com/umlaeute/v4l2loopback) ```sh sudo apt update sudo apt -y install linux-headers-$(uname -r) sudo apt -y install linux-modules-extra-$(uname -r) sudo apt -y install dkms curl http://deb.debian.org/debian/pool/main/v/v4l2loopback/v4l2loopback-dkms_0.12.5-1_all.deb -o v4l2loopback-dkms_0.12.5-1_all.deb sudo dpkg -i v4l2loopback-dkms_0.12.5-1_all.deb ``` > **Note**: If not able to install the debian package of v4l2loopback due to using a different Linux kernel, you can clone the repo, build the module, and setup the module dependencies like so: > > ```sh > git clone https://github.com/umlaeute/v4l2loopback.git > cd v4l2loopback > make & sudo make install > sudo make install-utils > sudo depmod -a > ``` 3. "Plug-in" two cameras by inserting the kernel module. To create different number video devices modify the `video_nr` argument. ```sh sudo modprobe v4l2loopback exclusive_caps=1 video_nr=1,2 ``` 4. Confirm that two video device nodes (video1 and video2) have been created. ```sh ls /dev/video* ``` 5. Install the necessary Gstreamer packages. ```sh sudo apt-get install -y \ libgstreamer1.0-0 gstreamer1.0-tools gstreamer1.0-plugins-base \ gstreamer1.0-plugins-good gstreamer1.0-libav ``` 6. Now that our cameras are set up, lets use Gstreamer to pass fake video streams through them. ```sh mkdir camera-logs sudo gst-launch-1.0 -v videotestsrc pattern=ball ! "video/x-raw,width=640,height=480,framerate=10/1" ! avenc_mjpeg ! v4l2sink device=/dev/video1 > camera-logs/ball.log 2>&1 & sudo gst-launch-1.0 -v videotestsrc pattern=smpte horizontal-speed=1 ! "video/x-raw,width=640,height=480,framerate=10/1" ! avenc_mjpeg ! v4l2sink device=/dev/video2 > camera-logs/smpte.log 2>&1 & ``` > **Note**: If this generates an error, be sure that there are no existing video streams targeting the video device nodes by running the following and then re-running the previous command: > > ```sh > if pgrep gst-launch-1.0 > /dev/null; then > sudo pkill -9 gst-launch-1.0 > fi > ``` ## Installing Akri You tell Akri what you want to find with an Akri Configuration, which is one of Akri's Kubernetes custom resources. The Akri Configuration is simply a `yaml` file that you apply to your cluster. Within it, you specify three things: 1. a Discovery Handler 2. any additional device filtering 3. an image for a Pod (that we call a "broker") that you want to be automatically deployed to utilize each discovered device For this demo, we will specify 1. Akri's udev Discovery Handler, which is used to discover devices in the Linux device file system. Akri's udev Discovery Handler supports 2. filtering by udev rules. We want to find all mock USB cameras in the Linux device file system, which can be specified with a simple udev rule `KERNEL=="video[0-9]*"`. It matches name of the mock USB cameras. > Note, when real USB cameras are used, the filtering udev rule can be more precise to avoid mistaken device match. For example, a better rule is `KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"` that adds a criteria on device capability. We may go further by adding criteria such as vendor name. An example is `KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"\, ENV{ID_VENDOR}=="Great Vendor"`. In order to write correct rule, check output of "udevadm" command for USB cameras. A example is "udevadm info --query=all --name=video1". 3. a broker Pod image, we will use a sample container that Akri has provided that pulls frames from the cameras and serves them over gRPC. All of Akri's components can be deployed by specifying values in its Helm chart during an installation. Instead of having to build a Configuration from scratch, Akri has provided [Helm templates](https://github.com/project-akri/akri/blob/main/deployment/helm/templates) for Configurations for each supported Discovery Handler. Lets customize the generic [udev Configuration Helm template](https://github.com/project-akri/akri/blob/main/deployment/helm/templates/udev-configuration.yaml) with our three specifications above. We can also set the name for the Configuration to be `akri-udev-video`. In order for the Agent to know how to discover video devices, the udev Discovery Handler must exist. Akri supports an Agent image that includes all supported Discovery Handlers. This Agent will be used if `agent.full=true` is set. By default, a slim Agent without any embedded Discovery Handlers is deployed and the required Discovery Handlers can be deployed as DaemonSets. This demo will use that strategy, deploying the udev Discovery Handlers by specifying `udev.discovery.enabled=true` when installing Akri. 1. Add the Akri Helm chart and run the install command, setting Helm values as described above. ```sh helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set udev.discovery.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.name=akri-udev-video \ --set udev.configuration.discoveryDetails.udevRules[0]='KERNEL=="video[0-9]*"' \ --set udev.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/udev-video-broker" ``` ## Inspecting Akri After installing Akri, since the /dev/video1 and /dev/video2 devices are running on this node, the Akri Agent will discover them and create an Instance for each camera. 1. List all that Akri has automatically created and deployed, namely Akri Configuration we created when installing Akri, two Instances (which are the Akri custom resource that represents each device), two broker Pods (one for each camera), a service for each broker Pod, a service for all brokers, the Controller Pod, Agent Pod, and the udev Discovery Handler Pod. `sh watch kubectl get pods,akric,akrii,services -o wide` Look at the Configuration and Instances in more detail. 2. Inspect the Configuration that was created via the Akri udev Helm template and values that were set when installing Akri by running the following. ```sh kubectl get akric -o yaml ``` 3. Inspect the two Instances. Notice that in the `brokerProperties` of each instance, you can see the device nodes (`/dev/video1` or `/dev/video2`) that the Instance represents. The `brokerProperties` of an Instance are set as environment variables in the broker Pods that are utilizing the device the Instance represents. This told the broker which device to connect to. We can also see in the Instance a usage slot and that it was reserved for this node. Each Instance represents a device and its usage. ```sh kubectl get akrii -o yaml ``` If this was a shared device (such as an IP camera), you may have wanted to increase the number of nodes that could use the same device by specifying `capacity`. There is a `capacity` parameter for each Configuration, which defaults to `1`. Its value could have been increased when installing Akri (via `--set .configuration.capacity=2` to allow 2 nodes to use the same device) and more usage slots (the number of usage slots is equal to `capacity`) would have been created in the Instance. ## Deploying a streaming application 1. Deploy a video streaming web application that points to both the Configuration and Instance level services that were automatically created by Akri. ```sh kubectl apply -f https://raw.githubusercontent.com/project-akri/akri/main/deployment/samples/akri-video-streaming-app.yaml watch kubectl get pods ``` 2. Determine which port the service is running on. Be sure to save this port number for the next step. ```sh kubectl get service/akri-video-streaming-app --output=jsonpath='{.spec.ports[?(@.name=="http")].nodePort}' && echo ``` 3. SSH port forwarding can be used to access the streaming application. In a new terminal, enter your ssh command to to access your VM followed by the port forwarding request. The following command will use port 50000 on the host. Feel free to change it if it is not available. Be sure to replace `` with the port number outputted in the previous step. ```sh ssh someuser@ -L 50000:localhost: ``` > **Note** we've noticed issues with port forwarding with WSL 2. Please use a different terminal. 4. Navigate to `http://localhost:50000/`. The large feed points to Configuration level service (`udev-camera-svc`), while the bottom feed points to the service for each Instance or camera (`udev-camera-svc-`). ## Cleanup 1. Bring down the streaming service. ```sh kubectl delete service akri-video-streaming-app kubectl delete deployment akri-video-streaming-app watch kubectl get pods ``` 2. Delete the configuration, and watch the associated instances, pods, and services be deleted. ```sh kubectl delete akric akri-udev-video watch kubectl get pods,services,akric,akrii -o wide ``` 3. If you are done using Akri, it can be uninstalled via Helm. ```sh helm delete akri ``` 4. Delete Akri's CRDs. ```sh kubectl delete crd instances.akri.sh kubectl delete crd configurations.akri.sh ``` 5. Stop video streaming from the video devices. ```sh if pgrep gst-launch-1.0 > /dev/null; then sudo pkill -9 gst-launch-1.0 fi ``` 6. "Unplug" the fake video devices by removing the kernel module. ```sh sudo modprobe -r v4l2loopback ``` ## Going beyond the demo 1. Plug in real cameras! You can [pass environment variables](/development/broker-development#Specifying-additional-broker-environment-variables-in-a-Configuration) to the frame server broker to specify the format, resolution width/height, and frames per second of your cameras. 2. Apply the [ONVIF Configuration](/discovery-handlers/onvif) and make the streaming app display footage from both the local video devices and onvif cameras. To do this, modify the [video streaming yaml](https://github.com/project-akri/akri/blob/main/deployment/samples/akri-video-streaming-app.yaml) as described in the inline comments in order to create a larger service that aggregates the output from both the `udev-camera-svc` service and `onvif-camera-svc` service. 3. Add more nodes to the cluster. 4. Modify the udev rule to find a more specific subset of cameras Instead of finding all video4linux device nodes, the udev rule can be modified to exclude certain device nodes, find devices only made by a certain manufacturer, and more. For example, the rule can be narrowed by matching cameras with specific properties. To see the properties of a camera on a node, do `udevadm info --query=property --name /dev/video0`, passing in the proper devnode name. In this example, `ID_VENDOR=Microsoft` was one of the outputted properties. To only find cameras made by Microsoft, the rule can be modified like the following: ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set udev.discovery.enabled=true \ --set udev.configuration.enabled=true \ --set udev.configuration.name=akri-udev-video \ --set udev.configuration.discoveryDetails.udevRules[0]='KERNEL=="video[0-9]*"\, ENV{ID_V4L_CAPABILITIES}==":capture:"\, ENV{ID_VENDOR}=="Microsoft"' \ --set udev.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/udev-video-broker" ``` 5. Discover other udev devices by creating a new udev configuration and broker. Learn more about the udev Discovery Handler Configuration [here](/discovery-handlers/udev). # Discovering and Using OPC UA Thermometers OPC UA is a communication protocol for industrial automation. It is a client/server technology that comes with a security and communication framework. This demo will help you get started using Akri to discover OPC UA PLC Servers and utilize them via a broker that contains an OPC UA Client. Specifically, a Akri Configuration called OPC UA Monitoring was created for this scenario, which will show how Akri can be used to detect anomaly values of a specific OPC UA Variable. To do so, the OPC UA Clients in the brokers will subscribe to that variable and serve its value over gRPC for an anomaly detection web application to consume. This Configuration could be used to monitor a barometer, CO detector, and more; however, for this example, that variable will represent the PLC values for temperature of a thermostat and any value outside the range of 70-80 degrees is an anomaly. The demo consists of the following components: 1. Two OPC UA PLC Servers 2. (Optional) Certificates for the Servers and Akri brokers 3. An OPC UA Monitoring broker that contains an OPC UA Client that subscribes to a specific NodeID (for that PLC variable) 4. Akri installation 5. An anomaly detection web application ## Demo Flow ![](/files/-MgSrrt82nrH4PtFSLqe) 1. An operator (meaning you!) applies to a single-node cluster the OPC UA Configuration, which specifies the addresses of the OPC UA Servers, which OPC UA Variable to monitor, and whether to use security. 2. Agent sees the OPC UA Configuration, discovers the servers specified in the Configuration, and creates an Instance for each server. 3. The Akri Controller sees the Instances in etcd and schedules an OPC UA Monitoring broker pod for each server. 4. Once the OPC UA Monitoring broker pod starts up, it will create an OPC UA Client that will create a secure channel with its server. 5. The OPC UA Client will subscribe to the OPC UA Variable with the NodeID with `Identifier` "FastUInt1" and `NamespaceIndex` 2 as specified in the OPC UA Configuration. The server will publish any time the value of that variable changes. 6. The OPC UA Monitoring broker will serve over gRPC the latest value of the OPC UA Variable and the address of the OPC UA Server that published the value. 7. The anomaly detection web application will test whether that value is an outlier to its pre-configured dataset. It then will display a log of the values on a web application, showing outliers in red and normal values in green. The following steps need to be completed to run the demo: * [Setting up a single-node cluster](#setting-up-a-cluster) * [(Optional) Creating X.509 v3 Certificates for the servers and Akri broker and storing them in a Kubernetes Secret](#creating-x509-v3-certificates) * [Creating two OPC UA Servers](#creating-opc-ua-servers) * [Running Akri](#running-akri) * [Deploying an anomaly detection web application as an end consumer of the brokers](#deploying-an-anomaly-detection-web-application-as-an-end-consumer-of-the-brokers) If at any point in the demo, you want to dive deeper into OPC UA or clarify a term, you can reference the [online OPC UA specifications](https://reference.opcfoundation.org/v104/). ## Setting up a cluster Reference our [cluster setup documentation](/user-guide/cluster-setup) to set up a cluster for this demo. For ease of setup, only create a single-node cluster, so if installing K3s or MicroK8s, you can skip the last step of the installation instructions of adding additional nodes. If you have an existing cluster, feel free to leverage it for the demo. This documentation assumes you are using a single-node cluster; however, you can certainly use a multi-node cluster. ## Creating X.509 v3 Certificates **If security is not desired, skip to** [**Creating OPC UA Servers**](#creating-opc-ua-servers)**, as each monitoring broker will use an OPC UA Security Policy of None if it cannot find credentials mounted in its pod.** Akri will deploy an OPC UA Monitoring broker for each OPC UA Server a node in the cluster can see. This broker contains an OPC UA Client that will need the proper credentials in order to communicate with the OPC UA Server in a secure fashion. Specifically, before establishing a session, an OPC UA Client and Server must create a secure channel over the communication layer to ensure message integrity, confidentiality, and application authentication. Proper application credentials in the form of X.509 v3 certificates are needed for application authentication. Every OPC UA Application, whether Client, Server, or DiscoveryServer, has a certificate store, which includes the application's own credentials along with a list of trusted and rejected application instance certificates. According to OPC UA specification, there are three ways to configure OPC UA Server and Clients' certificate stores so that they trust each other's certificates, which are explained in the [OPC UA proposal](https://github.com/deislabs/akri-docs/blob/main/proposals/opcua.md). This demo will walk through the third method of creating Client and Server certificates that are issued by a common Certificate Authority (CA). Then, that root CA certificate simply needs to be added to the trusted folder of Client and Servers' certificate stores, and they will automatically trust each other on the basis of having a common root certificate. The following image walks through how to configure the Client and Server certificate stores for Akri. ![OPC UA Certificate Creation Diagram](/files/-MgfsHL7kVCXSMf3kKNi) 1. Generate an X.509 v3 Certificate for Akri OPC UA Monitoring brokers and sign it with the same CA that has signed the certificates of all the OPC UA Servers that will be discovered. 2. Create a Kubernetes Secret named opcua-broker-credentials that contains four items with the following key names: client\_certificate, client\_key, ca\_certificate, and ca\_crl. 3. The credentials will be mounted in the broker at the path /etc/opcua-certs/client-pki. ### Generating certificates Create three (one for the broker and each server) OPC UA compliant X.509v3 certificates, ensuring that the certificate contains the [necessary components](http://opclabs.doc-that.com/files/onlinedocs/QuickOpc/Latest/User%27s%20Guide%20and%20Reference-QuickOPC/Providing%20Client%20Instance%20Certificate.html) such as an application URI. They should all be signed by a common Certificate Authority (CA). There are many tools for generating proper certificates for OPC UA, such as the [OPC Foundation's Certificate Generator](https://github.com/OPCFoundation/Misc-Tools) or openssl (as in this [walk through](https://github.com/OPCFoundation/Misc-Tools)). ### Creating an opcua-broker-credentials Kubernetes Secret The OPC UA Client certificate will be passed to the OPC UA Monitoring broker as a Kubernetes Secret mounted as a volume. Read more about the decision to use Kubernetes secrets to pass the Client certificates in the [Credentials Passing Proposal](https://github.com/deislabs/akri-docs/blob/main/proposals/credentials-passing.md). Create a Kubernetes Secret, projecting each certificate/crl/private key with the expected key name (i.e. `client_certificate`, `client_key`, `ca_certificate`, and `ca_crl`). Specify the file paths such that they point to the credentials made in the previous section. ```bash kubectl create secret generic opcua-broker-credentials \ --from-file=client_certificate=/path/to/AkriBroker/own/certs/AkriBroker\ \[\].der \ --from-file=client_key=/path/to/AkriBroker/own/private/AkriBroker\ \[\].pfx \ --from-file=ca_certificate=/path/to/ca/certs/SomeCA\ \[\].der \ --from-file=ca_crl=/path/to/ca/crl/SomeCA\ \[\].crl ``` When mounting certificates is enabled later in the [Running Akri section](#running-akri) with Helm via `--set opcua.configuration.mountCertificates='true'`, the secret named `opcua-broker-credentials` will be mounted into the OPC UA monitoring brokers. It is mounted to the volume `credentials` at the `mountPath` /etc/opcua-certs/client-pki, as shown in the [OPC UA Configuration Helm template](https://github.com/project-akri/akri/blob/main/deployment/helm/templates/opcua-configuration.yaml). This is the path where the brokers expect to find the certificates. ## Creating OPC UA Servers Now, we must create some OPC UA PLC Servers to discover. Instead of starting from scratch, we deploy OPC PLC server containers. You can read more about the containers and their parameters [here](https://github.com/Azure-Samples/iot-edge-opc-plc). 1. Create an empty YAML file called `opc-deployment.yaml`. 2. (Optional) If you are using security, place the OpcPlc certificate and the CA certificate as below. ``` plc ├── own │   ├── certs │   │   └── OpcPlc [hash].der │   └── private │   └── OpcPlc [hash].pfx └── trusted ├── certs │   └── someCA.der └── crl └── someCA.crl ``` 3. (A) If you are not using security, copy and paste the contents below into the YAML file. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: opcplc labels: app: opcplc spec: selector: matchLabels: app: opcplc template: metadata: labels: app: opcplc name: opc-plc-server spec: hostNetwork: true containers: - name: opcplc1 image: mcr.microsoft.com/iotedge/opc-plc:latest ports: - containerPort: 50000 args: [ "--portnum=50000", "--autoaccept", "--fastnodes=1", "--fasttype=uint", "--fasttypelowerbound=65", "--fasttypeupperbound=85", "--fasttyperandomization=True", "--showpnjsonph", "--unsecuretransport", ] - name: opcplc2 image: mcr.microsoft.com/iotedge/opc-plc:latest ports: - containerPort: 50001 args: [ "--portnum=50001", "--autoaccept", "--fastnodes=1", "--fasttype=uint", "--fasttypelowerbound=65", "--fasttypeupperbound=85", "--fasttyperandomization=True", "--showpnjsonph", "--unsecuretransport", ] ``` (B) If you are using security, copy and paste the contents below into the YAML file, replacing the path in the last line with your path to the folder that contains the certificates. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: opcplc labels: app: opcplc spec: selector: matchLabels: app: opcplc template: metadata: labels: app: opcplc name: opc-plc-server spec: hostNetwork: true containers: - name: opcplc1 image: mcr.microsoft.com/iotedge/opc-plc:latest ports: - containerPort: 50000 args: [ "--portnum=50000", "--autoaccept", "--fastnodes=1", "--fasttype=uint", "--fasttypelowerbound=65", "--fasttypeupperbound=85", "--fasttyperandomization=True", "--showpnjsonph", ] volumeMounts: - mountPath: /app/pki name: opc-certs - name: opcplc2 image: mcr.microsoft.com/iotedge/opc-plc:latest ports: - containerPort: 50001 args: [ "--portnum=50001", "--autoaccept", "--fastnodes=1", "--fasttype=uint", "--fasttypelowerbound=65", "--fasttypeupperbound=85", "--fasttyperandomization=True", "--showpnjsonph", ] volumeMounts: - mountPath: /app/pki name: opc-certs volumes: - name: opc-certs hostPath: path: ``` 4. Save the file, then simply apply your deployment YAML to create two OPC UA servers. ```bash kubectl apply -f opc-deployment.yaml ``` We have successfully created two OPC UA PLC servers, each with one fast PLC node which generates an **unsigned integer** with **lower bound = 65** and **upper bound = 85** at a **rate of 1**. It should be up and running. ## Running Akri 1. Make sure your OPC UA PLC Servers are running. 2. Now it is time to install the Akri using Helm. When installing Akri, we can specify that we want to deploy the OPC UA Discovery Handlers by setting the helm value `opcua.discovery.enabled=true`. We also specify that we want to create an OPC UA Configuration with `--set opcua.configuration.enabled=true`. In the Configuration, any values that should be set as environment variables in brokers can be set in `opcua.configuration.brokerProperties`. In this scenario, we will specify the `Identifier` and `NamespaceIndex` of the NodeID we want the brokers to monitor. In our case that is our temperature variable we made earlier, which has an `Identifier` of `FastUInt1` and `NamespaceIndex` of `2`. Your OPC PLC discovery URL will look something like `"opc.tcp://:50000/`. If using security, uncomment `--set opcua.configuration.mountCertificates='true'`. ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.name=akri-opcua-monitoring \ --set opcua.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/opcua-monitoring-broker" \ --set opcua.configuration.brokerProperties.IDENTIFIER='FastUInt1' \ --set opcua.configuration.brokerProperties.NAMESPACE_INDEX='2' \ --set opcua.configuration.discoveryDetails.discoveryUrls[0]="opc.tcp://:50000/" \ --set opcua.configuration.discoveryDetails.discoveryUrls[1]="opc.tcp://:50001/" \ # --set opcua.configuration.mountCertificates='true' ``` > Note: `FastUInt1` is the identifier of the [fast changing node](https://github.com/Azure-Samples/iot-edge-opc-plc#slow-and-fast-changing-nodes) that is provided by the OPC PLC server. Akri Agent will discover the two Servers and create an Instance for each Server. Watch two broker pods spin up, one for each Server. ```bash kubectl get pods -o wide --watch ``` To inspect more of the elements of Akri: * Run `kubectl get crd`, and you should see the CRDs listed. * Run `kubectl get akric`, and you should see `akri-opcua-monitoring`. * If the OPC PLC Servers were discovered and pods spun up, the instances can be seen by running `kubectl get akrii` and further inspected by running `kubectl get akrii akri-opcua-monitoring- -o yaml` ## Deploying an anomaly detection web application as an end consumer of the brokers A sample anomaly detection web application was created for this end-to-end demo. It has a gRPC stub that calls the brokers' gRPC services, getting the latest temperature value. It then determines whether this value is an outlier to the dataset using the Local Outlier Factor strategy. The dataset is simply a csv with the numbers between 70-80 repeated several times; therefore, any value significantly outside this range will be seen as an outlier. The web application serves as a log, displaying all the temperature values and the address of the OPC UA Server that sent the values. It shows anomaly values in red. The anomalies always have a value of 120 due to how we set up the `DoSimulation` function in the OPC UA Servers. 1. Deploy the anomaly detection app and watch a pod spin up for the app. ```bash kubectl apply -f https://raw.githubusercontent.com/project-akri/akri/main/deployment/samples/akri-anomaly-detection-app.yaml ``` ```bash kubectl get pods -o wide --watch ``` 2. Determine which port the service is running on. Be sure to save this port number for the next step. ```bash kubectl get service/akri-anomaly-detection-app --output=jsonpath='{.spec.ports[?(@.name=="http")].nodePort}' && echo ``` 3. SSH port forwarding can be used to access the streaming application. In a new terminal, enter your ssh command to to access your VM followed by the port forwarding request. The following command will use port 50000 on the host. Feel free to change it if it is not available. Be sure to replace `` with the port number outputted in the previous step. ```bash ssh someuser@ -L 50000:localhost: ``` > **Note** we've noticed issues with port forwarding with WSL 2. Please use a different terminal. 4. Navigate to `http://localhost:50000/`. It takes 3 seconds for the site to load, after which, you should see a log of the temperature values, which updates every few seconds. Note how the values are coming from two different DiscoveryURLs, namely the ones for each of the two OPC UA Servers. ## Clean up 1. Delete the anomaly detection application deployment and service. ```bash kubectl delete service akri-anomaly-detection-app kubectl delete deployment akri-anomaly-detection-app ``` 2. Delete the OPC UA Monitoring Configuration and watch the instances, pods, and services be deleted. ```bash kubectl delete akric akri-opcua-monitoring watch kubectl get pods,services,akric,akrii -o wide ``` 3. Bring down the Akri Agent, Controller, and CRDs. ```bash helm delete akri kubectl delete crd instances.akri.sh kubectl delete crd configurations.akri.sh ``` 4. Delete the OPC UA server deployment. ```bash kubectl delete -f opc-deployment.yaml ``` ## Extensions Now that you have the end to end demo running let's talk about some ways you can go beyond the demo to better understand the advantages of Akri. This section will cover: 1. Adding a node to the cluster 2. Using a Local Discovery Server to discover the Servers instead of passing the DiscoveryURLs to the OPC UA Monitoring Configuration 3. Modifying the OPC UA Configuration to filter out an OPC UA Server 4. Creating a different broker and end application 5. Creating a new OPC UA Configuration ### Adding a Node to the cluster To see how Akri easily scales as nodes are added to the cluster, add another node to your (K3s, MicroK8s, or vanilla Kubernetes) cluster. 1. If you are using MicroK8s, create another MicroK8s instance, following the same steps as in [Setting up a single-node cluster](#setting-up-a-cluster) above. Then, in your first VM that is currently running Akri, get the join command by running `microk8s add-node`. In your new VM, run one of the join commands outputted in the previous step. 1. Confirm that you have successfully added a node to the cluster by running the following in your control plane VM: ```bash kubectl get no ``` 2. You can see that another Agent pod has been deployed to the new node; however, no new OPC UA Monitoring brokers have been deployed. This is because the default `capacity` for OPC UA is 1, so by default only one Node is allowed to utilize a device via a broker. ```bash kubectl get pods -o wide ``` 3. Let's play around with the capacity value and use the `helm upgrade` command to modify our OPC UA Monitoring Configuration such that the capacity is 2. On the control plane node, run the following, once again uncommenting `--set opcua.configuration.mountCertificates='true'` if using security. Watch as the broker terminates and then four come online in a Running state. ```bash helm upgrade akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.name=akri-opcua-monitoring \ --set opcua.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/opcua-monitoring-broker" \ --set opcua.configuration.brokerProperties.IDENTIFIER='FastUInt1' \ --set opcua.configuration.brokerProperties.NAMESPACE_INDEX='2' \ --set opcua.configuration.discoveryDetails.discoveryUrls[0]="opc.tcp://:50000/" \ --set opcua.configuration.discoveryDetails.discoveryUrls[1]="opc.tcp://:50001/" \ --set opcua.capacity=2 \ # --set opcua.configuration.mountCertificates='true' ``` ```bash watch kubectl get pods,akrii -o wide ``` 4. Once you are done using Akri, you can remove your worker node from the cluster. For MicroK8s this is done by running on the worker node: ```bash microk8s leave ``` Then, to complete the node removal, on the host run the following, inserting the name of the worker node (you can look it up with `microk8s kubectl get no`): ```bash microk8s remove-node ``` ### Setting up and using a Local Discovery Server (Windows Only) **This walk-through only supports setting up an LDS on Windows, since that is the OS the OPC Foundation sample LDS executable was written for.** A Local Discovery Server (LDS) is a unique type of OPC UA server which maintains a list of OPC UA servers that have registered with it. The OPC UA Configuration takes in a list of DiscoveryURLs, whether for LDSes or a specific servers. Rather than having to pass in the DiscoveryURL for every OPC UA Server you want Akri to discover and deploy brokers to, you can set up a Local Discovery Server on the machine your servers are running on, make the servers register with the LDS on start up, and pass only the LDS DiscoveryURL into the OPC UA Monitoring Configuration. Agent will ask the LDS for the addresses of all the servers registered with it and the demo continues as it would've without an LDS. The OPC Foundation has provided a Windows based LDS executable which can be downloaded from their [website](https://opcfoundation.org/developer-tools/samples-and-tools-unified-architecture/local-discovery-server-lds/). Download version 1.03.401. It runs as a background service on Windows and can be started or stopped under Windows -> Services. The OPC Foundation has provided [documentation](https://apps.opcfoundation.org/LDS/) on configuring your LDS. Most importantly, it states that you must add the LDS executable to your firewall as an inbound rule. Make sure you have restarted your OPC UA Servers, since they attempt to register with their LDS on start up. Now, we can install Akri with the OPC UA Configuration, passing in the LDS DiscoveryURL instead of both servers' DiscoveryURLs. Replace "Windows host IP address" with the IP address of the Windows machine you installed the LDS on (and is hosting the servers). Be sure to uncomment mounting certificates if you are enabling security: ```bash helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.name=akri-opcua-monitoring \ --set opcua.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/opcua-monitoring-broker" \ --set opcua.configuration.brokerProperties.IDENTIFIER='FastUInt1' \ --set opcua.configuration.brokerProperties.NAMESPACE_INDEX='2' \ --set opcua.configuration.discoveryDetails.discoveryUrls[0]="opc.tcp://:4840/" \ # --set opcua.configuration.mountCertificates='true' ``` You can watch as an Instance is created for each Server and two broker pods are spun up. ```bash watch kubectl get pods,akrii -o wide ``` ### Modifying the OPC UA Configuration to filter out an OPC UA Server Instead of deploying brokers to all servers registered with specified Local Discovery Servers, an operator can choose to include or exclude a list of application names (the `applicationName` property of a server's `ApplicationDescription` as specified by UA Specification 12). For example, to discover all servers registered with the default LDS except for the server named "SomeServer0", do the following. ```bash helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.name=akri-opcua-monitoring \ --set opcua.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/opcua-monitoring-broker" \ --set opcua.configuration.brokerProperties.IDENTIFIER='FastUInt1' \ --set opcua.configuration.brokerProperties.NAMESPACE_INDEX='2' \ --set opcua.configuration.discoveryDetails.discoveryUrls[0]="opc.tcp://:4840/" \ --set opcua.configuration.discoveryDetails.applicationNames.action=Exclude \ --set opcua.configuration.discoveryDetails.applicationNames.items[0]="SomeServer0" \ # --set opcua.configuration.mountCertificates='true' ``` Alternatively, to only discover the server named "SomeServer0", do the following: ```bash helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.name=akri-opcua-monitoring \ --set opcua.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/opcua-monitoring-broker" \ --set opcua.configuration.brokerProperties.IDENTIFIER='FastUInt1' \ --set opcua.configuration.brokerProperties.NAMESPACE_INDEX='2' \ --set opcua.configuration.discoveryDetails.discoveryUrls[0]="opc.tcp://:4840/" \ --set opcua.configuration.discoveryDetails.applicationNames.action=Include \ --set opcua.configuration.discoveryDetails.applicationNames.items[0]="SomeServer0" \ # --set opcua.configuration.mountCertificates='true' ``` ### Creating a different broker and end application The OPC UA Monitoring broker and anomaly detection application support a very specific scenario: monitoring an OPC UA Variable for anomalies. The workload or broker you want to deploy to discovered OPC UA Servers may be different. OPC UA Servers' address spaces are widely varied, so the options for broker implementations are endless. Passing the NodeID `Identifier` and `NamespaceIndex` as environment variables may still suit your needs; however, if targeting one NodeID is too limiting or irrelevant, instead of passing a specific NodeID to your broker Pods, you could specify any other environment variables via `--set opcua.configuration.brokerProperties.KEY='VALUE'`. Or, your broker may not need additional information passed to it at all. Decide whether to pass environment variables, what servers to discover, and set the broker pod image to be your container image, say `ghcr.io//opcua-broker`. ```bash helm repo add akri-helm-charts https://project-akri.github.io/akri/ helm install akri akri-helm-charts/akri \ --set opcua.discovery.enabled=true \ --set opcua.configuration.enabled=true \ --set opcua.configuration.discoveryDetails.discoveryUrls[0]="opc.tcp://:50000/" \ --set opcua.configuration.discoveryDetails.discoveryUrls[1]="opc.tcp://:50001/" \ --set opcua.configuration.brokerPod.image.repository='ghcr.io//opcua-broker' # --set opcua.configuration.mountCertificates='true' ``` > Note: set `opcua.configuration.brokerPod.image.tag` to specify an image tag (defaults to `latest`). Now, your broker will be deployed to all discovered OPC UA servers. Next, you can create a Kubernetes deployment for your own end application like [anomaly-detection-app.yaml](https://github.com/project-akri/akri/blob/main/deployment/samples/akri-anomaly-detection-app.yaml) and apply it to your Kubernetes cluster. ### Creating a new OPC UA Configuration Helm allows us to parametrize the commonly modified fields in our Configuration files, and we have provided many. Run `helm inspect values akri-helm-charts/akri` to see what values of the generic OPC UA Configuration can be customized, such as the Configuration and Instance `ServiceSpec`s, `capacity`, and broker `PodSpec`. We saw in the previous section how broker Pod environment variables can be specified via `--set opcua.configuration.brokerProperties.KEY='VALUE'`. For more advanced configuration changes that are not aided by the generic OPC UA Configuration Helm chart, such as credentials naming, we suggest downloading the OPC UA Configuration file using Helm and then manually modifying it. See the documentation on [customizing an Akri installation](/user-guide/customizing-an-akri-installation) for more details. # Discovering and Using Authentication-Enabled Onvif Cameras Make sure you have at least one Onvif camera that is reachable so Onvif discovery handler can discovery your Onvif camera. To test accessing Onvif with credentials, make sure your Onvif camera is authentication-enabled. **Write down the username and password**, they are required in the flow below. ## Preparation Add Akri helm chart repo. ```bash # add akri helm charts repo helm repo add akri-helm-charts https://project-akri.github.io/akri/ # ensure helm repos are up-to-date helm repo update ``` ## Acquire Onvif camera's device uuid In real product scenarios, the device uuids are acquired directly from the vendors or already known before installing Akri Configuration. If you already know the device uuids, you can skip this and go to the next step. First use the following helm chart to deploy an Akri Configuration and see if your camera is discovered. ```bash helm install akri akri-helm-charts/akri-dev \ --set onvif.discovery.enabled=true \ --set onvif.configuration.name=akri-onvif \ --set onvif.configuration.enabled=true \ --set onvif.configuration.capacity=3 \ --set onvif.configuration.brokerPod.image.repository="nginx" \ --set onvif.configuration.brokerPod.image.tag="stable-alpine" ``` Here is the result of running the installation command above on a cluster with 1 control plane and 2 work nodes. There is one Onvif camera connects to the network, thus 1 pods running on each node. ```bash= $ kubectl get nodes,akric,akrii,pods NAME STATUS ROLES AGE VERSION node/kube-01 Ready control-plane 22d v1.26.1 node/kube-02 Ready 22d v1.26.1 node/kube-03 Ready 22d v1.26.1 NAME CAPACITY AGE configuration.akri.sh/akri-onvif 3 62s NAME CONFIG SHARED NODES AGE instance.akri.sh/akri-onvif-029957 akri-onvif true ["kube-03","kube-02"] 48s NAME READY STATUS RESTARTS AGE pod/akri-agent-daemonset-gnwb5 1/1 Running 0 62s pod/akri-agent-daemonset-zn2gb 1/1 Running 0 62s pod/akri-controller-deployment-56b9796c5-wqdwr 1/1 Running 0 62s pod/akri-onvif-discovery-daemonset-wcp2f 1/1 Running 0 62s pod/akri-onvif-discovery-daemonset-xml6t 1/1 Running 0 62s pod/akri-webhook-configuration-75d9b95fbc-wqhgw 1/1 Running 0 62s pod/kube-02-akri-onvif-029957-pod 1/1 Running 0 48s pod/kube-03-akri-onvif-029957-pod 1/1 Running 0 48s ``` Get the device uuid from the Akri Instance. Below is an example, the Onvif discovery handler discovers the camera and expose the device's uuid. **Write down the device uuid for later use**. Note that in real product scenarios, the device uuids are acquired directly from the vendors or already known before installing Akri Configuration. ```bash= $ kubectl get akrii akri-onvif-029957 -o yaml | grep ONVIF_DEVICE_UUID ONVIF_DEVICE_UUID: 3fa1fe68-b915-4053-a3e1-ac15a21f5f91 ``` ## Set up Kubernetes secrets Now we can set up the credential information to Kubernetes Secret. Replace the device uuid and the values of username/password with information of your camera. ```bash cat > /tmp/onvif-auth-secret.yaml<< EOF --- apiVersion: v1 kind: Secret metadata: name: onvif-auth-secret type: Opaque stringData: device_credential_list: |+ [ "credential_list" ] credential_list: |+ { "3fa1fe68-b915-4053-a3e1-ac15a21f5f91" : { "username" : "camuser", "password" : "HappyDay" } } EOF # add the secret to cluster kubectl apply -f /tmp/onvif-auth-secret.yaml ``` ## Upgrade the Akri configuration Upgrade the Akri Configuration to include the secret information and the sample video broker container. ```bash helm upgrade akri akri-helm-charts/akri-dev \ --install \ --set onvif.discovery.enabled=true \ --set onvif.configuration.enabled=true \ --set onvif.configuration.capacity=3 \ --set onvif.configuration.discoveryProperties[0].name=device_credential_list \ --set onvif.configuration.discoveryProperties[0].valueFrom.secretKeyRef.name=onvif-auth-secret \ --set onvif.configuration.discoveryProperties[0].valueFrom.secretKeyRef.namespace=default \ --set onvif.configuration.discoveryProperties[0].valueFrom.secretKeyRef.key=device_credential_list \ --set onvif.configuration.discoveryProperties[0].valueFrom.secretKeyRef.optional=false \ --set onvif.configuration.brokerPod.image.repository="ghcr.io/project-akri/examples/onvif-video-broker" \ --set onvif.configuration.brokerPod.image.tag="latest-dev" \ --set onvif.configuration.brokerPod.image.pullPolicy="Always" \ --set onvif.configuration.brokerProperties.CREDENTIAL_DIRECTORY="/etc/credential_directory" \ --set onvif.configuration.brokerProperties.CREDENTIAL_CONFIGMAP_DIRECTORY="/etc/credential_cfgmap_directory" \ --set onvif.configuration.brokerPod.volumeMounts[0].name="credentials" \ --set onvif.configuration.brokerPod.volumeMounts[0].mountPath="/etc/credential_directory" \ --set onvif.configuration.brokerPod.volumeMounts[0].readOnly=true \ --set onvif.configuration.brokerPod.volumes[0].name="credentials" \ --set onvif.configuration.brokerPod.volumes[0].secret.secretName="onvif-auth-secret" ``` With the secret information, the Onvif discovery handler is able to discovery the Onvif camera and the video broker is up and running ```bash= $ kubectl get nodes,akric,akrii,pods NAME STATUS ROLES AGE VERSION node/kube-01 Ready control-plane 22d v1.26.1 node/kube-02 Ready 22d v1.26.1 node/kube-03 Ready 22d v1.26.1 NAME CAPACITY AGE configuration.akri.sh/akri-onvif 3 18m NAME CONFIG SHARED NODES AGE instance.akri.sh/akri-onvif-029957 akri-onvif true ["kube-03","kube-02"] 22s NAME READY STATUS RESTARTS AGE pod/akri-agent-daemonset-bq494 1/1 Running 0 18m pod/akri-agent-daemonset-c2rng 1/1 Running 0 18m pod/akri-controller-deployment-56b9796c5-rtm5q 1/1 Running 0 18m pod/akri-onvif-discovery-daemonset-rbgwq 1/1 Running 0 18m pod/akri-onvif-discovery-daemonset-xwjlp 1/1 Running 0 18m pod/akri-webhook-configuration-75d9b95fbc-cr6bc 1/1 Running 0 18m pod/kube-02-akri-onvif-029957-pod 1/1 Running 0 22s pod/kube-03-akri-onvif-029957-pod 1/1 Running 0 22s # dump the logs from sample video broker $ kubectl logs kube-02-akri-onvif-029957-pod [Akri] ONVIF request http://192.168.1.145:2020/onvif/device_service http://www.onvif.org/ver10/device/wsdl/GetService [Akri] ONVIF media url http://192.168.1.145:2020/onvif/service [Akri] ONVIF request http://192.168.1.145:2020/onvif/service http://www.onvif.org/ver10/media/wsdl/GetProfiles [Akri] ONVIF profile list contains: profile_1 [Akri] ONVIF profile list contains: profile_2 [Akri] ONVIF profile list profile_1 [Akri] ONVIF request http://192.168.1.145:2020/onvif/service http://www.onvif.org/ver10/media/wsdl/GetStreamUri [Akri] ONVIF streaming uri list contains: rtsp://192.168.1.145:554/stream1 [Akri] ONVIF streaming uri rtsp://192.168.1.145:554/stream1 [VideoProcessor] Processing RTSP stream: rtsp://----:----@192.168.1.145:554/stream1 info: Microsoft.Hosting.Lifetime[0] Now listening on: http://[::]:8083 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down. info: Microsoft.Hosting.Lifetime[0] Hosting environment: Production info: Microsoft.Hosting.Lifetime[0] Content root path: /app Ready True Adding frame from rtsp://----:----@192.168.1.145:554/stream1, Q size: 1, frame size: 862986 Adding frame from rtsp://----:----@192.168.1.145:554/stream1, Q size: 2, frame size: 865793 Adding frame from rtsp://----:----@192.168.1.145:554/stream1, Q size: 2, frame size: 868048 Adding frame from rtsp://----:----@192.168.1.145:554/stream1, Q size: 2, frame size: 869655 Adding frame from rtsp://----:----@192.168.1.145:554/stream1, Q size: 2, frame size: 871353 ``` ## Deploying the sample video streaming application Deploy the sample video streaming application Instructions described from the step 4 of [camera demo](https://docs.akri.sh/demos/usb-camera-demo#inspecting-akri) Deploy a video streaming web application that points to both the Configuration and Instance level services that were automatically created by Akri. Copy and paste the contents into a file and save it as `akri-video-streaming-app.yaml` ```bash cat > /tmp/akri-video-streaming-app.yaml<< EOF --- apiVersion: apps/v1 kind: Deployment metadata: name: akri-video-streaming-app spec: replicas: 1 selector: matchLabels: app: akri-video-streaming-app template: metadata: labels: app: akri-video-streaming-app spec: serviceAccountName: akri-video-streaming-app-sa containers: - name: akri-video-streaming-app image: ghcr.io/project-akri/examples/video-streaming-app:latest-dev imagePullPolicy: Always securityContext: runAsUser: 1000 allowPrivilegeEscalation: false runAsNonRoot: true readOnlyRootFilesystem: true capabilities: drop: ["ALL"] env: # Streamer works in two modes; either specify the following commented # block of env vars to explicitly target cameras (update the s for # your specific cameras) or # specify a Akri configuration name to pick up cameras automatically # - name: CAMERAS_SOURCE_SVC # value: "akri-udev-video-svc" # - name: CAMERA_COUNT # value: "2" # - name: CAMERA1_SOURCE_SVC # value: "akri-udev-video--svc" # - name: CAMERA2_SOURCE_SVC # value: "akri-udev-video--svc" - name: CONFIGURATION_NAME value: akri-onvif --- apiVersion: v1 kind: Service metadata: name: akri-video-streaming-app namespace: default labels: app: akri-video-streaming-app spec: selector: app: akri-video-streaming-app ports: - name: http port: 80 targetPort: 5000 type: NodePort --- apiVersion: v1 kind: ServiceAccount metadata: name: akri-video-streaming-app-sa --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: akri-video-streaming-app-role rules: - apiGroups: [""] resources: ["services"] verbs: ["list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: akri-video-streaming-app-binding roleRef: apiGroup: "" kind: ClusterRole name: akri-video-streaming-app-role subjects: - kind: ServiceAccount name: akri-video-streaming-app-sa namespace: default EOF ``` Deploy the video stream app ```bash kubectl apply -f /tmp/akri-video-streaming-app.yaml ``` Determine which port the service is running on. **Save this port number for the next step**: ```bash kubectl get service/akri-video-streaming-app --output=jsonpath='{.spec.ports[?(@.name=="http")].nodePort}' && echo ``` SSH port forwarding can be used to access the streaming application. Open a new terminal, enter your ssh command to to access your machine followed by the port forwarding request. The following command will use port 50000 on the host. Feel free to change it if it is not available. Be sure to replace `` with the port number outputted in the previous step. ```bash= ssh someuser@ -L 50000:localhost: ``` Navigate to using browser. The large feed points to Configuration level service, while the bottom feed points to the service for each Instance or camera. ## Clean up Close the page from the browser Delete the sample streaming application resources ```bash kubectl delete -f /tmp/akri-video-streaming-app.yaml ``` Delete the Secret information ```bash kubectl delete -f /tmp/onvif-auth-secret.yaml ``` Delete deployment and Akri installation to clean up the system. ```bash helm delete akri kubectl delete crd configurations.akri.sh kubectl delete crd instances.akri.sh ``` # Introduction and Demo Videos If you prefer to learn through videos rather than written documentation, the following is a list of informative talks and demos on Akri. 1. [Bridge Your IoT Leaf Devices to Local Clusters with Ease Using Akri and Dynamic Resource Allocation](https://www.youtube.com/watch?v=ZZfDne7gMMI) - Latest Akri introduction at KubeCon EU 2024. 2. [Introducing industrial edge](https://www.youtube.com/watch?v=A3Lr7kZtUXo) - An introduction to Akri and how it fits to SUSE's industrial edge solution. Includes a demo of discovering an USB camera. 3. [Azure Arc Jumpstart with Akri](https://www.youtube.com/watch?v=KLA3N8xbYPE) - A talk in the Azure Arc Jumpstart channel. Includes a demo of discovering an ONVIF camera with Akri and feeding the stream to an edge AI model. 4. [Discovering and Managing IoT Devices from Kubernetes with Akri](https://www.youtube.com/watch?v=9wCQCV0m5Kk) - A deep dive for Akri. Includes a step-by-step demo of discovering the ONVIF cameras and performing firmware update. To try more demos/examples with step-by-step guidance, check the rest of the pages under [Demo](https://docs.akri.sh/demos/usb-camera-demo) section. # Overview This document will describe Akri's components. The word "resource" is used to describe what is being searched for and ultimately utilized. Resources offer services. For example, they can be USB or IP cameras, which serve video frames, or GPUs, which provide computation. They can be locally attached, embedded, or remotely accessible to worker nodes, such as USB devices, GPUs, and IP cameras, respectively. ## How Akri Works Akri's architecture is made up of five key components: two custom resources, Discovery Handlers, an Agent (device plugin implementation), and a custom Controller. The first custom resource, the Akri Configuration, is where **you name it**. This tells Akri what kind of device it should look for. At this point, **Akri finds it**! Akri's Discovery Handlers look for the device and inform the Agent of discovered devices. The Agent then creates Akri's second custom resource, the Akri Instance, to track the availability and usage of the device. Having found your device, the Akri Controller helps **you use it**. It sees each Akri Instance (which represents a leaf device) and deploys a ("broker") Pod that knows how to connect to the resource and utilize it. ![](/files/-MgSrsaHZqjEoggTnkNn) ## Custom Resource Definitions There are two Akri CRDs: 1. Configuration 2. Instance ### Akri Configuration CRD The configuration of Akri is enabled by the Configuration CRD. Akri users will create Configurations to describe what resources should be discovered and what pod should be deployed on the nodes that discover a resource. Take a look at the [Akri Configuration CRD](https://github.com/project-akri/akri/blob/main/deployment/helm/crds/akri-configuration-crd.yaml). It specifies what components all Configurations must have, including the following: * the desired discovery protocol used for finding resources, i.e. ONVIF, OPC-UA or udev. * a capacity (spec.capacity) that defines the maximum number of nodes that may schedule workloads on this resource. * a PodSpec (spec.brokerPodSpec) that defines the "broker" pod that will be scheduled to each of these reported resources. * a ServiceSpec (spec.instanceServiceSpec) that defines the service that provides a single stable endpoint to access each individual resource's set of broker pods. * a ServiceSpec (spec.configurationServiceSpec) that defines the service that provides a single stable endpoint to access the set of all brokers for all resources associated with the Configuration. Akri Helm Chart already provides three Configurations, one for discovering IP cameras using the ONVIF protocol, one for OPC-UA devices, and one for discovering node devices via udev. Let's look at an [example ONVIF Configuration yaml](https://github.com/project-akri/akri/blob/main/test/yaml/akri-onvif-video-configuration.yaml). You can see it specifies the protocol ONVIF, an image for the broker pod, a capacity of 5, and two Kubernetes services. In this case, the broker pod is a sample frame server we have provided. To get only the frames from a specific camera, a user could point an application at the Instance service, while the Configuration service provides the frames from all the cameras.The ONVIF Configuration can be customized using Helm. When installing the ONVIF Configuration to your Akri enabled cluster, you can specify [the values](https://github.com/project-akri/akri/blob/main/deployment/helm/values.yaml) you want to be inserted into the [ONVIF Configuration template](https://github.com/project-akri/akri/blob/main/deployment/helm/templates/onvif-configuration.yaml). Learn more about [deploying the ONVIF sample here](/discovery-handlers/onvif). ### Akri Instance CRD Each Instance represents an individual resource that is visible to the cluster. So, if there are 5 IP cameras visible to the cluster, there will be 5 Instances. Akri coordination and resource sharing is enabled by the Instance CRD. These instances store internal Akri state and are not intended to be edited by users. For a more in-depth understanding on how resource sharing is accomplished, see [Resource Sharing In-depth](/architecture/resource-sharing-in-depth). ## Agent The Akri Agent implements [Kubernetes Device-Plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) for discovered resources. The basic flow of the Akri Agent is: 1. Watch for Configuration changes to determine what resources to search for 2. Monitor resource availability (as edge devices may come and go) to determine what resources to advertise 3. Inform Kubernetes of resource health/availability as it changes This basic flow combined with the state stored in the Instance allows multiple nodes to share a resource while respecting the limitations defined by Configuration.capacity. For a more in-depth understanding, see [Agent In-depth](/architecture/agent-in-depth). ## Discovery Handlers A Discovery Handlers discover devices around the cluster, whether connected to Nodes (ie USB sensors), embedded in Nodes (ie GPUs), or on the network (ie IP cameras) and report them to the Agent. They are oftentimes protocol implementations for discovering a set of devices, whether a network protocol like OPC UA or a proprietary protocol. Discovery Handlers implement the `DiscoveryHandler` service defined in [`discovery.proto`](https://github.com/project-akri/akri/blob/main/discovery-utils/proto/discovery.proto). In order to be utilized, a Discovery Handler must register with the Agent, which hosts the `Registration` service defined in [`discovery.proto`](https://github.com/project-akri/akri/blob/main/discovery-utils/proto/discovery.proto). To get started creating a Discovery Handler, see [Discovery Handler development](/development/handler-development). ## Controller The Akri controller serves two purposes: 1. Handle (create and/or delete) the Pods & Services that enable resource availability 2. Ensure that Instances are aligned to the cluster state at any given moment To achieve these goals, the basic flow of the controller is: 1. Watch for Instance changes to determine what Pods and Services should exist 2. Watch for Nodes that are contained in Instances that no longer exist This basic flow allows the Akri controller to ensure that protocol brokers and Kubernetes Services are running on all nodes exposing desired resources while respecting the limitations defined by `Configuration.capacity`. For a more in-depth understanding, see [Controller In-depth](/architecture/controller-in-depth). ## Akri Flow - In Depth ![](/files/-MgSrsaJ1iNsufKLwFms) {% hint style="info" %} For the sake of this example, some content has been excluded from the Pod, Configuration and Instances shown below. {% endhint %} 1. Operator applies a Configuration with a capacity of 3 to the single node cluster. ```yaml kind: Configuration metadata: name: akri- spec: discoveryHandler: name: protocolA discovery_details: {} brokerPodSpec: containers: - name: custom-broker image: "ghcr.io/…" # ... capacity: 3 ``` 2. The Akri Agent sees the Configuration and discovers a leaf device using the protocol specified in the Configuration. It creates a device plugin for that leaf device and registers it with the kubelet. When creating the device plugin, it tells the kubelet to set connection information for that specific device and additional metadata from a Configuration's `brokerProperties` as environment variables in all Pods that request this device's resource. This information is also set in the `brokerProperties` section of the Instance the Agent creates to represent the discovered leaf device. In the Instance, the Agent also lists itself as a node that can access the device under `nodes`. Note how Instance has 3 available `deviceUsage` slots, since capacity was set to 3 and no brokers have been scheduled to the leaf device yet. ```yaml kind: Instance metadata: name: akri-- spec: configurationName: akri- shared: true deviceUsage: akri---0: "" akri---1: "" akri---2: "" brokerProperties: BROKER_ENV_VAR_1: BROKER_ENV_VAR_N: nodes: - "" ``` 3. The Controller is notified by the API Server of Instance changes. It is informed that a new Instance has been created. It schedules a pod to one of the nodes on the Instance’s nodes list, adding the Instance’s name as a resource limit of the pod. Note that the pod is currently in pending state. ```yaml kind: Pod metadata: labels: app: akri---pod controller: akri.sh akri.sh/configuration: akri- akri.sh/instance: akri-- akri.sh/target-node: name: akri---pod spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchFields: - key: metadata.name operator: In values: - containers: image: ghcr.io/… name: custom-broker resources: limits: akri.sh/akri--: "1" requests: akri.sh/akri--: "1" status: # ... phase: Pending ``` 4. The kubelet on the selected node sees the scheduled pod and resource limit. It checks to see if the resource is available by calling `allocate` on the device plugin running in the Agent for the requested leaf device. When calling `allocate`, the kubelet requests a specific `deviceUsage` slot. Let's say the kubelet requested `akri---1`. The leaf device's device plugin checks to see that the requested `deviceUsage` slot has not been taken by another node. If it is available, it reserves that `deviceUsage` slot for this node (as shown below) and returns true. In the `allocate` response, the Agent also tells kubelet to mount the `Instance.brokerProperties` as environment variables in the broker Pod. ```yaml kind: Instance metadata: name: akri-- spec: configurationName: akri- shared: true deviceUsage: akri---0: "" akri---1: "" akri---2: "" brokerProperties: BROKER_ENV_VAR_1: BROKER_ENV_VAR_N: nodes: - "" ``` 5. Allocate will return `false` if kubelet requests a `deviceUsage` slot that is already taken. See the [resource sharing document](/architecture/resource-sharing-in-depth) for a better understanding on how this is resolved. Otherwise, upon a `true` result, the kubelet will run the pod. The broker is now running and has the information necessary to communicate with the specific device. # Resource Sharing To enable multiple nodes to share a single resource, there are two vital pieces: 1. The `Configuration.capacity` 2. The `Instance.deviceUsage` ## Configuration.capacity The configuration's capacity determines how many Nodes are allowed to schedule a workload for a given resource. If the capacity is set to 5 and there are 10 worker nodes that can access the resource, only 5 will have Running workloads at any given moment (the remaining nodes will have workloads in a Pending state). This provides 2 important values: 1. High availability - if a Running workload stops or fails, one of the Pending workloads will be scheduled and will start Running 2. Connection throttling - this supports resources that can only handle so many requests or connections at once ## Instance.deviceUsage When the Akri Agent discovers a resource and creates an Instance, the deviceUsage map is initialized based on the `Configuration.capacity`. If the capacity is 5, then the deviceUsage map will have 5 mappings, or slots. The slots are named using a simple pattern, in this case, the initial deviceUsage might look like: ```yaml deviceUsage: my-resource-00095f-0: "" my-resource-00095f-1: "" my-resource-00095f-2: "" my-resource-00095f-3: "" my-resource-00095f-4: "" ``` Each slot is initialized to be mapped to an empty string, signifying that no Node is utilizing this slot. When a Node utilizes a slot, it will change the mapping to include its name (i.e., `my-resource-00095f-2: "node-a"`) During this initialization, a separate, but similar, mapping is sent to the kubelet ... for our example with 5 unutilized slots, this mapping would look like this: ```yaml my-resource-00095f-0: "Healthy" my-resource-00095f-1: "Healthy" my-resource-00095f-2: "Healthy" my-resource-00095f-3: "Healthy" my-resource-00095f-4: "Healthy" ``` When the kubelet attempts to schedule a workload on a specific Node, that Node's Akri Agent will be queried with a slot name (this slot name is chosen by the kubelet from the mapping list that Akri Agent sent it). Akri Agent will query the appropriate Instance to see if that resource is still visible and if the mapping for that slot is still empty. If both of these requirements are met, then the Akri Agent will update the `Instance.deviceUsage` map to claim the slot, and will allow the kubelet to schedule its intended workload. After this, the `Instance.deviceUsage` may look something like this: ```yaml deviceUsage: my-resource-00095f-0: "" my-resource-00095f-1: "" my-resource-00095f-2: "" my-resource-00095f-3: "node-a" my-resource-00095f-4: "" ``` When this Instance is changed, in this case for `node-a` to claim slot `my-resource-00095f-3`, every Akri Agent that can access this instance will react by notifying the kubelet that this slot is no longer available: ```yaml my-resource-00095f-0: "Healthy" my-resource-00095f-1: "Healthy" my-resource-00095f-2: "Healthy" my-resource-00095f-3: "Unhealthy" my-resource-00095f-4: "Healthy" ``` These two steps will ensure that a specific slot is only used by one Node. There is a possible race condition here. What happens if Kubernetes tries to schedule a workload after the `Instance.deviceUsage` slot has been claimed, but before other Nodes have reported the slot as Unhealthy? In this case, we can depend on the Instance as the truth. If the kubelet sends a query with a slot name that is claimed by another node in `Instance.deviceUsage`, an error is returned to the kubelet and the workload will not be scheduled. Instead, the pod will stay in a `Pending` state until the Akri Controller brings it down. The Akri Agent will immediately notify the kubelet of the accurate `deviceUsage` slot availability and continue to periodically do this (as usual). Once the pod has been brought down by the Controller, if there are still some slots available, the Controller may reschedule the pod to that Node. Then, the kubelet can attempt to reserve a slot again, this time hopefully not hitting a collision. ### Special case: workload disappearance There is one case that is not addressed above: when a workload fails, finishes, or generally no longer exists. In this case, the slot that the workload claimed needs to be released. Unfortunately, the kubelet's Device-Plugin framework does not make finding this simple. There is no deallocate or "pod failed" notification. However, the kubelet does let Akri Agent access a list of workloads currently using slots (and the slots they are using), this is used to clear the `Instance.deviceUsage` map when a slot is used there, but not reflected in the kubelet, with a 20 seconds delay ensuring the workload got enough time to spawn. # Controller The Akri Controller executes on the master Node in the cluster. It is primarily tasked with: 1. Enabling cluster access to leaf devices 2. Handling node disappearances These tasks enable Akri to provide resources with high availability, while allowing the Kubernetes application to be agnostic about what specific Nodes or Pods are executing at any given moment. ## Enabling cluster access to resources The first step to enable cluster access to resources (leaf devices) is, of course, finding them. The work of discovering resources and making them known to the Kubernetes cluster is handled by the [Akri Agent](/architecture/agent-in-depth). The Akri Agents ensure that Instances are created and updated to enforce capability sharing. Once a capability has been discovered and Instances are created, it is up to the Akri Controller to provide cluster access. To provide access to discovered resources, the Akri Controller works to ensure that the Pods and Services described in the relevant Configuration are running. This is accomplished by listening for changes, additions, and deletions of Instances. When an instance is created or updated, the Akri Controller needs to do several things: 1. Ensure that the protocol broker Pod based on `Configuration.brokerPodSpec` is created 2. Ensure that the broker Service based on `Configuration.instanceServiceSpec` is created 3. Ensure that the capability Service based on `Configuration.configurationServiceSpec` is created When an instance is deleted, the Akri Controller needs to do several things: 1. Ensure that the protocol broker Pod based on `Configuration.brokerPodSpec` is removed 2. Ensure that the protocol broker Service based on `Configuration.instanceServiceSpec` is removed 3. Ensure that the capability Service based on `Configuration.configurationServiceSpec` is removed, if there are no Pods supporting the Service (note that many instances can contribute supporting Pods to a given configuration) ## Handling node disappearances One of the conditions we need to be aware of is node disappearance. In this case, we cannot depend on the disappeared node's Akri Agent to modify the relevant Instance. To free up any `Configuration.capacity` that a node was using prior to disappearing, the Akri Controller watches for Node disappearance events and cleans up any lingering node references in any `Instance.nodes` and `Instance.deviceUsage`. # Agent The Akri Agent executes on all worker Nodes in the cluster. It is primarily tasked with: 1. Handling resource availability changes 2. Enabling resource sharing These two tasks enable Akri to find configured resources (leaf devices), expose them to the Kubernetes cluster for workload scheduling, and allow resources to be shared by multiple Nodes. ## Handling resource availability changes The first step in handling resource availability is determining what resources (leaf devices) to look for. This is accomplished by finding existing Configurations and watching for changes to them. Once the Akri Agent understands what resources to look for (via `Configuration.discovery_handler`), it will [find any resources that are visible](#resource-discovery). For each resource that is found: 1. An Instance is created and uploaded to etcd 2. A connection with the kubelet is established according to the Kubernetes Device Plugin framework. This connection is used to convey availability changes to the kubelet. The kubelet will, in turn, expose these availability changes to the Kubernetes scheduler. Each protocol will periodically reassess what resources are visible and update both the Instance and the kubelet with the current availability. This process allows Akri to dynamically represent resources that appear and disappear. ## Enabling resource sharing To enable resource sharing, the Akri Agent creates and updates the `Instance.deviceUsage` map and communicates with kubelet. The `Instance.deviceUsage` map is used to coordinate between Nodes. The kubelet communication allows Akri Agent to communicate any resource availability changes to the Kubernetes scheduler. For more detailed information, see the [in-depth resource sharing doc](/architecture/resource-sharing-in-depth). Akri Agent also exposes all discovered resources at Configuration level. Configuration level resources can be referred by the name of Configuration so Configuration name can be used to request resources without the need to know the specific Instances id to request. Agent will behind the scenes do the work of selecting which Instances to reserve. For more detailed information about Configuration level resource, see the [Configuration-level resources doc](/architecture/configuration-level-resource-in-depth). ## Resource discovery The Agent discovers resources via Discovery Handlers (DHs). A Discovery Handler is anything that implements the `DiscoveryHandler` service defined in [`discovery.proto`](https://github.com/project-akri/akri/blob/main/discovery-utils/proto/discovery.proto). In order to be utilized, a DH must register with the Agent, which hosts the `Registration` service defined in [`discovery.proto`](https://github.com/project-akri/akri/blob/main/discovery-utils/proto/discovery.proto). The Agent maintains a list of registered DHs and their connectivity statuses, which is either `Waiting`, `Active`, or `Offline(Instant)`. When registered, a DH's status is `Waiting`. Once a Configuration requesting resources discovered by a DH is applied to the Akri-enabled cluster, the Agent will create a connection with the DH requested in the Configuration and set the status of the DH to `Active`. If the Agent is unable to connect or loses a connection with a DH, its status is set to `Offline(Instant)`. The `Instant` marks the time at which the DH became unresponsive. If the DH has been offline for more than 5 minutes, it is removed from the Agent's list of registered Discovery Handlers. If a Configuration is deleted, the Agent drops the connection it made with all DHs for that Configuration and marks the DHs' statuses as `Waiting`. Note, while probably not commonplace, the Agent allows for multiple DHs to be registered for the same protocol. IE: you could have two udev DHs running on a node on different sockets. The Agent's registration service defaults to running on the socket `/var/lib/akri/agent-registration.sock` but can be Configured with Helm. While Discovery Handlers must register with this service over UDS, the Discovery Handler's service can run over UDS or an IP based endpoint. Supported Rust DHs each have a [library](https://github.com/project-akri/akri/tree/main/discovery-handlers) and a [binary implementation](https://github.com/project-akri/akri/tree/main/discovery-handler-modules). This allows them to either be run within the Agent binary or in their own Pod. Reference the [Discovery Handler development document](/development/handler-development) to learn how to implement a Discovery Handler. ## Passing additional properties to Discovery Handlers In addition to the `discoveryDetails` in Configuration that sets details for narrowing the Discovery Handlers' search, the `discoveryProperties` can be used to pass additional information to Discovery Handler. One of scenarios that can leverage `discoveryProperties` is to pass credential data to Discovery Handlers to perform authenticated resource discovery. It is common for a device to require authentication in order to access its properties. The Discovery Handler then need these credentials to properly discover and filter the device. The credential data can be placed in `discoverProperties`, if it is specified in Configuration, Agent reads the content and generate a list of string key-value pair properties and pass the list to Discovery Handler along with `discoveryDetails`. Agent supports plain text, K8s `secret` and `configMap` in the schema of `discoverProperties`. An example below shows how each type of property is specified in `discoveryProperties`. The `name` of property is required and needs to be in C\_IDENTIFIER format. The value can be specified by `value` or `valueFrom`. For value specified by `valueFrom`, it can be from `secret` or `configMap`. The `optional` attribute is default to `false`, it means if the data doesn't exist (in the `secret` or `configMap`), the Configuration deployment will fail. If `optional` is `true`, Agent will ignore the entry if the data doesn't exist, and pass all exist properties to Discovery Handler, the Configuration deployment will success. ```yaml discoveryProperties: - name: property_from_plain_text value: “plain text data” - name: property_from_secret valueFrom: secretKeyRef: name: mysecret namespace: mysecret-namespace key: secret-key optional: false - name: property_from_configmap valueFrom: configMapKeyRef: name: myconfigMap namespace: myconfigmap-namespace key: configmap-key optional: true ``` For the example above, with the content of secret and configMap. ```yaml --- apiVersion: v1 kind: Secret metadata: name: mysecret namespace: mysecret-namespace type: Opaque stringData: secret-key: "secret1" --- apiVersion: v1 kind: ConfigMap metadata: name: myconfigMap namespace: myconfigmap-namespace data: configmap-key: "configmap1" ``` Agent read all properties and pass the string key-value pair list to Discovery Handle. ```yaml "property_from_plain_text": “plain text data” "property_from_secret": "secret1" "property_from_configmap": "configmap1" ``` # Configuration-Level Resources Akri supports creating a Kubernetes resource (i.e. device plugin) for each individual device. Since each device in Akri is represented as an Instance custom resource, these are called Instance-level resources. Instance-level resources are named in the format `-`. Akri also creates a Kubernetes Device Plugin for a Configuration called Configuration-level resource. A Configuration-level resource is a resource that represents all of the devices discovered via a Configuration. With Configuration-level resources, instead of needing to know the specific Instances to request, resources could be requested by the Configuration name and the Agent will do the work of selecting which Instances to reserve. The example below shows a deployment that requests the resource at Configuration level and would deploy a nginx broker to each discovered device respectively. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: onvif-camera-broker-deployment labels: app: onvif-camera-broker spec: replicas: 1 selector: matchLabels: app: onvif-camera-broker template: metadata: labels: app: onvif-camera-broker spec: containers: - name: onvif-camera-broker image: nginx resources: limits: akri.sh/onvif-camera: "2" requests: akri.sh/onvif-camera: "2" ``` With Configuration-level resources, users could use higher level Kubernetes objects (Deployments, ReplicaSets, DaemonSets, etc.) or develop their own deployment strategies, rather than relying on the Akri Controller to deploy Pods to discovered devices. ### Maintaining Device Usage The [in-depth resource sharing doc](/architecture/resource-sharing-in-depth) describes how the `Configuration.capacity` and `Instance.deviceUsage` are used to achieve resource sharing between nodes. The same data is used to achieve sharing the same resource between Configuration-level and Instance-level resources. The `Instance.deviceUsage` in Akri Instances is extended to support Configuration device plugin. The `Instance.deviceUsage` may look like this: ```yaml deviceUsage: my-resource-00095f-0: "" my-resource-00095f-1: "" my-resource-00095f-2: "" my-resource-00095f-3: "node-a" my-resource-00095f-4: "" ``` where empty string means the slot is free and non-empty string indicates the slot is used (by the node). To support Configuration device plugin, the `Instance.deviceUsage` format is extended to hold the additional information, the deviceUsage can be a "\" (for Instance) or a "C:\:\" (for Configuration). For example, the `Instance.deviceUsage` shows the slot `my-resource-00095f-2` is used by virtual device id "0" of the Configuration device plugin on `node-b`. The slot `my-resource-00095f-3` is used by Instance device plugin on `node-a`. The other 3 slots are free. ```yaml deviceUsage: my-resource-00095f-0: "" my-resource-00095f-1: "" my-resource-00095f-2: "C:0:node-b" my-resource-00095f-3: "node-a" my-resource-00095f-4: "" ``` ## Deployment Strategies with Configuration-level resources The Akri Agent and Discovery Handlers enable device discovery and Kubernetes resource creation: they discover devices, create Kubernetes resources to represent the devices, and ensure only `capacity` containers are using a device at once via the device plugin framework. The Akri Controller eases device use. If a broker is specified in a Configuration, the Controller will automatically deploy Kubernetes Pods or Jobs to discovered devices. Currently the Controller only supports two deployment strategies: either deploying a non-terminating Pod (that Akri calls a "broker") to each Node that can see a device or deploying a single Job to the cluster for each device discovered. There are plenty of scenarios that do not fit these two strategies such as a ReplicaSet like deployment of n number of Pods to the cluster. With Configuration-level resources, users could easily achieve their own scenarios without the Akri Controller, as selecting resources is more declarative. A user specifies in a resource request how many OPC UA servers are needed rather than needing to delineate the exact ones already discovered by Akri, as explained in Akri's current documentation on [requesting Akri resources](https://github.com/deislabs/akri-docs/blob/main/docs/docs/user-guide/requesting-akri-resources.md). For example, with Configuration-level resources, the following Deployment could be applied to a cluster: ```yaml apiVersion: "apps/v1" kind: Deployment metadata: name: onvif-broker-deployment spec: replicas: 2 selector: matchLabels: name: onvif-broker template: metadata: labels: name: onvif-broker spec: containers: - name: nginx image: "nginx:latest" resources: requests: "akri.sh/akri-onvif": "2" limits: "akri.sh/akri-onvif": "2" ``` Pods will only be successfully scheduled to a Node and run if the resources exist and are available. In the case of the above scenario, if there were two cameras on the network, two Pods would be deployed to the cluster. If there are not enough resources, say there is only one camera on the network, the two Pods will be left in a `Pending` state until another is discovered. This is the case with any deployment on Kubernetes where there are not enough resources. However, `Pending` Pods do not use up cluster resources. # Developer Guide This document will walk you through how to set up a local development environment, build Akri component containers, and test Akri using your newly built containers. It also includes instructions on running Akri locally, naming guidelines, and points to documentation on extending Akri with new Discovery Handlers and brokers. > Note: different tools are needed depending on what parts of Akri you are developing. This document aims to make that clear. ## Table of Contents * [Requirements](#requirements) * [Build and Test Akri's Components](#build-and-test-rust-components) * [Running Akri's Components Locally](#running-locally) * [Building Akri Containers](#building-bontainers) * [Installing Akri with newly built containers](#installing-akri-with-newly-built-containers) * [Useful Helm commands](#useful-helm-commands) * [Testing with Debug Echo Discovery Handler](#testing-with-debug-echo-discovery-handler) * [Discovery Handler and Broker Development](#discovery-handler-and-broker-bevelopment) * [Developing Akri's non-Rust components](#developing-non-rust-components) * [Naming Guidelines](#naming-guidelines) ## Requirements ### Linux Environment To develop, you'll **need a Linux environment** whether on amd64 or arm64v8. We recommend using an Ubuntu VM; however, WSL2 should work for building and testing (but has not been extensively tested). ### Tools for developing Akri's Rust components The majority of Akri is written in Rust. To install Rust and Akri's component's dependencies, run Akri's setup script: ```sh ./build/setup.sh ``` If you previously installed Rust ensure you are using the v1.88.0 toolchain that Akri's build system uses: ```sh sudo curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain=1.88.0 rustup default 1.88.0 cargo version ``` ## Build and test Rust components 1. Fork and clone [Akri](https://github.com/project-akri/akri). Then, navigate to the repo's top folder. 2. To install Rust and Akri's component's dependencies, run Akri's setup script: ```sh ./build/setup.sh ``` If you previously installed Rust, ensure you are using the v1.88.0 toolchain that Akri's build system uses: ```sh sudo curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain=1.88.0 ``` Then, configure your current shell to see Cargo and set `v1.88.0` as default toolchain. ```sh source $HOME/.cargo/env rustup default 1.88.0 cargo version ``` 3. Build Controller, Agent, Discovery Handlers, and udev broker ```sh cargo build ``` > Note: To build a specific component, use the `-p` parameter along with the [workspace member](https://github.com/project-akri/akri/blob/main/Cargo.toml). For example, to only build the Agent, run `cargo build -p agent` 4. To run all unit tests: ```sh cargo test ``` > Note: To test a specific component, use the `-p` parameter along with the [workspace member](https://github.com/project-akri/akri/blob/main/Cargo.toml). For example, to only test the Agent, run `cargo test -p agent` ## Running locally Before running Akri agent or controller locally, please ensure the Akri configuration and instance CRDs are applied to cluster, otherwise use the below command to apply them. ```sh kubectl apply -f akri/deployment/helm/crds/akri-configuration-crd.yaml kubectl apply -f akri/deployment/helm/crds/akri-instance-crd.yaml ``` To locally run Akri's Agent, Controller, and Discovery Handlers as part of a Kubernetes cluster, follow these steps: 1. Create or provide access to a valid cluster configuration by setting `KUBECONFIG` (can be done in the command line) ... for the sake of this, the config is assumed to be in `$HOME/.kube/config`. Reference Akri's [cluster setup instructions](https://docs.akri.sh/user-guide/cluster-setup) if needed. 2. Build the repo with all default features by running `cargo build` 3. Run the desired component by navigating to the appropriate directory and using `cargo run` ```` Run the **Controller** locally with info-level logging and using `8081` to serve Akri's metrics (for Prometheus integration): ```sh cd akri/controller RUST_LOG=info METRICS_PORT=8081 KUBECONFIG=$HOME/.kube/config cargo run ``` > `METRICS_PORT` can be set to any value as it is only used if Prometheus is enabled. Just ensure that the Controller and Agent > use different ports if they are both running. Run the **Agent** locally with info-level logging, debug echo enabled for testing, and a metrics port of `8082`. The Agent must be run privileged in order to connect to the kubelet. Specify the user path to cargo `$HOME/.cargo/bin/cargo` so you do not have to re-install cargo for the sudo user: ```sh cd akri/agent sudo -E DEBUG_ECHO_INSTANCES_SHARED=true ENABLE_DEBUG_ECHO=1 RUST_LOG=info METRICS_PORT=8082 KUBECONFIG=$HOME/.kube/config DISCOVERY_HANDLERS_DIRECTORY=~/tmp/akri AGENT_NODE_NAME=myNode $HOME/.cargo/bin/cargo run ``` > Note: `DISCOVERY_HANDLERS_DIRECTORY` is where Akri agent creates an unix domain socket for discovery handler's registration. This example uses ~/tmp/akri that should exist or is created before executing this command. By default, the Agent does not have embedded Discovery Handlers. To allow embedded Discovery Handlers in the Agent, turn on the `agent-full` feature and the feature for each Discovery Handler you wish to embed -- Debug echo is always included if `agent-full` is turned on. For example, to run the Agent with OPC UA, ONVIF, udev, and debug echo Discovery Handlers add the following to the above command: `--features "agent-full udev-feat ```` opcua-feat onvif-feat"\`. ```` To run **Discovery Handlers** locally, simply navigate to the Discovery Handler under `akri/discovery-handler-modules/` and run using `cargo run`, setting where the Discovery Handler socket should be created in the `DISCOVERY_HANDLERS_DIRECTORY` variable. The discovery handlers must be run privileged in order to connect to the Agent. For example, to run the ONVIF Discovery Handler locally: ```sh cd akri/discovery-handler-modules/onvif-discovery-handler/ sudo -E RUST_LOG=info DISCOVERY_HANDLERS_DIRECTORY=~/tmp/akri AGENT_NODE_NAME=myNode $HOME/.cargo/bin/cargo run ``` To run the [debug echo Discovery Handler](#testing-with-debug-echo-discovery-handler), an environment variable, `DEBUG_ECHO_INSTANCES_SHARED`, must be set to specify whether it should register with the Agent as discovering shared or unshared devices. Run the debug echo Discovery Handler to discover mock unshared devices like so: ```sh cd akri/discovery-handler-modules/debug-echo-discovery-handler/ sudo -E RUST_LOG=info DEBUG_ECHO_INSTANCES_SHARED=false DISCOVERY_HANDLERS_DIRECTORY=~/tmp/akri AGENT_NODE_NAME=myNode $HOME/.cargo/bin/cargo run ``` ```` ## Building Containers `Makefile` has been created to help with the more complicated task of building the Akri components and containers for the various supported platforms. ### Tools for building Akri's Rust containers In order to cross-build Akri's Rust code for both ARM and x64 containers, several tools are leveraged. * `qemu` can be installed with: ```sh sudo apt-get install -y qemu qemu qemu-system-misc qemu-user-static qemu-user binfmt-support ``` For `qemu` to be fully configured on Ubuntu 18.04, after running apt-get install, run these commands: ```sh sudo mkdir -p /lib/binfmt.d sudo sh -c 'echo :qemu-arm:M::\\x7fELF\\x01\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x28\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xfe\\xff\\xff\\xff:/usr/bin/qemu-arm-static:F > /lib/binfmt.d/qemu-arm-static.conf' sudo sh -c 'echo :qemu-aarch64:M::\\x7fELF\\x02\\x01\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\xb7\\x00:\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xfe\\xff\\xff\\xff:/usr/bin/qemu-aarch64-static:F > /lib/binfmt.d/qemu-aarch64-static.conf' sudo systemctl restart systemd-binfmt.service ``` ### Establish a container repository Containers for Akri are currently hosted in `ghcr.io/project-akri/akri` using the new [GitHub container registry](https://github.blog/2020-09-01-introducing-github-container-registry/). Any container repository can be used for private containers. If you want to enable GHCR, you can follow the [getting started guide](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry). To build containers, log into the desired repository: ```sh CONTAINER_REPOSITORY= sudo docker login $CONTAINER_REPOSITORY ``` ### Build intermediate containers To ensure quick builds, we have created a number of intermediate containers that rarely change. By default, `Makefile` will try to create containers with tag following this format: `/$USER/: