udev
Background
Udev is the device manager for Linux. It manages device nodes in the /dev
directory, such as microphones, security chips, usb cameras, and so on. Udev can be used to find devices that are attached to or embedded in Linux nodes.
All of Akri's components can be deployed by specifying values in its Helm chart during an installation. This document will cover the values that should be set to (1) deploy the udev Discovery Handlers and (2) apply a Configuration that tells Akri to discover devices using that Discovery Handler.
Deploying the udev Discovery Handler
In order for the Agent to discover udev devices, a udev Discovery Handler must exist. Akri supports an Agent image that includes all supported Discovery Handlers. This Agent will be used if agent.full=true
is set. By default, a slim Agent without any embedded Discovery Handlers is deployed and the required Discovery Handlers can be deployed as DaemonSets. This documentation will use that strategy, deploying udev Discovery Handlers by specifying udev.discovery.enabled=true
when installing Akri.
udev Configuration Settings
Instead of having to assemble your own udev Configuration yaml, we have provided a Helm template. Helm allows us to parametrize the commonly modified fields in our configuration files, and we have provided many for udev (to see them, run helm inspect values akri-helm-charts/akri
). To apply the udev Configuration to your cluster, simply set udev.configuration.enabled=true
when installing Akri. Be sure to also specify one or more udev rules for the Configuration, as explained below.
Discovery Handler Discovery Details Settings
Discovery Handlers are passed discovery details that are set in a Configuration to determine what to discover, filter out of discovery, and so on. The udev Discovery Handler requires that one discovery detail to be provided: udev rules.
udev.configuration.discoveryDetails.udevRules
array of udev rules
empty
udev.configuration.discoveryDetails.groupRecursive
boolean
false
If set to true, group devices with a matching parent
The udev Discovery Handler parses the udev rules listed in a Configuration, searches for them using udev, and returns a list of discovered device nodes (ie: /dev/video0). It parses the udev rules via a grammar grammar Akri has created. It expects the udev rules to be formatted according to the Linux Man pages.
Udev rule format
While udev rules are normally used to both find devices and perform actions on devices, the Akri udev discovery handler is only interested in finding devices. Consequently, the discovery handler will throw an error if any of the rules contain an action operation ("=" , "+=" , "-=" , ":=") or action fields such as IMPORT
in the udev rules. You should only use match operations ("==", "!=") and the following udev fields: ATTRIBUTE
, ATTRIBUTE
, DEVPATH
, DRIVER
, DRIVERS
, KERNEL
, KERNELS
, ENV
, SUBSYSTEM
, SUBSYSTEMS
, TAG
, and TAGS
. To see some examples, reference our example supported rules and unsupported rules that we run some tests against.
Broker Pod Settings
If you would like non-terminating workloads ("broker" Pods) to be deployed automatically to discovered cameras, a broker image should be specified (under brokerPod
) in the Configuration. Alternatively, if it meets your scenario, you could use the Akri frame server broker ("ghcr.io/project-akri/akri/udev-video-broker"). If you would rather manually deploy pods to utilize the cameras advertized by Akri, don't specify a broker pod and see our documentation on requesting resources advertized by Akri.
Note only a
brokerJob
ORbrokerPod
should be specified.
udev.configuration.brokerPod.image.repository
image string
""
image of broker Pod that should be deployed to discovered devices
udev.configuration.brokerPod.image.tag
tag string
"latest"
image tag of broker Pod that should be deployed to discovered devices
udev.configuration.brokerPod.resources.memoryRequest
string
"10Mi"
the minimum amount of RAM that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri udev sample broker. Adjust to the size of your broker.
udev.configuration.brokerPod.resources.cpuRequest
string
"10m"
the minimum amount of CPU that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri udev sample broker. Adjust to the size of your broker.
udev.configuration.brokerPod.resources.memoryLimit
string
"30Mi"
the maximum amount of RAM this Pod can consume. Default based on the Akri udev sample broker. Adjust to the size of your broker.
udev.configuration.brokerPod.resources.cpuLimit
string
"29m"
the maximum amount of CPU this Pod can consume. Default based on the Akri udev sample broker. Adjust to the size of your broker.
Broker Job Settings
If you would like terminating Jobs to be deployed automatically to discovered devices, a broker image should be specified (under brokerJob
) in the Configuration. A Kubernetes Job deploys a set number of terminating Pods.
Note only a
brokerJob
ORbrokerPod
should be specified.
udev.configuration.brokerJob.image.repository
image string
""
image of broker Job that should be deployed to discovered devices
udev.configuration.brokerJob.image.tag
tag string
"latest"
image tag of broker Job that should be deployed to discovered devices
udev.configuration.brokerJob.resources.memoryRequest
string
"10Mi"
the minimum amount of RAM that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri udev sample broker. Adjust to the size of your broker.
udev.configuration.brokerJob.resources.cpuRequest
string
"10m"
the minimum amount of CPU that must be available to this Pod for it to be scheduled by the Kubernetes Scheduler. Default based on the Akri udev sample broker. Adjust to the size of your broker.
udev.configuration.brokerJob.resources.memoryLimit
string
"30Mi"
the maximum amount of RAM this Pod can consume. Default based on the Akri udev sample broker. Adjust to the size of your broker.
udev.configuration.brokerJob.resources.cpuLimit
string
"29m"
the maximum amount of CPU this Pod can consume. Default based on the Akri udev sample broker. Adjust to the size of your broker.
udev.configuration.brokerJob.command
string array
Empty
command to be executed in the Pod
udev.configuration.brokerJob.restartPolicy
string array
OnFailure
RestartPolicy
for the Job. Can either be OnFailure
or Never
for Jobs.
udev.configuration.brokerJob.backoffLimit
number
2
udev.configuration.brokerJob.parallelism
number
1
udev.configuration.brokerJob.completions
number
1
Disabling Automatic Service Creation
By default, if a broker Pod is specified, the generic udev Configuration will create services for all the brokers of a specific Akri Instance and all the brokers of an Akri Configuration. The creation of these services can be disabled.
udev.configuration.createInstanceServices
true, false
true
a service should be automatically created for each broker Pod
udev.configuration.createConfigurationService
true, false
true
a single service should be created for all brokers of a Configuration
Capacity Setting
By default, if a broker Pod is specified, a single broker Pod is deployed to each device. To modify the Configuration so that a device is accessed by more or fewer nodes via broker Pods, update the udev.configuration.capacity
setting to reflect the correct number. For example, if your high availability needs are met by having 1 redundant pod, you can update the Configuration like this by setting udev.configuration.capacity=2
.
udev.configuration.capacity
number
1
maximum number of brokers that can be deployed to utilize a device (up to 1 per Node)
Choosing a udev rule
To see what devices will be discovered on a specific node by a udev rule, you can use udevadm
. For example, to find all devices in the sound subsystem, you could run:
To see all the properties of a specific device discovered, you can use udevadm info
:
Now, you can see a bunch of attributes you could use to narrow your udev rule. Maybe you decide you want to find all sound devices made by the vendor Great Vendor
. You set the following udev rule under the udev Discovery Handler in your Configuration:
Testing a udev rule
To test which devices Akri will discover with a udev rule, you can run the rule locally adding a tag action to it. Then you can search for all devices with that tag, which will be the ones discovered by Akri.
Create a new rules file called
90-akri.rules
in the/etc/udev/rules.d
directory, and add your udev rule(s) to it. For this example, we will be testing the ruleSUBSYSTEM=="sound", KERNEL=="card[0-9]*"
. AddTAG+="akri_tag"
to the end of each rule. Note how 90 is the prefix to the file name. This makes sure these rules are run after the others in the default70-snap.core.rules
, preventing them from being overwritten. Feel free to explore70-snap.core.rules
to see numerous examples of udev rules.
Reload the udev rules and trigger them.
List the devices that have been tagged, which Akri will discover. Akri will only discover devices with device nodes (devices within the
/dev
directory). These device node paths will be mounted into broker Pods so the brokers can utilize the devices.Explore the attributes of each device in order to decide how to refine your udev rule.
Modify the rule as needed, being sure to reload and trigger the rules each time.
Remove the tag from the devices -- note how
+=
turns to-=
-- and reload and trigger the udev rules. Alternatively, if you are trying to discover devices with fields that Akri does not yet support, such asATTRS
, you could leave the tag and add it to the rule in your Configuration withTAG=="akri_tag"
.Confirm that the tag has been removed and no devices are listed.
Create an Akri Configuration with your udev rule!
Installing Akri with a udev Configuration and Discovery Handler
Leveraging the above settings, Akri can be installed with the udev Discovery Handler and a udev Configuration with our udev rule specified.
Note: See the cluster setup steps for information on how to set the crictl configuration variable
AKRI_HELM_CRICTL_CONFIGURATION
The following installation examples have been given to show how to the udev Configuration can be tailored to you cluster:
Modifying the udev rule
Specifying a broker pod image
For more advanced Configuration changes that are not aided by our Helm chart, we suggest creating a Configuration file using Helm and then manually modifying it. To do this, see our documentation on Customizing an Akri Installation
Modifying the udev rule
The udev Discovery Handler will find all devices that are described by ANY of the udev rules. For example, to discover devices made by either Great Vendor or Awesome Vendor, you could add a second udev rule.
Akri will now discover these devices and advertize them to the cluster as resources. Each discovered device is represented as an Akri Instance. To list them, run kubectl get akrii
. Note akrii
is a short name for Akri Instance. All the instances will be named in the format <configuration-name>-<hash>
. You could change the name of the Configuration and resultant Instances to be sound-device
by adding --set udev.configuration.name=sound-devices
to your installation command. Now, you can schedule pods that request these Instances as resources, as explained in the requesting akri resources document.
Specifying a broker pod image
Instead of manually deploying Pods to resources advertized by Akri, you can add a broker image to the udev Configuration. Then, a broker will automatically be deployed to each discovered device. The controller will inject the information the broker needs to find its device as environment variables. Namely, it injects an environment variable named UDEV_DEVPATH_{INSTANCE_HASH}
which contains the device's sysfs path (i.e. /devices/pci0000:00/0000:00:1f.3/sound/card0/input4
). Additionally, if the devnode path is found, it also injects an environment variable named UDEV_DEVNODE_{INSTANCE_HASH}
which contains the devnode path for that device (i.e. /dev/snd/pcmC0D0c
). The broker can grab these environment variables and proceed to interact with the device. To add a broker to the udev configuration, set the udev.configuration.brokerPod.image.repository
value to point to your image. As an example, the installation below will deploy an empty nginx pod for each instance. Instead, you can point to your image, say ghcr.io/<USERNAME>/sound-broker
.
Note: set
udev.configuration.brokerPod.image.tag
to specify an image tag (defaults tolatest
).
Akri will automatically create a broker for each discovered device. It will also create a service for each broker and one for all brokers of the Configuration that applications can point to. See the Customizing Akri Installation to learn how to modify the broker pod spec and service specs in the Configuration.
Setting the broker Pod security context
By default in the generic udev Configuration, the udev broker is run in privileged security context. This container security context can be customized via Helm. For example, to instead run all processes in the Pod with user ID 1000 and group 1000, do the following:
Modifying a Configuration
Akri has provided further documentation on modifying the broker PodSpec, instanceServiceSpec, or configurationServiceSpec More information about how to modify an installed Configuration, add additional Configurations to a cluster, or delete a Configuration can be found in the Customizing an Akri Installation document.
Grouping related device nodes
Akri currently provides a way to group device nodes under the topmost matching node, this allows to handle a complex device with multiple device nodes as one Instance.
For example with the following udev device tree and the rule ENV{ID_SERIAL}=="Great Vendor Complex Camera"
:
This would result in a single instance grouping video0
, video1
and pcmC0D0c
.
All the device nodes will get mounted into the broker pod and will be listed in the environment variables with UDEV_DEVNODE
prefix
This behavior can be enabled by setting the udev.configuration.discoveryDetails.groupRecursive
to true
.
Implementation details
The udev implementation can be understood by looking at several things:
UdevDiscoveryDetails defines the required properties
UdevDiscoveryHandler defines udev discovery
samples/brokers/udev-video-broker defines the udev broker
udev_rule_grammar.pest defines the grammar for parsing udev rules and enumerate which fields are supported (such as
ATTR
andTAG
), which are yet to be supported (ATTRS
andTAGS
), and which fields will never be supported, mainly due to be assignment rather than matching fields (such asACTION
andGOTO
).
Last updated