• Installing Prometheus
• Enabling Prometheus with Akri
• Visualizing metrics with Grafana
• Akri's currently exposed metrics
• Exposing metrics from an Akri Broker Pod

1. 1.

   Get the kube-prometheus stack Helm repo.

   1

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

   2

   helm repo add stable https://charts.helm.sh/stable

   3

   helm repo update

   Copied!
2. 2.

   Install the chart, specifying what namespace you want Prometheus to run in. It does not have to be the same namespace in which you are running Akri. For example, it may be in a namespace called monitoring as in the command below. By default, Prometheus only discovers PodMonitors within its namespace. This should be disabled by settingpodMonitorSelectorNilUsesHelmValues to false so that Akri's custom PodMonitors can be discovered. Additionally, the Grafana service can be exposed to the host by making it a NodePort service. It may take a minute or so to deploy all the components.

   1

   helm install prometheus prometheus-community/kube-prometheus-stack \

   2

   --set grafana.service.type=NodePort \

   3

   --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \

   4

   --namespace monitoring

   Copied!

   The Prometheus dashboard can also be exposed to the host by adding --set prometheus.service.type=NodePort. If intending to expose metrics from a Broker Pod via a ServiceMonitor also set serviceMonitorSelectorNilUsesHelmValues to false.

Note: See the cluster setup steps for information on how to set the crictl configuration variable AKRI_HELM_CRICTL_CONFIGURATION

1. 1.

   Determine the port that the Grafana Service is running on, specifying the namespace if necessary, and save it for the next step.

   1

   kubectl get service/prometheus-grafana --namespace=monitoring --output=jsonpath='{.spec.ports[?(@.name=="service")].nodePort}' && echo

   Copied!
2. 2.

   SSH port forwarding can be used to access Grafana. Open a new terminal, and enter your ssh command to access the machine running Akri and Prometheus followed by the port forwarding request. The following command will use port 50000 on the host. Feel free to change it if it is not available. Be sure to replace <Grafana Service port> with the port number outputted in the previous step.

   1

   ssh [email protected]<IP address> -L 50000:localhost:<Grafana Service port>

   Copied!
3. 3.

   Navigate to http://localhost:50000/ and enter Grafana's default username admin and password prom-operator.

   Once logged in, the username and password can be changed in account settings. Now,

   you can create a Dashboard to display the Akri metrics.

1. 1.

   Akri can be installed with the udev Configuration, filtering for only usb video cameras and specifying a

   Configuration name of akri-udev-video, by running:

   1

   helm repo add akri-helm-charts https://project-akri.github.io/akri/

   2

   helm install akri akri-helm-charts/akri \

   3

   $AKRI_HELM_CRICTL_CONFIGURATION \

   4

   --set udev.enabled=true \

   5

   --set udev.name=akri-udev-video \

   6

   --set udev.udevRules[0]='KERNEL=="video[0-9]*"' \

   7

   --set udev.brokerPod.image.repository="ghcr.io/project-akri/akri/udev-video-broker"

   Copied!

   Note: This instruction assumes you are using vanilla Kubernetes. Be sure to reference the user guide to determine whether the distribution you are using requires crictl path configuration.

   Note: To expose the Agent and Controller's Prometheus metrics, add --set prometheus.enabled=true.

   Note: If Prometheus is running in a different namespace as Akri and was not enabled to discover ServiceMonitors in other namespaces when installed, upgrade your Prometheus Helm installation to set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues to false.

   1

   helm upgrade prometheus prometheus-community/kube-prometheus-stack \

   2

   $AKRI_HELM_CRICTL_CONFIGURATION \

   3

   --set grafana.service.type=NodePort \

   4

   --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \

   5

   --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \

   6

   --namespace monitoring

   Copied!

2. 2.

   Then, create a Service for exposing these metrics, targeting all Pods labeled with the Configuration name akri-udev-video.

   1

   apiVersion: v1

   2

   kind: Service

   3

   metadata:

   4

   name: akri-udev-video-broker-metrics

   5

   labels:

   6

   app: akri-udev-video-broker-metrics

   7

   spec:

   8

   selector:

   9

   akri.sh/configuration: akri-udev-video

   10

   ports:

   11

   - name: metrics

   12

   port: 8080

   13

   type: ClusterIP

   Copied!

   The metrics also could have been exposed by adding the metrics port to the Configuration level service in the udev Configuration.

3. 3.

   Apply the Service to your cluster.

   1

   kubectl apply -f akri-udev-video-broker-metrics-service.yaml

   Copied!
4. 4.

   Create the associated ServiceMonitor. Note how the selector matches the app name of the Service.

   1

   apiVersion: monitoring.coreos.com/v1

   2

   kind: ServiceMonitor

   3

   metadata:

   4

   name: akri-udev-video-broker-metrics

   5

   labels:

   6

   release: prometheus

   7

   spec:

   8

   selector:

   9

   matchLabels:

   10

   app: akri-udev-video-broker-metrics

   11

   endpoints:

   12

   - port: metrics

   Copied!
5. 5.

   Apply the ServiceMonitor to your cluster.

   1

   kubectl apply -f akri-udev-video-broker-metrics-service-monitor.yaml

   Copied!
6. 6.

   The frame count metric reports the number of video frames that have been requested by some application. It will remain at zero unless an application is deployed that utilizes the video Brokers. Deploy the Akri sample streaming application by running the following:

   1

   kubectl apply -f https://raw.githubusercontent.com/project-akri/akri/main/deployment/samples/akri-video-streaming-app.yaml

   2

   watch kubectl get pods

   Copied!