1 Overview

Each Edge Application is defined by a specific set of information within the symmedia Hub. The containers to be deployed, along with the required metadata such as descriptions to be displayed in the portal UI, required configurations, etc., are defined in a single JSON document called the Application Definition. Its format is based on the Deployment Manifest (the specification can be found here: https://docs.microsoft.com/en-us/azure/iot-edge/module-composition?view=iotedge-2020-11) and extends it with additional data required by the Hub.

The Hub aggregates all installed applications for an Edge Device, creates resources and transforms them into an IoT Hub compliant Deployment Manifest, which is then deployed.

For documentation and illustration purposes, each section also describes how settings in the application definition affect the generated deployment manifest. If you have deployed IoT Edge modules manually using Visual Studio Code, you will recognize the format.

This document also describes the configuration settings and actions performed for each resource type that can be generated.

2 Example 

This section provides a quick overview of a valid Application Definition example. The sections are described in detail below. 

3 Application Definition Sections - General

3.1 id & version

The id field is a unique identifier for the app. Together with the version field it describes a unique application definition. Once an application has been published in a specific version, it cannot be changed afterwards. To correct mistakes or update the container versions, a new application version should be published. Once a new version is published, the new version will automatically be set as the current version, and the Operators will see an option to upgrade the installed Apps to the new version.

The id should have a prefix to prevent name-clashes.

The version has to follow semantic versioning and match this regex:

An visualization with examples can be found here: https://regex101.com/r/vkijKf/1/

3.2 displayName, shortDescription & longDescription

• displayName: Title of the Application

• shortDescription: Text in the Application card in the Applications overview

• longDescription: Text in the detail view of an Application

The fields each consist of multiple fields, one for each language this field has been translated for, the lowercase ISO-639-1 code of the language being the key, the translated text being the value. This field is supposed to originate in the app service for each app, the deployment service is not using this field, it is meant for web GUIs to offer the display name in multiple languages. At the moment, only the “en” text is displayed. The translation feature is coming soon in the UI.

All of these fields have to define at least an english (en) translation.

3.3 Media / teaserImage

Here an URL to an image is defined which will be displayed as Image for the Application.

3.4 storageLimit

Maximums storage that the whole App is allowed to consume. That is the sum of all container storages and volumes. This value will be shown in the UI. More details are explained in Chapter 6. 

3.5 userSettings

This section is an array containing the settings a user is offered during app installation.

The values the user has offered can be used as placeholders in different locations in the AppDefinition. The name of the setting is the name that needs to be used fo the placeholder. To provide a value to a container it needs to be added as environment parameter (see 4.7). It is also possible to use the value for the port definition as described in the example.

The settings can be defined as mandatory and they can have validations and default values. Each entry has these fields:

3.6 cloudResources: Webhooks to external systems

In the cloudResources section webhooks can be defined that are excecuted during the installation or uninstallation of an Application.

3.6.1 Install Webhook

Calls a specified external webhook in the symmedia backend before deploying an Application on the Edge Device and waits up to 60 seconds for it to return. The webhook can return a list of new token replacement values containing 0-n entries. Those values can then be used in ${TOKEN} replacements further on. Returned entries must not collide with any existing or built-in token key, or deployment fails.

Webhooks are commonly used to prepare data mapping tables in an app’s cloud backend, or to create resources like blob containers on the fly and return the SAS-Token values back.

Under ideal circumstances, install webhooks are only called once until the user either removes the app and installs it again, the provided settings change, or the application version changes. Webhooks should be prepared to be called more than once with the same data.

The cloud resource type is "Webhook". The following fields need to be present:

The provided URL will be called up to 3 times with 2 second waiting interval. If the execution fails, or does not complete in 60 seconds, the deployment / installation is marked as failed.

An example:

The webhook is called as HTTP POST with the following content:

Note: Webhooks need to verify that the “event” field is what they expect. The event field can be “Install” or “Uninstall”, but might also take on different values in the future. In case a value is not understood, the webhook must return a success but ignore the event.

The webhook needs to return content following this schema:

Or in case of an error:

3.6.2 Uninstall Webhook

When an application is removed, the according webhook is called again, with an event of “Uninstall”. The uninstall webhook is usually only called once per uninstall, but webhooks should be prepared to be called more than once.

Please note that the “providedParameters” field is not provided during an uninstall event.

NOTE: This can be changed on request.

3.7 containers

This section is an array defining the containers that are deployed as part of this application. The details of each container entry are described in the next section.

4 Container definitions

Each container entry corresponds to an edge module that will be deployed. It is recommended to use prefixes to avoid name-clashes.

4.1 name

This is the name of the container, and also the name of the edge module as referenced in the edge hub routes. It will also be identical to the host name inside the internal docker network, and in case the edge module needs to be contacted via network from another edge module, this name can be used as hostname.

In the generated IoT Hub manifest, this setting affects the container name in the $edgeAgent section that declares all docker container settings, and the top-level section defining the desired.properties.

4.2 image

The container image including the tag to be used. The container image must be uploaded to the symmedia Registry.

As per recommendation from Microsoft: container tags / versions should always be specified as detailed as possible. I.e. use 1.4.1 instead of "latest" or ":1.4", to ensure tight control over the container version, and prevent behavioral discrepancies between edge modules with identical configuration, that have been installed at different times.

In the generated IoT Hub manifest, this setting is reflected in the "Module > settings > image" field.

4.3 startupOrder

An optional section containing a number that defines the startup priority of each container. The possible values range from 1 (top priority) to 989 (low priority). If you use a zero or a negative value, it is considered the lowest priority. Values above 989 are automatically set to 989. Keep in mind that system and infrastructure modules are not defined here and are always started before these containers.

If multiple containers have the same startupOrder value, those values remain as duplicates. The system will "randomly" start containers with duplicate values before moving on to containers with higher values in the startup order.

4.4 commandLine

An optional section containing an array of strings. Each string is passed as an additional command line parameter. 

After the variables have been replaced, and potential configuration folding has been applied (if the module is used by multiple applications), this section is copied directly to the target in the IoT Hub manifest at $edgeAgent > properties.desired > modules > MODULE > settings > createOptions > Cmd

4.5 mappedFiles

An array of docker file / directory bindings that will be added. Each entry can have the following fields:

For a difference between file (bind) mounts and volume mounts please refer to:
https://docs.docker.com/storage/volumes/
https://docs.docker.com/storage/bind-mounts/

When specifying paths you can assume all containers to be Linux containers. All paths are absolute and start with "/", except for volume mounts.

The resulting IoT deployment manifest entry looks like this: 

4.6 ports

An array of docker port maps that will be added. Each entry can have the following fields:

For details about Docker port mappings, please see: https://docs.docker.com/engine/reference/run/#expose-incoming-ports . The deploy service does not perform validation of the port configuration, this is left to the Docker host.

The resulting IoT deployment manifest entry looks like this: 

4.7 environment

This section is an object whose keys are defining the name of the environment variables. The values can be simple values or placeholders (i.e. built-in placeholders or user settings) that will be replaced by their values during deployment.

The resulting IoT deployment manifest entry looks like this: 

4.8 propertiesDesired

This section defines the starting desired properties of an edge module that is deployed. Desired properties are configuration entries that can be dynamically changed via IoT Hub calls by cloud services, and are an integral part of IoT edge to reconfigure services without redeployments.

Some modules are configured via environment variables, some have complex configurations via these settings. After the variables have been replaced, and potential configuration folding has been applied (if the module is used by multiple applications), this section is copied directly to the target in the IoT Hub manifest at MODULE > properties.desired.

4.9 labels

Labels will be added to the createOptions parameter in the deployment manifest which enables you to configure the module containers at runtime. For more information see https://learn.microsoft.com/en-us/azure/iot-edge/how-to-use-create-options?view=iotedge-1.4

4.10 resourceSettings

Application resource management is under development and not yet fully enabled. The memory limit is in place on the Edge Device, but the CPU limit is not yet active. The App developer will need to set the values for this feature to be enabled soon.

Every container will be limited by CPU, memory and storage usage (see 3.4).

• cpus: Specify how much of the available CPU resources a container can use. For instance, if the host machine has two CPUs and you set cpus: "1.5", the container is limited at most one and a half of the CPUs. (Like docker flag --cpus). The CPU cores are not reserved which means that this value is not a guarantee for the container.

• memory: The maximum amount of memory the container can use. If you set this option, the minimum allowed value is 6m (6 megabytes). That is, you must set the value to at least 6 megabytes. It takes a positive integer, followed by a suffix of b, k, m, g, to indicate bytes, kilobytes, megabytes, or gigabytes. (Like docker flag --memory). The memory is reserved and guaranteed to the container.

Example:

"resourceSettings": {
	"cpus": "1.5",
	"memory": "15m"
}

5 Variable Replacement

Most time a string value is expected in the Application Definition variables (placeholders) can be used to customize the deployment for each edge device. Variables are usually defined as ${NAME}, with ${$} defining a single $ sign (as escape mechanism). The variable names and values are provided by:

• Built-in system variables

• The parameters described in userSettings and provided during app installation

Variable replacement can not be used for:

• ApplicationID & Version

• Translations (DisplayNames, ShortDescription & LongDescription)

• Media

• isPublic-Flag

• Resource Limitations (StorageLimit & CPU and Memory for the Containers)

• Within the PortDefinition of a Container: Variable replacement can only be used for HostIp, HostPort and ContainerPort.

5.1 Built-in Variables

5.1 Replacement Sequence

The following sequence is performed during the variable replacement stage. Each step can use any variable values made available in the earlier step, where applicable:

• The system built-in variables are determined

• The userSettings variables are determined. 

• The webhooks are called in the order listed.

• The containers' configurations are processed in the order listed.

6 Application Management Resource Limits

6.1 Introduction

We are introducing a new resource limiting feature for Edge Device applications on the symmedia Hub for the following reasons:

• We want to prevent the Edge Device from failing due to resource usage

  • The SSD can be fully utilized by applications that store a lot of data.

  • An application with an intensive workload or in an error state could…

    • use 100% of the CPU and make the Edge Device unresponsive.

    • use all of the memory and cause out-of-memory issues

  • Too many applications could be deployed on an Edge Device in total.

• App developers should be able to reserve sufficient resources for their apps to improve reliability.

The resource management feature is in the development phase and the new fields are in place, ready to be filled. It is necessary that all apps use the resource limitation feature, because a single "unlimited" app could still cause severe problems on an Edge Device and would render the whole mechanism useless. That´s why we’re now starting a transition period, during where all app developers will implement the resource limits in their app definitions, so that we´ll soon be able to enable the mechanism to improve reliability. The memory limit is the only value which is applied on the Edge Device currently.

6.2 Concept

From a high-level perspective, our approach can be described by the following four points:.

1. The app developer creates an application definition and defines the required resources in it.

2. The app is registered and validated in the Hub.

3. When a user installs an app, the Hub checks the required resources of the new app and all other installed apps and calculates, whether the app fits on the Edge Device considering the available resources of the specific Edge Device.

4. The apps are installed and run with resource constraints.

Technically, we use the Docker´s built-in resource constraints to limit the memory and number of CPU cores (see https://docs.docker.com/engine/containers/resource_constraints/). The application definition defines an amount of CPU cores and memory per container. These values have been aggregated to define the total resource consumption of an application.

Limiting memory resources is a bit more complex: For each application, a specific ProjectId is generated by the Hub. On the Edge Device, each ProjectId is assigned a quota based on the storage request in the application definition. The quota is then applied to all volumes and in-container storage. If the quota is exceeded, the App (all relevant containers) is stopped to ensure uninterrupted operation of the Edge Device. Therefore, it is recommended to use volumes with sizes and not to employ container-internal storage. If a volume becomes full, the entire app is not affected.

6.3 Set required Resources of an App in the Application Definition

The following has been added to the application definition schema regarding resource limits:

Binary units are used for all resource settings. This means that 1 megabyte (MiB) is equal to 1024 kilobytes (KiB). Most operating systems (including Linux, Windows & Mac) use these units by default.

6.3.1 Storage limit

This value defines the maximum and guaranteed amount of storage that the application is allowed to store on the SSD. The value is a positive integer, followed by a suffix of b, k, m, g, to indicate bytes, kibibytes (KiB), mebibytes (MiB), or gibibytes (GiB).

6.3.2 CPU and Memory

• CPU: Specify how much of the available CPU resources a container can use. For example, if the host machine has two CPUs and cpus: "1.5" is set, the container is guaranteed a maximum of one and a half of the CPUs. (Like the Docker flag --cpus)

• Memory: The maximum amount of memory that the container can use. The minimum value allowed is 6m (6 mebibytes). It takes a positive integer, followed by a suffix of b, k, m, g, to bytes, kibibytes (KiB), mebibytes (MiB), or gibibytes (GiB) (Like docker flag --memory).