Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException

[svanbodegraven]

IoT Edge is throwing a Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException when deploying the Azure Blob Storage module from the Azure Marketplace, when following https://learn.microsoft.com/en-us/azure/iot-edge/how-to-deploy-blob?view=iotedge-1.4.

Expected Behavior

The module should deploy.

Current Behavior

Here's the stack trace:

<6> 2024-02-20 03:45:59.349 +00:00 [INF] - Executing command: "Command Group: (\n [Command Group: (\n [Prepare module AzureStorage]\n [Create module AzureStorage]\n)]\n [Start module AzureStorage]\n)"
<6> 2024-02-20 03:45:59.349 +00:00 [INF] - Executing command: "Command Group: (\n [Prepare module AzureStorage]\n [Create module AzureStorage]\n)"
<6> 2024-02-20 03:46:06.852 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-02-20 03:46:06.852 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-02-20 03:46:06.852 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\bedgeAgent\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2024-02-20 03:46:06.855 +00:00 [INF] - Initiating streaming logs for edgeAgent
<6> 2024-02-20 03:46:06.886 +00:00 [INF] - Received 4992 bytes of logs for edgeAgent
<6> 2024-02-20 03:46:06.886 +00:00 [INF] - Successfully handled request GetModuleLogs
<3> 2024-02-20 03:46:16.613 +00:00 [ERR] - Executing command for operation ["create"] failed.
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling Create module AzureStorage: runtime operation error: create module "AzureStorage", StatusCode:500, at: at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 232
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func1 func, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 171 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.CreateModuleAsync(ModuleSpec moduleSpec) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 102 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.<>c__DisplayClass26_0.<<Throttle>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 145 --- End of stack trace from previous location --- at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.Throttle[T](Func1 identityOperation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 164
at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 69
<3> 2024-02-20 03:46:16.613 +00:00 [ERR] - Executing command for operation ["Command Group: (\n [Command Group: (\n [Prepare module AzureStorage]\n [Create module AzureStorage]\n)]\n [Start module AzureStorage]\n)"] failed.
Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling Create module AzureStorage: runtime operation error: create module "AzureStorage", StatusCode:500, at: at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 232
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func1 func, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 171 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.CreateModuleAsync(ModuleSpec moduleSpec) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 102 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.<>c__DisplayClass26_0.<<Throttle>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 145 --- End of stack trace from previous location --- at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.Throttle[T](Func1 identityOperation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 164
at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 69
at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 69
<3> 2024-02-20 03:46:16.613 +00:00 [ERR] - Step failed in deployment 12, continuing execution. Failure when running command Command Group: (
[Command Group: (
[Prepare module AzureStorage]
[Create module AzureStorage]
)]
[Start module AzureStorage]
). Will retry in -01s.
<6> 2024-02-20 03:46:16.613 +00:00 [INF] - Plan execution ended for deployment 12
<3> 2024-02-20 03:46:16.613 +00:00 [ERR] - Edge agent plan execution failed.
System.AggregateException: One or more errors occurred. (Error calling Create module AzureStorage: runtime operation error: create module "AzureStorage")
---> Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling Create module AzureStorage: runtime operation error: create module "AzureStorage", StatusCode:500, at: at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 232
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func1 func, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 171 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.CreateModuleAsync(ModuleSpec moduleSpec) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 102 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.<>c__DisplayClass26_0.<<Throttle>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 145 --- End of stack trace from previous location --- at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.Throttle[T](Func1 identityOperation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 164
at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 69
at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35
at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 69
at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunner.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunner/OrdererdRetryPlanRunner.cs:line 91
--- End of inner exception stack trace ---
at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunner.OrderedRetryPlanRunner.<>c.b__7_0(List1 f) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunner/OrdererdRetryPlanRunner.cs:line 129 at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunner.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 208 <4> 2024-02-20 03:46:16.618 +00:00 [WRN] - Reconcile failed because of the an exception System.AggregateException: One or more errors occurred. (Error calling Create module AzureStorage: runtime operation error: create module "AzureStorage") ---> Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling Create module AzureStorage: runtime operation error: create module "AzureStorage", StatusCode:500, at: at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 232 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func1 func, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 171
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagementHttpClient.CreateModuleAsync(ModuleSpec moduleSpec) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 102
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.<>c__DisplayClass26_0.<b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 145
--- End of stack trace from previous location ---
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.Throttle[T](Func1 identityOperation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 164 at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35 at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 69 at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 35 at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 69 at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunner.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunner/OrdererdRetryPlanRunner.cs:line 91 --- End of inner exception stack trace --- at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunner.OrderedRetryPlanRunner.<>c.<ExecuteAsync>b__7_0(List1 f) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunner/OrdererdRetryPlanRunner.cs:line 129
at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunner.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token)
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 208
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 208
<6> 2024-02-20 03:46:22.494 +00:00 [INF] - Plan execution started for deployment 12
<6> 2024-02-20 03:46:22.494 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-02-20 03:46:22.494 +00:00 [INF] - Plan execution ended for deployment 12
<6> 2024-02-20 03:46:28.334 +00:00 [INF] - Plan execution started for deployment 12
<6> 2024-02-20 03:46:28.334 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-02-20 03:46:28.334 +00:00 [INF] - Plan execution ended for deployment 12
<6> 2024-02-20 03:46:32.314 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-02-20 03:46:32.314 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-02-20 03:46:32.315 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\bedgeAgent\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}

Steps to Reproduce

Follow the steps in the article. Here is the deployment with keys redacted.

{
"modulesContent": {
"$edgeAgent": {
"properties.desired": {
"schemaVersion": "1.1",
"runtime": {
"type": "docker",
"settings": {}
},
"systemModules": {
"edgeAgent": {
"settings": {
"image": "mcr.microsoft.com/azureiotedge-agent:1.4"
},
"type": "docker"
},
"edgeHub": {
"restartPolicy": "always",
"settings": {
"image": "mcr.microsoft.com/azureiotedge-hub:1.4",
"createOptions": "{"HostConfig":{"PortBindings":{"443/tcp":[{"HostPort":"443"}],"5671/tcp":[{"HostPort":"5671"}],"8883/tcp":[{"HostPort":"8883"}]}}}"
},
"status": "running",
"type": "docker"
}
},
"modules": {
"SimulatedTemperatureSensor": {
"env": {},
"restartPolicy": "always",
"settings": {
"image": "mcr.microsoft.com/azureiotedge-simulated-temperature-sensor"
},
"status": "running",
"type": "docker"
},
"AzureStorage": {
"env": {
"LOCAL_STORAGE_ACCOUNT_NAME": {
"value": "lucifer"
},
"LOCAL_STORAGE_ACCOUNT_KEY": {
"value": "REDACTED"
}
},
"restartPolicy": "always",
"settings": {
"image": "mcr.microsoft.com/azure-blob-storage:latest",
"createOptions": "{"HostConfig":{"Binds":["C:\Data:C:\BlobRoot"],"PortBindings":{"11002/tcp":[{"HostPort":"11002"}]}}}"
},
"status": "running",
"type": "docker"
}
}
}
},
"$edgeHub": {
"properties.desired": {
"schemaVersion": "1.1",
"storeAndForwardConfiguration": {
"timeToLiveSecs": 7200
},
"routes": {
"SimulatedTemperatureSensorToIoTHub": {
"route": "FROM /messages/modules/SimulatedTemperatureSensor/* INTO $upstream"
}
}
}
},
"SimulatedTemperatureSensor": {
"properties.desired": {
"SendData": true,
"SendInterval": 5
}
},
"AzureStorage": {
"properties.desired": {
"deviceToCloudUploadProperties": {
"uploadOn": true,
"uploadOrder": "OldestFirst",
"cloudStorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=edgeremotestorage;AccountKey=REDACTED;EndpointSuffix=core.windows.net",
"storageContainersForUpload": {
"data": {
"target": "uav-bom"
}
},
"deleteAfterUpload": true
}
}
}
}
}

Context (Environment)

Output of iotedge check

Is all green except for the optional DNS bit.

Device Information

• Windows Pro, 22000+
• x64
• As per your installer, Linux running on Hyper-V

[nyanzebra]

Hi @svanbodegraven it looks like edgeAgent is not able to even deploy the AzureStorage module. I verified locally that I am able to get AzureStorage module up and running. Just to confirm, do other modules also come up fine and is it just AzureStorage with issues?

[konichi3]

@svanbodegraven Do you still the issue and repro the issue consistently?

[konichi3]

Closing due to inactivity.