Support Unix domain sockets in IoT Edge on Windows (#518)

Changes to Edge Agent:
- When Edge Agent builds the Docker "createOptions" string to send to iotedged, the bind-mount specification on Windows needs to point to the _parent directory_ for each socket file, not to the files themselves. On Windows, Docker can't bind-mount a Unix domain socket file.

Changes to edgelet:
- Change the default management and workload URIs (YAML config) for Windows to specify Unix domain sockets instead of HTTP endpoints.
- Deserialize the management and workload URIs into file paths on Windows by parsing them with the `file://` scheme instead of `unix://`. This is because the `url` crate does lots of special handling of Windows paths for the `file://` scheme, but doesn't do it for "non-special" schemes like `unix://` (per the URL spec).
- Depend on `mio-uds-windows`, `tokio-uds-windows`, and `hyperlocal-windows` crates on Windows. These are adaptations of `mio-uds`, `tokio-uds`, and `hyperlocal` respectively.
- Where appropriate, remove the `#[cfg(unix)]` attribute from Unix domain socket-specific features that expected a Unix-only implementation, and add it in places where we still need to differentiate on platform. Notable examples:
  - In `edgelet-http::unix::listener(path)`, only use umask to reset socket file permissions on Unix platforms.
  - On Unix platforms check for existence of the Unix domain socket file with `file.exists()`. On Windows, use `fs::symlink_metadata()` instead because it works on reparse points (Unix domain sockets in Windows are reparse points).
- Implement `UnixStreamExt::pid(&self)` for Windows
- A test that previously used `UnixStream::pair()` to do its setup doesn't work on Windows because Winsock2 doesn't implement BSD's `pair` function. So create an ugly workaround for the test on Windows.
- Use TempDir more pervasively in tests that create a socket file. In a Unix-only world we got away with simpler solutions that don't work if you can't rely on the existence of `/tmp`.
- Move the test helper routine `run_uds_server` out of a linux-specific source file and into the parent `mod.rs`.
- When the Edge Agent container is created in Windows, bind-mount the parent directory for each socket file rather than the file itself, just as Edge Agent does for other modules.

In the Windows installer script:
- Create the parent directory for each socket file, and give Modify rights to a well-known group (`NT AUTHORITY\Authenticated Users`) that will exist in any container so that non-privileged modules can access it. Since we give the rights to the parent folder, we don't need to recreate the permissions every time iotedged restarts, like we do for Unix platforms. (_Note: this part of the script is disabled for now, until we have everything in place to fully support UDS in IoT Edge on Windows, specifically RS5-based module images and the ability to use process-isolated containers on non-Server Windows._)

edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet.Docker/CombinedEdgeletConfigProvider.cs

@@ -4,6 +4,8 @@ namespace Microsoft.Azure.Devices.Edge.Agent.Edgelet.Docker
 {
     using System;
     using System.Collections.Generic;
+    using System.IO;
+    using System.Runtime.InteropServices;
     using global::Docker.DotNet.Models;
     using Microsoft.Azure.Devices.Edge.Agent.Core;
     using Microsoft.Azure.Devices.Edge.Agent.Docker;
@@ -78,7 +80,7 @@ void MountSockets(IModule module, CreateContainerParameters createOptions)
                 SetMountOptions(createOptions, workloadUri);
             }
 
-            // If Management URI is Unix domain socket, and the module is the EdgeAgent, then mount it ino the container.
+            // If Management URI is Unix domain socket, and the module is the EdgeAgent, then mount it into the container.
             var managementUri = new Uri(this.configSource.Configuration.GetValue<string>(Constants.EdgeletManagementUriVariableName));
             if (managementUri.Scheme == "unix"
                 && module.Name.Equals(Constants.EdgeAgentModuleName, StringComparison.OrdinalIgnoreCase))
@@ -91,10 +93,20 @@ static void SetMountOptions(CreateContainerParameters createOptions, Uri uri)
         {
             HostConfig hostConfig = createOptions.HostConfig ?? new HostConfig();
             IList<string> binds = hostConfig.Binds ?? new List<string>();
-            binds.Add($"{uri.AbsolutePath}:{uri.AbsolutePath}");
+            string path = BindPath(uri);
+            binds.Add($"{path}:{path}");
 
             hostConfig.Binds = binds;
             createOptions.HostConfig = hostConfig;
         }
+
+        static String BindPath(Uri uri)
+        {
+            // On Windows we need to bind to the parent folder. We can't bind
+            // directly to the socket file.
+            return RuntimeInformation.IsOSPlatform(OSPlatform.Windows)
+                ? Path.GetDirectoryName(uri.LocalPath)
+                : uri.AbsolutePath;
+        }
     }
 }

edge-agent/test/Microsoft.Azure.Devices.Edge.Agent.Edgelet.Docker.Test/CombinedEdgeletConfigProviderTest.cs

@@ -2,6 +2,7 @@ namespace Microsoft.Azure.Devices.Edge.Agent.Edgelet.Docker.Test
 {
     using System;
     using System.Collections.Generic;
+    using System.Runtime.InteropServices;
     using global::Docker.DotNet.Models;
     using Microsoft.Azure.Devices.Edge.Agent.Core;
     using Microsoft.Azure.Devices.Edge.Agent.Docker;
@@ -31,12 +32,21 @@ public void TestVolMount()
             module.SetupGet(m => m.Config).Returns(new DockerConfig("nginx:latest"));
             module.SetupGet(m => m.Name).Returns(Constants.EdgeAgentModuleName);
 
+            var unixUris = new Dictionary<string, string>
+            {
+                {Constants.EdgeletWorkloadUriVariableName, "unix:///path/to/workload.sock" },
+                {Constants.EdgeletManagementUriVariableName, "unix:///path/to/mgmt.sock" }
+            };
+
+            var windowsUris = new Dictionary<string, string>
+            {
+                {Constants.EdgeletWorkloadUriVariableName, "unix:///C:/path/to/workload/sock" },
+                {Constants.EdgeletManagementUriVariableName, "unix:///C:/path/to/mgmt/sock" }
+            };
+
             IConfigurationRoot configRoot = new ConfigurationBuilder().AddInMemoryCollection(
-                new Dictionary<string, string>
-                {
-                    {Constants.EdgeletWorkloadUriVariableName, "unix:///var/run/iotedgedworkload.sock" },
-                    {Constants.EdgeletManagementUriVariableName, "unix:///var/run/iotedgedmgmt.sock" }
-                }).Build();
+                 RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ? windowsUris : unixUris
+            ).Build();
             var configSource = Mock.Of<IConfigSource>(s => s.Configuration == configRoot);
             ICombinedConfigProvider<CombinedDockerConfig> provider = new CombinedEdgeletConfigProvider(new[] { new AuthConfig() }, configSource);
 
@@ -48,8 +58,14 @@ public void TestVolMount()
             Assert.NotNull(config.CreateOptions.HostConfig);
             Assert.NotNull(config.CreateOptions.HostConfig.Binds);
             Assert.Equal(2, config.CreateOptions.HostConfig.Binds.Count);
-            Assert.Equal("/var/run/iotedgedworkload.sock:/var/run/iotedgedworkload.sock", config.CreateOptions.HostConfig.Binds[0]);
-            Assert.Equal("/var/run/iotedgedmgmt.sock:/var/run/iotedgedmgmt.sock", config.CreateOptions.HostConfig.Binds[1]);
+            if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
+            {
+                Assert.Equal("C:\\path\\to\\workload:C:\\path\\to\\workload", config.CreateOptions.HostConfig.Binds[0]);
+                Assert.Equal("C:\\path\\to\\mgmt:C:\\path\\to\\mgmt", config.CreateOptions.HostConfig.Binds[1]);
+            } else {
+                Assert.Equal("/path/to/workload.sock:/path/to/workload.sock", config.CreateOptions.HostConfig.Binds[0]);
+                Assert.Equal("/path/to/mgmt.sock:/path/to/mgmt.sock", config.CreateOptions.HostConfig.Binds[1]);
+            }
         }
 
         [Fact]

edgelet/contrib/config/windows/config.yaml

@@ -102,12 +102,20 @@ hostname: "<ADD HOSTNAME HERE>"
 #
 # The following uri schemes are supported:
 #     http - connect over TCP
+#     unix - connect over Unix domain socket
+#
+# If the 'unix' scheme is selected, the daemon expects that the parent
+# directory of the specified socket file already exists, and that the Windows
+# group 'NT AUTHORITY\Authenticated Users' has been given 'Modify' rights on
+# the directory. For example, if the URI "unix:///C:/path/to/sock.file" is
+# specified, then the directory "C:\path\to" must exist with the correct
+# permissions.
 #
 ###############################################################################
 
 connect:
-  management_uri: "http://<GATEWAY_ADDRESS>:15580"
-  workload_uri: "http://<GATEWAY_ADDRESS>:15581"
+  management_uri: "unix:///C:/ProgramData/iotedge/mgmt/sock"
+  workload_uri: "unix:///C:/ProgramData/iotedge/workload/sock"
 
 ###############################################################################
 # Listen settings
@@ -120,12 +128,20 @@ connect:
 #
 # The following uri schemes are supported:
 #     http - listen over TCP
+#     unix - listen over Unix domain socket
+#
+# If the 'unix' scheme is selected, the daemon expects that the parent
+# directory of the specified socket file already exists, and that the Windows
+# group 'NT AUTHORITY\Authenticated Users' has been given 'Modify' rights on
+# the directory. For example, if the URI "unix:///C:/path/to/sock.file" is
+# specified, then the directory "C:\path\to" must exist with the correct
+# permissions.
 #
 ###############################################################################
 
 listen:
-  management_uri: "http://<GATEWAY_ADDRESS>:15580"
-  workload_uri: "http://<GATEWAY_ADDRESS>:15581"
+  management_uri: "unix:///C:/ProgramData/iotedge/mgmt/sock"
+  workload_uri: "unix:///C:/ProgramData/iotedge/workload/sock"
 
 ###############################################################################
 # Home Directory

edgelet/edgelet-docker/src/runtime.rs

@@ -3,6 +3,7 @@
 use std::collections::HashMap;
 use std::convert::From;
 use std::ops::Deref;
+use std::path::PathBuf;
 use std::time::Duration;
 
 use base64;
@@ -22,7 +23,7 @@ use edgelet_core::{
     LogOptions, Module, ModuleRegistry, ModuleRuntime, ModuleRuntimeState, ModuleSpec,
     SystemInfo as CoreSystemInfo,
 };
-use edgelet_http::UrlConnector;
+use edgelet_http::{UrlConnector, UrlExt};
 use edgelet_utils::log_failure;
 
 use error::{Error, ErrorKind, Result};
@@ -53,9 +54,12 @@ impl DockerModuleRuntime {
         let client = Client::builder().build(UrlConnector::new(docker_url)?);
 
         // extract base path - the bit that comes after the scheme
-        let base_path = get_base_path(docker_url);
+        let base_path = get_base_path(docker_url)?;
         let mut configuration = Configuration::new(client);
-        configuration.base_path = base_path.to_string();
+        configuration.base_path = base_path
+            .to_str()
+            .expect("URL points to a path that cannot be represented in UTF-8")
+            .to_string();
 
         let scheme = docker_url.scheme().to_string();
         configuration.uri_composer = Box::new(move |base_path, path| {
@@ -96,10 +100,10 @@ impl DockerModuleRuntime {
     }
 }
 
-fn get_base_path(url: &Url) -> &str {
+fn get_base_path(url: &Url) -> Result<PathBuf> {
     match url.scheme() {
-        "unix" => url.path(),
-        _ => url.as_str(),
+        "unix" => Ok(url.to_uds_file_path()?),
+        _ => Ok(url.as_str().into()),
     }
 }
 

edgelet/edgelet-http-mgmt/src/client/module.rs

@@ -1,14 +1,15 @@
 // Copyright (c) Microsoft. All rights reserved.
 
 use std::fmt;
+use std::path::PathBuf;
 use std::str::FromStr;
 use std::sync::Arc;
 use std::time::Duration;
 
 use edgelet_core::SystemInfo as CoreSystemInfo;
 use edgelet_core::*;
 use edgelet_docker::{self, DockerConfig};
-use edgelet_http::{UrlConnector, API_VERSION};
+use edgelet_http::{UrlConnector, UrlExt, API_VERSION};
 use futures::future::{self, FutureResult};
 use futures::prelude::*;
 use futures::stream;
@@ -29,9 +30,9 @@ impl ModuleClient {
     pub fn new(url: &Url) -> Result<Self, Error> {
         let client = Client::builder().build(UrlConnector::new(url)?);
 
-        let base_path = get_base_path(url);
+        let base_path = get_base_path(url)?;
         let mut configuration = Configuration::new(client);
-        configuration.base_path = base_path.to_string();
+        configuration.base_path = base_path.to_str().ok_or(ErrorKind::Utf8)?.to_string();
 
         let scheme = url.scheme().to_string();
         configuration.uri_composer = Box::new(move |base_path, path| {
@@ -45,10 +46,10 @@ impl ModuleClient {
     }
 }
 
-fn get_base_path(url: &Url) -> &str {
+fn get_base_path(url: &Url) -> Result<PathBuf, Error> {
     match url.scheme() {
-        "unix" => url.path(),
-        _ => url.as_str(),
+        "unix" => Ok(url.to_uds_file_path()?),
+        _ => Ok(url.as_str().into()),
     }
 }
 

edgelet/edgelet-http-mgmt/src/error.rs

@@ -50,6 +50,8 @@ pub enum ErrorKind {
     NotModified,
     #[fail(display = "Parse error")]
     Parse,
+    #[fail(display = "UTF-8 encode/decode error")]
+    Utf8,
 }
 
 impl Fail for Error {