Writable DOM and dynamic support for 6.0 #163

steveharter · 2020-11-03T16:09:35Z

High-level design and initial API thoughts to review before submitting the formal API issue.

Links to feature request dotnet/runtime#29690

jkotas · 2020-11-03T16:38:38Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+Thus a consumer would reference STJD.dll and then author code such as:
+```cs
+    var options = new JsonSerializerOptions();
+    options.EnableDynamicTypes();


What is going to be the default for ASP.NET ? Is ASP.NET going to enable the dynamic by default (and thus be fat out of the box), or not and be inconvenient to use out of the box? It is a very hard choice to make.

IMO it should be opt-in since:

It is faster, uses less private bytes and should have faster startup.

A rough benchmark shows that dynamic is ~2.6x slower when deserializing + serializing a smallish POCO with a few properties including a collection and nested object.

Non-breaking: backwards compat for deserializing System.Object types as JsonElement.

It is easy to opt-in.

nit: if dynamic is enabled by default it is not easy to opt-out: you'd have to remove the custom converter from options.Converters although we could add a helper for that.

However, if it is common or useful enough (including compat with Newtonsoft) then built-in could be considered. @pranavkm thoughts on how common the C# dynamic usage is?

From the code I've encountered, it's far more common to see the use of JObject in application code over the use of dynamic both to read and build a DOM. I would have expected that having a editable DOM that's available by default would be something we would want to enable.

A few points against enabling dynamic by default:

Globally changing JSON defaults in ASP.NET Core is fairly easy. We have documented how to enable or opt-out of defaults and users haven't had too many complaints.

Generally any kind of opaque type (dynamic / JObject / JsonNode) makes it harder to document an API. For 6.0, one of our themes is to make APIs more descriptive, so encouraging this pattern is counter to goals. That said, I doubt there's a big overlap of users that use dynamic in their APIs and also have Swagger enabled.

Undoing this option is apparently more tedious than enabling it. It would follow we make the easier choice and not enable it by default.

However the volume of support for the issue makes me want to second guess myself. I'm just surprised how popular it is and I don't think I have enough data to unilaterally make a decision here. I'll follow up with @bradygaster to get you a decision. It's great we're having this discussion early, we have ample time to get the right defaults before 6.0 ships.

I'm not a big fan of the dynamic support in Newtonsoft.Json. People want it to work in such a way that they just use .NET primitive types and everything just works. The reality is the fact you're using JValue/JObject/JArray leaks out.

re: dotnet/runtime#29690

IMO make a JsonConverter for ExpandoObject that spits out a value that uses ExpandoObject for JSON objects, List<object> for JSON arrays, and standard primitive values for everything else. Leave dynamic off JsonNode.

I'm not a big fan of the dynamic support in Newtonsoft.Json. People want it to work in such a way that they just use .NET primitive types and everything just works.

Can you elaborate a bit? Both Newtonsoft and the design here support .NET primitive types when in "edit mode" meaning after deserialization or for a new object, the user can assign a raw integer, List, string, standard POCO type, etc. to a property on a dynamic object (or add an element to dynamic array) and that gets serialized that as expected.

The reality is the fact you're using JValue/JObject/JArray leaks out.

During deserialization, when using "dynamic" at least, we need a type that implements IDynamicMetaObjectProvider thus we need to create JsonValue,JsonObject,JsonArray (or equivalent) on deserialization. We also want "no guess" semantics when it comes to numbers and strings meaning a JSON number won't automatically be converted to a CLR Int64, Decimal, Double, etc based on JSON contents. Similarly for string, it may be a quoted number or a DateTime, so we just let GetValue<T>() return what the user expects based on the <T>.

jkotas · 2020-11-03T16:39:59Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+
+There are implementations of `IDynamicMetaObjectProvider` including [DynamicObject](https://docs.microsoft.com/en-us/dotnet/api/system.dynamic.dynamicobject?view=net-5.0) and [ExpandObject](https://docs.microsoft.com/en-us/dotnet/api/system.dynamic.expandoobject?view=netcore-3.1). However, both implementations are not ideal:
+- There are many public members used for wiring up dynamic support, but are of little to no value for consumers so they would be confusing when mixed in with other methods intended to be used for writable DOM support.
+- It is not possible to efficiently combine writable DOM classes that are not tied to `dynamic` and `System.Linq.Expressions`:


The assumes the current implementation of dynamic. What would it take to change the implementation of dynamic to allow this combining to be done efficiently?

The goal is to avoid the dependency to SLE.dll for cases that don't actually use dynamic at run-time.

Today, even if IDynamicMetaObjectProvider is placed into a new small assembly, the one member the interface has expands into many more types in SLE.dll:

System.Dynamic.DynamicMetaObject GetMetaObject (System.Linq.Expressions.Expression parameter)

One approach is to add a new IDynamicMetaObjectProviderEx interface in a small assembly (probably with little to no code) and where the equivalent of GetMetaObject only returns types from that same small assembly.

Another approach, which may not be feasible, is to expand the linker in some way to handle this.

cc @MadsTorgersen

The goal is to avoid the dependency to SLE.dll for cases that don't actually use dynamic at run-time.

I agree with this.

Today, even if IDynamicMetaObjectProvider is placed into a new small assembly,

This is just one of the design options. For example, an interesting option may be to introduce a new light-weight interface in System.Runtime that supports dynamic scenarios like this one.

pakrym · 2020-11-03T16:53:11Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+There are implementations of `IDynamicMetaObjectProvider` including [DynamicObject](https://docs.microsoft.com/en-us/dotnet/api/system.dynamic.dynamicobject?view=net-5.0) and [ExpandObject](https://docs.microsoft.com/en-us/dotnet/api/system.dynamic.expandoobject?view=netcore-3.1). However, both implementations are not ideal:
+- There are many public members used for wiring up dynamic support, but are of little to no value for consumers so they would be confusing when mixed in with other methods intended to be used for writable DOM support.
+- It is not possible to efficiently combine writable DOM classes that are not tied to `dynamic` and `System.Linq.Expressions`:
+  - If combined, there would need to be an assembly reference to the very large `System.Linq.Expressions.dll` event when `dynamic` features are not needed.


Can the linker be updated to be able to handle the case where IDynamicMetaObjectProvider implementation is never used as dynamic and trim the reference to System.Linq.Expressions?

The dynamic support is only enabled by the serializer by calling the new options.EnableDynamicTypes() method, so we have a root method that the linker would be able to understand and trim appropriately. That root method adds the custom converter that references IDynamicMetaObjectProvider.

If EnableDynamicTypes is not called, and the dynamic keyword is used, there will be a run-time error during deserialization.

Also, as explained elsewhere, the writable DOM can be used without dynamic and without referencing SLE.dll.

stephentoub · 2020-11-03T16:57:12Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+- The dynamic support code example.
+  - During 5.0, [dynamic support](https://github.com/dotnet/runtime/issues/29690) was not able to be implemented due to schedule constraints. Instead, a [code example](https://github.com/dotnet/runtime/pull/42097) was provided that enables `dynamic` and a writeable DOM that can be used by any 3.0+ runtime in order to unblock the community and gather feedback for a 6.0 feature.
+- Azure prototyping.
+  - The Azure team needs a writable DOM, and supporting `dynamic` is important for some scenarios but not all. A [prototype](https://docs.microsoft.com/en-us/dotnet/api/azure.core.dynamicjson?view=azure-dotnet-preview) has been created. Work is also being coordinated with the C# team to support Intellisense with `dynamic`, although that is not expected to be implemented in time to be used in 6.0.


Work is also being coordinated with the C# team to support Intellisense with dynamic, although that is not expected to be implemented in time to be used in 6.0.

Issue link in dotnet/roslyn or dotnet/csharplang?

@KrzysztofCwalina @MadsTorgersen is there a link based on our discussion from a few days ago?

I see dotnet/roslyn#24226 but not sure if that is the one being used to track the discussion around supporting intellisense through defining "roles" which can be linked to a known schema where the members are known but not necessarily the Type names.

accepted/2020/serializer/WriteableDomAndDynamic.md

stephentoub · 2020-11-03T17:00:40Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+Having `System.Text.Json.dll` directly reference `System.Linq.Expressions.dll` is not desired:
+- Implementations of Blazor and stand-alone apps that do not use `dynamic` features do not want to carry the large `System.Linq.Expressions.dll`.
+  - Referencing `System.Linq.Expressions.dll` will require ~5.5MB of disk size. `System.Linq.Expressions.dll` is around 5MB in size but since it also references to `System.Linq` (~400K) and `System.ObjectModel` (~80K) (which that are not referenced by STJ) it becomes ~5.5MB (crossgen'd size).
+  - STJ is considering removing its reference to using reflection emit, so referencing `System.Linq.Expressions.dll` would also require keeping the references to the various System.Reflection.* assemblies. However, since these are currently implemented in the runtime, it is not a factor today for disk size.


However, since these are currently implemented in the runtime, it is not a factor today for disk size.

Can you elaborate on this? The core logic behind reflection / reflection emit is in the runtime, but there's plenty of surface area in managed code exposing all that functionality. Is that surface area inconsequential?

Also, we should assume that we will be able trim these one way or the other in future.

Can you elaborate on this? The core logic behind reflection / reflection emit is in the runtime, but there's plenty of surface area in managed code exposing all that functionality. Is that surface area inconsequential?

Thanks yes I'll re-word that. The surface area is only ~30K, but good to point out.

accepted/2020/serializer/WriteableDomAndDynamic.md

pakrym · 2020-11-03T17:05:09Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+        public JsonValueKind ValueKind { get; }
+    }
+
+    public sealed class JsonArray : JsonNode, IList<object>


IList<object>?

Why not IList<JsonNode>

See https://github.com/dotnet/designs/pull/163/files#diff-34fc37119c1d4491b844207aeed3a9c846c3e2d835fd712e164b1e47b89687c0R358

You could get around this by making JsonNode have implicit constructors to convert from primitive values to JsonNode.

The doc does mention some reasons why implicit operators are not used (not supported in all languages, consistency for types that we don't know, not needed when using dynamic).

You could get around this by making JsonNode have implicit constructors to convert from primitive values to JsonNode.

This is how the initial prototype solved the issue, but kept JsonArray and IList<JsonNode>:
https://github.com/ahsonkhan/corefx/blob/5bc2806f33090e78b38fafe4d5b46d5a0a4c1f08/src/System.Text.Json/ref/System.Text.Json.cs#L240-L259

not supported in all languages

Does that mean we shouldn't have any implicit operators in the framework?

consistency for types that we don't know

What do you mean by this?

Does that mean we shouldn't have any implicit operators in the framework?

For serialization, yes IMHO. Partly because it results in an inconsistent programming model (see below; also applies to explicit operators) but also because an implicit operator should not throw and it should be a simple, lossless, fast conversion which doesn't hold for serialization.

consistency for types that we don't know

What do you mean by this?

We can only add operators to known types such as string etc. We can't add operators to custom types (e.g. a third party Currency struct or a DateTimeOffset custom converter). So by not supporting operators we level the "known" types along with the "unknown" types and thus have a consistent programming model across both.

pakrym · 2020-11-03T17:05:24Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+        public override T GetValue<T>();
+    }
+
+    public sealed class JsonObject : JsonNode, IDictionary<string, object>


Why not IDictionary<string, JsonNode>?

Currently the prototype has IDictionary<string, JsonNode>; I'll update the API doc to match.

The original thinking was that someone may want to do this:

JsonObject jObject = ... jObject["MyString"] = "hello"; // "hello" is a raw string, not a JsonObject (and no implicit cast from string->JsonObject)

and if we have IDictionary<string, JsonNode> then you'd have to do this:

JsonObject jObject = ... jObject["MyString"] = new JsonValue("hello");

Since then, however, indexers were added to JsonNode that must return JsonNode, not object, in order to get the chaining of arrays and properties so it makes sense to use IDictionary<string, JsonNode> and IList<JsonNode> for arrays.

pakrym · 2020-11-03T17:08:36Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+    string str;
+
+    // This works, but has no caching:
+    byte[] utf8 = JsonSerializer.SerializeToUtf8Bytes(node, options);


I would've had a hard time figuring out that to write out the JsonNode I need to use the serializer.

Serializer feels more like a thing for customer-provider types and not something that already represents JSON.

The design, like Newtonsoft's, allows any CLR type to be used in "edit mode". Thus the shape of the JSON is not always known.

"Serialize()" instance methods were added to JsonNode to make this easier (don't need to pass in options) and more discoverable. The existing JsonSerializer.Serialize() methods can still be used however.

However, no "Deserialize()" static methods were added -- the existing JsonSerializer.Deserialize() methods should be used. Note that since a JsonNode can also be a property type or element type of an existing POCO\collection, in many cases no Serialize\Deserialize methods are necessary.

Serializer feels more like a thing for customer-provider types and not something that already represents JSON.

I agree. Why not just ToJsonString()?

I agree. Why not just ToJsonString()?

I'm open here. I recently added static Parse() methods and instance-based "To()" methods. However, since they are all "optional" they are in a second-class priority in a way.

accepted/2020/serializer/WriteableDomAndDynamic.md

pranavkm · 2020-11-09T18:46:27Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+Thus a consumer would reference STJD.dll and then author code such as:
+```cs
+    var options = new JsonSerializerOptions();
+    options.EnableDynamicTypes();


From the code I've encountered, it's far more common to see the use of JObject in application code over the use of dynamic both to read and build a DOM. I would have expected that having a editable DOM that's available by default would be something we would want to enable.

A few points against enabling dynamic by default:

Globally changing JSON defaults in ASP.NET Core is fairly easy. We have documented how to enable or opt-out of defaults and users haven't had too many complaints.

Generally any kind of opaque type (dynamic / JObject / JsonNode) makes it harder to document an API. For 6.0, one of our themes is to make APIs more descriptive, so encouraging this pattern is counter to goals. That said, I doubt there's a big overlap of users that use dynamic in their APIs and also have Swagger enabled.

Undoing this option is apparently more tedious than enabling it. It would follow we make the easier choice and not enable it by default.

However the volume of support for the issue makes me want to second guess myself. I'm just surprised how popular it is and I don't think I have enough data to unilaterally make a decision here. I'll follow up with @bradygaster to get you a decision. It's great we're having this discussion early, we have ample time to get the right defaults before 6.0 ships.

JamesNK

More thought needs to go into querying JSON data and using LINQ.

What does https://www.newtonsoft.com/json/help/html/QueryJson.htm look like?
What does https://www.newtonsoft.com/json/help/html/DeserializeWithLinq.htm look like?

JamesNK · 2020-11-12T07:36:10Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+    public abstract class JsonNode
+    {


To chain indexing through multiple levels and to support LINQ queries:

Suggested change

public abstract class JsonNode

{

public abstract class JsonNode : IEnumerable<JsonNode>

{

public virtual JsonNode? this[object key]

{

get => throw new InvalidOperationException("Cannot access child value on {0}.".FormatWith(CultureInfo.InvariantCulture, GetType()));

set => throw new InvalidOperationException("Cannot set child value on {0}.".FormatWith(CultureInfo.InvariantCulture, GetType()));

}

Yeah it's messy, but do you want to be correct and hard to use, or messy and user friendly?

When using dynamic, this will work fine.

When not using dynamic, yes the current design is a bit rigid although is not that bad in "edit mode" since it does allow standard CLR types to be used and not forcing JsonNode-derived types. However, based on feedback I think we're leaning to a more loosely typed model -- the JsonDocument\JsonElement model today is very rigid, and I don't think we want the full set of "correctness" here.

The design was recently updated to assume JsonNode-based indexers for JsonArray and JsonObject.

JamesNK · 2020-11-12T07:39:44Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+        public JsonValueKind ValueKind { get; }
+    }
+
+    public sealed class JsonArray : JsonNode, IList<object>


Suggested change

public sealed class JsonArray : JsonNode, IList<object>

public sealed class JsonArray : JsonNode, IList<JsonNode>

Plus add implicit cast conversions from int, decimal, string, et al to JsonNode.

JamesNK · 2020-11-12T07:40:09Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+        public override bool TryGetValue(Type type, out object? value);
+    }
+
+    public sealed class JsonObject : JsonNode, IDictionary<string, object>


Suggested change

public sealed class JsonObject : JsonNode, IDictionary<string, object>

public sealed class JsonObject : JsonNode, IDictionary<string, JsonNode>

See earlier comment about implicit cast operators.

JamesNK · 2020-11-12T07:41:47Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+{
+    public abstract class JsonNode
+    {
+        public JsonNode(JsonSerializerOptions options = null);


Should this be public? IMO make this internal (is private internal the one that combines protected and internal?). Prevents people from implementing their own types that inherit from JsonToken.

Yes this should be internal. Thanks

accepted/2020/serializer/WriteableDomAndDynamic.md

JamesNK · 2020-11-12T07:55:06Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+- The existing serializer methods can be used which support UTF8, string, Utf8JsonReader\Writer and Stream. Any future overloads will just work.
+- Serializer-specific functionality including custom converters and quoted numbers just work.
+
+The `JsonSerializerOptions` is passed to `JsonNode` and used to specify any run-time added custom converters, handle other options such as quoted numbers, and to respect the `PropertyNameCaseInsensitive` to determine whether the string-based lookup of property names on `JsonObject` are case-sensitive (for both direct use of `JsonObject` and also indirect use through `dynamic`). If we have `static JsonValue Parse()` methods, those would also need an override to specify the `JsonSerializerOptions`.


JsonNode storing JsonSerializerOptions seems weird. Why not specify serialization options when doing the serialization related activities? (converting DOM to and from JSON)

I have had requests for JObject to support case-insensitive property name lookups but my solution was to add GetProperty(string, StringComparison) methods.

IMO if you want a way to customize how JSON DOM behaves then it should be its own JsonNodeOptions type with PropertyNameCaseInsensitive.

The custom converters hang off of the options, which allow conversion to any compatible type (enums, string-based numbers, etc).

Another example for using JsonNodeOptions is duplicate property name handling:
https://github.com/ahsonkhan/corefx/blob/5bc2806f33090e78b38fafe4d5b46d5a0a4c1f08/src/System.Text.Json/ref/System.Text.Json.cs#L270

JamesNK · 2020-11-12T07:58:10Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+Thus a consumer would reference STJD.dll and then author code such as:
+```cs
+    var options = new JsonSerializerOptions();
+    options.EnableDynamicTypes();


re: dotnet/runtime#29690

IMO make a JsonConverter for ExpandoObject that spits out a value that uses ExpandoObject for JSON objects, List<object> for JSON arrays, and standard primitive values for everything else. Leave dynamic off JsonNode.

JamesNK · 2020-11-12T08:14:15Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+        },
+    }
+```
+so it is possible to omit the options instance for non-root members. When a property or element is added, the options instance is set to the parent's instance:


How would this technically work? Do JsonToken instances have a reference to their parent? If so, what happens when a JsonObject instance is added as the child of two different parents?

In Newtonsoft.Json the JToken.Parent property works because adding a token that already has a parent to a different parent will clone it.

TonyValenti · 2020-12-24T16:26:54Z

Any chance there will be a prototype soon? We'd really like to test out a writable DOM for JSON.

TonyValenti · 2021-02-22T17:02:36Z

@steveharter -
Hi Steve!
I actually envision the method as being something like:

public static class JsonValue {
  public static JsonValue<T> Create<T>(T Value){
    return new JsonValue<T>(Value);
  }
}

This is to keep generics from being necessary when referenced at the call site. Similar to how a number of classes have a "Create" method that allows you to avoid specifying the generic arguments. For example:

var V = KeyValuePair.Create("Key", 32);

Creates a KeyValuePair<string, int>

eiriktsarpalis · 2021-03-01T16:18:46Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+    },
+}
+```
+and the terse syntax which omits the options instance for non-root members. When a property or element is added, the options instance is set to the parent's instance:


Wouldn't that lead to confusion if the same node instance is assigned to trees with different configs? It doesn't seem too unlikely to me (e.g. when some application caches a JsonValue in a static).

Per offline discussion, to support caching or related scenarios where multiple options are used, I added additional clarity in that section that allows different option instances to be specified within a tree, but during a given serialization, only the options from the node passed to serialize()\ToJsonString()\etc is used.

accepted/2020/serializer/WriteableDomAndDynamic.md

steveharter · 2021-03-02T16:51:23Z

I actually envision the method as being something like:
public static class JsonValue {
public static JsonValue Create(T Value)

Thanks. I added that as a possible factory method in the doc.

danmoseley · 2021-03-02T19:33:54Z

~~cc @elinor-fung who owns a source generator for Interop which I assume is still in scope for 6.0 (?)~~

(edit - disregard, wrong PR..)

terrajobst · 2021-03-02T20:10:28Z

Video

The Azure SDK has a design (and usability studies) that uses structs. Could we lift their design?
- It sounds like they aren't entirely happy with their design either, but it seems we might be able to merge it.
The Azure SDK team has a set of JSON payloads that users have to pass to their APIs. We should write sample code that shows how one would do that with the DOM to make sure it's not super busy/complicated.
- @KrzysztofCwalina will provide @steveharter with examples
The proposal is lazy, but it's class-based which will cause more allocations as one navigates
JsonNode
- GetValue<T> with value types will throw when the value is null. It seems robust code should use int? or call TryGetValue().
- Path should be a method
- The Deserialize() method should always take options
- It seems GetValue<T> and Deserialize<T> are very similar. Should they be combined?
- ToString() should return the JSON, with the special handling of simple string values not including the quotes, like JsonDocument
- WriteTo() should probably have an overload that accepts a Stream, maybe also one that accepts a TextWriter
- WriteTo() probably needs an async version
- Parse() should probably be the inverse of WriteTo, in terms of types it accepts and async overloads
- Should we remove ValueKind? JsonValue<T> will return Unspecified for deferred serialized objects
JsonArray
- The Add(object?) is a typo and should be removed, but it's useful for use with anonymous objects
- We should consider Add<T>(T value) that constructs a JsonValue<T> automatically
- It seems a bit unfortunate that this has deferred deserialization semantics, meaning, if you walk the DOM after calling Add<T>() it's a JsonValue<T> with no children, even if you added a compound object.
- People may want to pass in a size that they can index into. This would be different from List<T> where the specified size would implicitly add null values which makes them valid for indexing. Alternatively, we could make arrays work like a dictionary where the key is an index.
Action items
- Create samples for the Azure SDK (which is creating payloads)
- One sample for modifying an existing payload
- One sample for pure reading

General types:

namespace System.Text.Json.Node
{
    public abstract class JsonNode : System.Dynamic.IDynamicMetaObjectProvider {}
    public sealed class JsonArray : JsonNode, IList<JsonNode?> {}
    public sealed class JsonObject : JsonNode, IDictionary<string, JsonNode?> {}
    public abstract class JsonValue : JsonNode {}
    public sealed class JsonValue<T> : JsonValue {}
}

Full API

namespace System.Text.Json.Node
{
    public abstract class JsonNode : System.Dynamic.IDynamicMetaObjectProvider
    {
        internal JsonNode(); // prevent external derived classes.

        public JsonNode Clone(); // Deep clone.

        // Returns the options specified during creation\deserialization otherwise null.
        // If null, child nodes obtain value from parent nodes recursively.
        public JsonSerializerOptions? Options { get; }

        // JsonArray terse syntax support (no JsonArray cast necessary).
        public virtual JsonNode? this[int index] { get; set; }

        // JsonObject terse syntax (no JsonObject cast necessary).
        public virtual JsonNode? this[string propertyName] { get; set; }

        // JsonValue terse syntax (no JsonValue<T> cast necessary).
        // Return the internal value, a JsonElement conversion, or a custom conversion to the provided type.
        // Allows for common programming model when JsonValue<T> is based on JsonElement or a CLR value.
        // "TypeToGet" vs "T" to prevent collision on JsonValue<T>. (could just be "T" if made non-virtual and dispatch to internal virtual methods instead)
        public virtual JsonValue? GetValue<TypeToGet>();
        public virtual bool TryGetValue<TypeToGet>(out TypeToGet? value);
        // Overloads with the converter for <TypeToGet>:
        public virtual JsonValue? GetValue<TypeToGet>(JsonConverter converter);
        public virtual bool TryGetValue<TypeToGet>(JsonConverter converter, out TypeToGet? value);

        // Return the parent and root nodes; useful for LINQ.
        public JsonNode? Parent { get; }
        public JsonNode? Root { get; }

        // The JSON Path; same "JsonPath" syntax we use for JsonException information.
        public string Path { get; }

        // Serialize\deserialize wrappers. These are helpers and thus can be considered optional.
        // "TypeToDeserialize" could be "T" instead.
        public abstract TypeToDeserialize? Deserialize<TypeToDeserialize>();
        public abstract bool TryDeserialize<TypeToDeserialize>(out TypeToDeserialize? value);
        public string ToJsonString(); // serialize as a string
        // public byte[] ToUtf8Bytes(); // not proposed, but may be useful
        // WriteTo() terminology consistent with Utf8JsonWriter.
        public abstract void WriteTo(System.Text.Json.Utf8JsonWriter writer);
        // Parse() terminology consistent with Utf8JsonReader\JsonDocument.
        public static JsonNode? Parse(string? json, JsonSerializerOptions? options = null);
        public static JsonNode? ParseUtf8Bytes(ReadOnlySpan<byte> utf8Json, JsonSerializerOptions? options = null);
        public static JsonNode? ReadFrom(ref Utf8JsonReader reader, JsonSerializerOptions? options = null);

        // The ValueKind from deserialization. Not used internally but may be useful for consumers.
        public JsonValueKind ValueKind { get; }

        // JsonElement interop
        public static JsonNode GetNode(JsonElement jsonElement);
        public static bool TryGetNode(JsonElement jsonElement, [NotNullWhen(true)] out JsonNode? jsonNode);

        // Dynamic support; implemented explicitly to help hide.
        System.Dynamic.DynamicMetaObject System.Dynamic.IDynamicMetaObjectProvider.GetMetaObject(System.Linq.Expressions.Expression parameter);

        // Explicit operators (can throw) from known primitives.
        public static explicit operator bool(JsonNode value);
        public static explicit operator byte(JsonNode value);
        public static explicit operator DateTime(JsonNode value);
        public static explicit operator DateTimeOffset(JsonNode value);
        public static explicit operator decimal(JsonNode value);
        public static explicit operator double(JsonNode value);
        public static explicit operator Guid(JsonNode value);
        public static explicit operator short(JsonNode value);
        public static explicit operator int(JsonNode value);
        public static explicit operator long(JsonNode value);
        [CLSCompliantAttribute(false)]
        public static explicit operator sbyte(JsonNode value);
        public static explicit operator float(JsonNode value);
        public static explicit operator char(JsonNode value);
        [CLSCompliantAttribute(false)]
        public static explicit operator ushort(JsonNode value);
        [CLSCompliantAttribute(false)]
        public static explicit operator uint(JsonNode value);
        [CLSCompliantAttribute(false)]
        public static explicit operator ulong(JsonNode value);

        public static explicit operator bool?(JsonNode value);
        public static explicit operator byte?(JsonNode value);
        public static explicit operator DateTime?(JsonNode value);
        public static explicit operator DateTimeOffset?(JsonNode value);
        public static explicit operator decimal?(JsonNode value);
        public static explicit operator double?(JsonNode value);
        public static explicit operator Guid?(JsonNode value);
        public static explicit operator short?(JsonNode value);
        public static explicit operator int?(JsonNode value);
        public static explicit operator long?(JsonNode value);
        [CLSCompliantAttribute(false)]
        public static explicit operator sbyte?(JsonNode value);
        public static explicit operator float?(JsonNode value);
        public static explicit operator string?(JsonNode value);
        public static explicit operator char?(JsonNode value);
        [CLSCompliantAttribute(false)]
        public static explicit operator ushort?(JsonNode value);
        [CLSCompliantAttribute(false)]
        public static explicit operator uint?(JsonNode value);
        [CLSCompliantAttribute(false)]
        public static explicit operator ulong?(JsonNode value);

        // Implicit operators (won't throw) from known primitives.
        public static implicit operator JsonNode(bool value);
        public static implicit operator JsonNode(byte value);
        public static implicit operator JsonNode(DateTime value);
        public static implicit operator JsonNode(DateTimeOffset value);
        public static implicit operator JsonNode(decimal value);
        public static implicit operator JsonNode(double value);
        public static implicit operator JsonNode(Guid value);
        public static implicit operator JsonNode(short value);
        public static implicit operator JsonNode(int value);
        public static implicit operator JsonNode(long value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode(sbyte value);
        public static implicit operator JsonNode(float value);
        public static implicit operator JsonNode?(char value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode?(ushort value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode?(uint value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode?(ulong value);

        public static implicit operator JsonNode(bool? value);
        public static implicit operator JsonNode(byte? value);
        public static implicit operator JsonNode(DateTime? value);
        public static implicit operator JsonNode(DateTimeOffset? value);
        public static implicit operator JsonNode(decimal? value);
        public static implicit operator JsonNode(double? value);
        public static implicit operator JsonNode(Guid? value);
        public static implicit operator JsonNode(short? value);
        public static implicit operator JsonNode(string? value);
        public static implicit operator JsonNode(int? value);
        public static implicit operator JsonNode(long? value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode(sbyte? value);
        public static implicit operator JsonNode(float? value);
        public static implicit operator JsonNode?(char? value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode?(ushort? value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode?(uint? value);
        [CLSCompliantAttribute(false)]
        public static implicit operator JsonNode?(ulong? value);
    }

    public sealed class JsonArray : JsonNode, IList<JsonNode?>
    {
        public JsonArray(JsonElement jsonElement, JsonSerializerOptions? options = null);

        // Param-based constructors for easy constructor initializers.
        public JsonArray(JsonSerializerOptions? options, params JsonNode[] items);
        public JsonArray(params JsonNode[] items);

        public override JsonNode Clone();

        public override void WriteTo(System.Text.Json.Utf8JsonWriter writer);

        public static JsonArray? Parse(string? json, JsonSerializerOptions? options = null);
        public static JsonArray? ParseUtf8Bytes(ReadOnlySpan<byte> utf8Json, JsonSerializerOptions? options = null);
        public static JsonArray? ReadFrom(ref Utf8JsonReader reader, JsonSerializerOptions? options = null);

        // IList<JsonNode?> (some hidden via explicit implementation):
        public int Count { get ;}
        bool ICollection<JsonNode?>.IsReadOnly { get ;}
        public void Add(object? item);
        public void Add(JsonNode? item);
        public void Clear();
        public bool Contains(JsonNode? item);
        public IEnumerator<JsonNode?> GetEnumerator();
        public int IndexOf(JsonNode? item);
        public void Insert(int index, JsonNode? item);
        public bool Remove(JsonNode? item);
        public void RemoveAt(int index);
        void ICollection<JsonNode?>.CopyTo(JsonNode?[]? array, int arrayIndex);
        IEnumerator IEnumerable.GetEnumerator();
    }

    public sealed class JsonObject : JsonNode, IDictionary<string, JsonNode?>
    {
        public JsonObject(JsonSerializerOptions? options = null);
        public JsonObject(JsonElement jsonElement, JsonSerializerOptions? options = null);

        public override JsonNode Clone();

        public bool TryGetPropertyValue(string propertyName, outJsonNode? jsonNode);

        public override void WriteTo(System.Text.Json.Utf8JsonWriter writer);

        public static JsonObject? Parse(string? json, JsonSerializerOptions? options = null);
        public static JsonObject? ParseUtf8Bytes(ReadOnlySpan<byte> utf8Json, JsonSerializerOptions? options = null);
        public static JsonObject? ReadFrom(ref Utf8JsonReader reader, JsonSerializerOptions? options = null);

        // IDictionary<string, JsonNode?> (some hidden via explicit implementation):
        public int Count { get; }
        bool ICollection<KeyValuePair<string,JsonNode?>>.IsReadOnly { get; }
        ICollection<string> IDictionary<string,JsonNode?>.Keys { get; }
        ICollection<JsonNode?> IDictionary<string,JsonNode?>.Values { get; }
        public void Add(string propertyName,JsonNode? value);
        public void Clear();
        public bool ContainsKey(string propertyName);
        public IEnumerator<KeyValuePair<string,JsonNode?>> GetEnumerator();
        public bool Remove(string propertyName);
        void ICollection<KeyValuePair<string,JsonNode?>>.Add(KeyValuePair<string,JsonNode> item);
        bool ICollection<KeyValuePair<string,JsonNode?>>.Contains(KeyValuePair<string,JsonNode> item);
        void ICollection<KeyValuePair<string,JsonNode?>>.CopyTo(KeyValuePair<string,JsonNode>[] array, int arrayIndex);
        bool ICollection<KeyValuePair<string,JsonNode?>>.Remove(KeyValuePair<string,JsonNode> item);
        IEnumerator IEnumerable.GetEnumerator();
        bool IDictionary<string,JsonNode?>.TryGetValue(string propertyName, outJsonNode? jsonNode);
    }

    // Separate class to make it easy to check type via "if (node is JsonValue)"
    // and to support passing of a value class polymorphically (without the <T>)
    public abstract class JsonValue : JsonNode
    {
        public JsonValue(JsonSerializerOptions? options = null);

        // Possible factory method that doesn't require specifying <T> due to generic type inference:
        // public static JsonValue<T> Create(T value);
    }

    public sealed class JsonValue<T> : JsonValue
    {
        public JsonValue(T value, JsonSerializerOptions? options = null);

        // Allow a custom converter and JsonValueKind to be specified.
        public JsonValue(T value, JsonConverter? converter = null, JsonValueKind valueKind, JsonSerializerOptions? options = null);

        public override JsonNode Clone();

        public override TypeToReturn GetValue<TypeToReturn>();
        public override TypeToReturn GetValue<TypeToReturn>(JsonConverter converter);
        public override bool TryGetValue<TypeToReturn>(out TypeToReturn value);
        public override bool TryGetValue<TypeToReturn>(JsonConverter converter, out TypeToReturn value);

        // The internal raw value.
        public override T Value {get; set;}

        public override void WriteTo(System.Text.Json.Utf8JsonWriter writer);

        public static JsonValue<T> Parse(string? json, JsonSerializerOptions? options = null);
        public static JsonValue<T> ParseUtf8Bytes(ReadOnlySpan<byte> utf8Json, JsonSerializerOptions? options = null);
        public static JsonValue<T> ReadFrom(ref Utf8JsonReader reader, JsonSerializerOptions? options = null);
    }
}

Tragetaschen · 2021-03-03T15:13:51Z

accepted/2020/serializer/WriteableDomAndDynamic.md

+JsonValue jValue = jObject["MyProperty"];
+Debug.Assert(jValue is JsonValue<int>);
+int i = jValue.GetValue<int>(); // Same programming model as the sample above.
+Debug.Assert(i == 42);


Writable DOM and dynamic support for 6.0

5390675

steveharter added the api-needs-work label Nov 3, 2020

steveharter requested review from pakrym, terrajobst, ericstj, KrzysztofCwalina, layomia and bartonjs November 3, 2020 16:09

steveharter self-assigned this Nov 3, 2020

jkotas reviewed Nov 3, 2020

View reviewed changes

pakrym reviewed Nov 3, 2020

View reviewed changes

stephentoub reviewed Nov 3, 2020

View reviewed changes

pakrym reviewed Nov 3, 2020

View reviewed changes

accepted/2020/serializer/WriteableDomAndDynamic.md Outdated Show resolved Hide resolved

stephentoub reviewed Nov 3, 2020

View reviewed changes

accepted/2020/serializer/WriteableDomAndDynamic.md Outdated Show resolved Hide resolved

pakrym reviewed Nov 3, 2020

View reviewed changes

accepted/2020/serializer/WriteableDomAndDynamic.md Outdated Show resolved Hide resolved

pakrym reviewed Nov 3, 2020

View reviewed changes

accepted/2020/serializer/WriteableDomAndDynamic.md Outdated Show resolved Hide resolved

pakrym reviewed Nov 3, 2020

View reviewed changes

accepted/2020/serializer/WriteableDomAndDynamic.md Outdated Show resolved Hide resolved

pakrym reviewed Nov 3, 2020

View reviewed changes

accepted/2020/serializer/WriteableDomAndDynamic.md Outdated Show resolved Hide resolved

pranavkm reviewed Nov 9, 2020

View reviewed changes

steveharter added 2 commits November 9, 2020 15:22

Added TryGetValue overloads and misc doc updates

227cfa4

Assume ILLinker and not a separate assembly for dynamic

ba76a82

JamesNK reviewed Nov 12, 2020

View reviewed changes

Update design based on feedback and additional prototyping

5f8039a

eiriktsarpalis reviewed Mar 1, 2021

View reviewed changes

accepted/2020/serializer/WriteableDomAndDynamic.md Outdated Show resolved Hide resolved

steveharter added 2 commits March 2, 2021 10:18

Add nullable operators and JsonSerializerOptions clarity

9f4c4d6

Add possible JsonNode.Create factory method

ae8644b

steveharter added 2 commits March 2, 2021 11:24

Add comments around using S.T.J namespace instead of S.T.J.Node

a87dce7

Update background section

150cbf0

terrajobst added api-needs-work and removed api-ready-for-review labels Mar 2, 2021

Tragetaschen suggested changes Mar 3, 2021

View reviewed changes

steveharter added 3 commits March 3, 2021 13:58

Address API review feedback

3ec49eb

Remove Serializer dependency in Parse() and misc other changes

5d80894

Remove JsonNode.AsJsonElement() until we have scenarios

6437453

eiriktsarpalis mentioned this pull request Mar 15, 2021

System.Text.JsonSerializer - Provide a mechanism to begin deseralization at a specific name/value pair. dotnet/runtime#49598

Closed

ericstj mentioned this pull request Mar 25, 2021

Support "dynamic" and writable DOM dotnet/runtime#29690

Closed

steveharter added 5 commits March 25, 2021 15:52

Update based on feedback and prototyping

227c850

Remove element additions

a4353cf

Update operators and add samples

e6166ac

Remove JsonArray ctor overload with capacity

2620f5d

Update based on API review and implementation

8808b11

steveharter added api-approved and removed api-needs-work labels Apr 23, 2021

steveharter merged commit 1834392 into dotnet:main Apr 23, 2021

steveharter deleted the WritableDomAndDynamic branch April 23, 2021 17:29

jkotas mentioned this pull request May 25, 2021

How useful is C# "dynamic" with System.Text.Json.Nodes.JsonObject? dotnet/runtime#53195

Closed

TonyValenti mentioned this pull request Jun 8, 2021

Implicit vs Explicit Null (aka Null vs Not-Set) - JSON and WebAPI Related dotnet/runtime#53868

Closed

layomia mentioned this pull request Oct 12, 2021

Add string indexers for JsonElement dotnet/runtime#60279

Closed

-    public abstract class JsonNode
-    {
+    public abstract class JsonNode : IEnumerable<JsonNode>
+    {
+        public virtual JsonNode? this[object key]
+        {
+            get => throw new InvalidOperationException("Cannot access child value on {0}.".FormatWith(CultureInfo.InvariantCulture, GetType()));
+            set => throw new InvalidOperationException("Cannot set child value on {0}.".FormatWith(CultureInfo.InvariantCulture, GetType()));
+        }

	public sealed class JsonArray : JsonNode, IList<object>
	public sealed class JsonArray : JsonNode, IList<JsonNode>

	public sealed class JsonObject : JsonNode, IDictionary<string, object>
	public sealed class JsonObject : JsonNode, IDictionary<string, JsonNode>

Writable DOM and dynamic support for 6.0 #163

Writable DOM and dynamic support for 6.0 #163

Conversation

steveharter commented Nov 3, 2020

Choose a reason for hiding this comment

steveharter Nov 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveharter Nov 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pakrym Nov 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JamesNK left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JamesNK Nov 12, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TonyValenti commented Dec 24, 2020

TonyValenti commented Feb 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveharter commented Mar 2, 2021

danmoseley commented Mar 2, 2021 • edited Loading

terrajobst commented Mar 2, 2021 • edited Loading

Choose a reason for hiding this comment

steveharter Nov 9, 2020 •

edited

Loading

steveharter Nov 24, 2020 •

edited

Loading

pakrym Nov 3, 2020 •

edited

Loading

JamesNK left a comment •

edited

Loading

JamesNK Nov 12, 2020 •

edited

Loading

danmoseley commented Mar 2, 2021 •

edited

Loading

terrajobst commented Mar 2, 2021 •

edited

Loading