System.Text.JsonSerializer - Provide a mechanism to begin deseralization at a specific name/value pair.

[sam-wheat]

Background and Motivation

It is common to enclose json serialized data in a named wrapper object i.e. { "data": {...} }. The wrapper object is typically ignored. It would be useful and convenient to be able to direct JsonSerializer to deserialize only the portion of a document starting at a specific name/value pair. This would allow the developer to avoid the use of temporary wrapper classes or custom JsonConverters.

Please consider the FRED API which is widely used in the finance world. Note the following:

• Both Json and XML data is wrapped in a named container object. In this example the container is <categories>...</categories> for xml data and "categories": {...} for json data.
• The name of the containing object differs based on the type of object being requested. For example, see the Series object. This is noteworthy because it precludes the use of a single generic class that can be used to wrap data elements of any desired type.

How might we efficiently implement clients to call the FRED API and deserialize either XML or Json data?

One approach is to create a base class as shown below with an abstract Parse method that can be implemented by respective Json / XML overrides.

This class also provides methods to download data by calling various APIs as defined by the data provider. In theory these methods should not need to be overridden by an inheriting class as the deseralization work is done only in the Parse method :

public abstract class BaseClient
{
	protected virtual async Task<Stream> Download(string uri)
	{
		uri = uri + (uri.Contains("?") ? "&" : "?") + "api_key=" + "xxxxx";
		var response = await new HttpClient() { BaseAddress = new Uri(uri) }.GetAsync(uri);
		return await response.Content.ReadAsStreamAsync();
	}

	protected abstract Task<T> Parse<T>(string uri, string root) where T : class, new();

        // should not need to override this
	public virtual async Task<Category> GetCategory(string categoryID)
	{
		string uri = "https://api.stlouisfed.org/fred/category?category_id=" + categoryID;
		return (await Parse<List<Category>>(uri, "categories"))?.FirstOrDefault();
	}
	
	public virtual async Task<Series> GetSeries(string seriesID) {...}
	
	public virtual async Task<Tag> GetTag(string tagID) {...}
	
	// etc... many methods here.
}

Here is an example of how an XML client might be implemented. Note we can pass a string in indicating where in the document the deserializer should start its work. This string is used to define the XmlRootAttribute object as shown. This implementation is basically effortless and allows us to use the stream returned by the Download method:

public class XMLClient : BaseClient
{
	protected override async Task<T> Parse<T>(string uri, string root)
	{
		return (T)new XmlSerializer(typeof(T), new XmlRootAttribute(root)).Deserialize(await Download(uri)); 
	}
}

Here is an example of a JSONClient. Unlike the XML implementation we have no mechanism to tell JsonSerializer where to begin it's work.

If we use use a wrapper class we must a) define a wrapper class for each type of object defined by the API, and b) override all Get... methods on the base i.e. Parse<List<Category>> becomes Parse<List<CategoryWrapper>>. In the example code below I use a JsonDocument to avoid these steps. However, this results in a costly string allocation and precludes us from using the stream provided by the Download method.

Another option is to write custom JsonConverter classes but this really defeats the purpose using a library for a well-known serialization protocol.

public class JSONClient : BaseClient
{
	protected override async Task<T> Parse<T>(string uri, string root)
	{
		uri = uri + (uri.Contains("?") ? "&" : "?") + "file_type=json";
		var document = JsonDocument.Parse((await Download(uri)), new JsonDocumentOptions { AllowTrailingCommas = true });
		string json = document.RootElement.GetProperty(root).GetRawText(); // string allocation 
		return JsonSerializer.Deserialize<T>(json);
	}
}

Proposed API

The functionality might be implemented by adding a property like RootElement to JsonSerializerOptions:

JsonSerializerOptions options = new JsonSerializerOptions { RootElement = "categories" };

Usage Examples

Given the following Json:

{
	"categories": [
		{
			"id": 125,
			"name": "Trade Balance",
			"parent_id": 13
		}
	]
}

This code will produce a List containing one object of the structure shown above:

string json = {...};
JsonSerializerOptions options = new JsonSerializerOptions { RootElement = "categories" };
List<Category> categories = JsonSerializer.Deserialize<Category>(json, options);

Alternative Designs

Please suggest.

Risks

None that I'm aware of.

[ghost]

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and Motivation

It is common to enclose json serialized data in a named wrapper object i.e. { "data": {...} }. The wrapper object is typically ignored. It would be useful and convenient to be able to direct JsonSerializer to deserialize only the portion of a document starting at a specific name/value pair. This would allow the developer to avoid the use of temporary wrapper classes or custom JsonConverters.

Please consider the FRED API which is widely used in the finance world. Note the following:

• Both Json and XML data is wrapped in a named container object. In this example the container is <categories>...</categories> for xml data and "categories": {...} for json data.
• The name of the containing object differs based on the type of object being requested. For example, see the Series object. This is noteworthy because it precludes the use of a single generic class that can be used to wrap data elements of any desired type.

How might we efficiently implement clients to call the FRED API and deserialize either XML or Json data?

One approach is to create a base class as shown below with an abstract Parse method that can be implemented by respective Json / XML overrides.

This class also provides methods to download data by calling various APIs as defined by the data provider. In theory these methods should not need to be overridden by an inheriting class as the deseralization work is done only in the Parse method :

public abstract class BaseClient
{
	protected virtual async Task<Stream> Download(string uri)
	{
		uri = uri + (uri.Contains("?") ? "&" : "?") + "api_key=" + "xxxxx";
		var response = await new HttpClient() { BaseAddress = new Uri(uri) }.GetAsync(uri);
		return await response.Content.ReadAsStreamAsync();
	}

	protected abstract Task<T> Parse<T>(string uri, string root) where T : class, new();

        // should not need to override this
	public virtual async Task<Category> GetCategory(string categoryID)
	{
		string uri = "https://api.stlouisfed.org/fred/category?category_id=" + categoryID;
		return (await Parse<List<Category>>(uri, "categories"))?.FirstOrDefault();
	}
	
	public virtual async Task<Series> GetSeries(string seriesID) {...}
	
	public virtual async Task<Tag> GetTag(string tagID) {...}
	
	// etc... many methods here.
}

Here is an example of how an XML client might be implemented. Note we can pass a string in indicating where in the document the deserializer should start its work. This string is used to define the XmlRootAttribute object as shown. This implementation is basically effortless and allows us to use the stream returned by the Download method:

public class XMLClient : BaseClient
{
	protected override async Task<T> Parse<T>(string uri, string root)
	{
		return (T)new XmlSerializer(typeof(T), new XmlRootAttribute(root)).Deserialize(await Download(uri)); 
	}
}

Here is an example of a JSONClient. Unlike the XML implementation we have no mechanism to tell JsonSerializer where to begin it's work.

If we use use a wrapper class we must a) define a wrapper class for each type of object defined by the API, and b) override all Get... methods on the base i.e. Parse<List<Category>> becomes Parse<List<CategoryWrapper>>. In the example code below I use a JsonDocument to avoid these steps. However, this results in a costly string allocation and precludes us from using the stream provided by the Download method.

Another option is to write custom JsonConverter classes but this really defeats the purpose using a library for a well-known serialization protocol.

public class JSONClient : BaseClient
{
	protected override async Task<T> Parse<T>(string uri, string root)
	{
		uri = uri + (uri.Contains("?") ? "&" : "?") + "file_type=json";
		var document = JsonDocument.Parse((await Download(uri)), new JsonDocumentOptions { AllowTrailingCommas = true });
		string json = document.RootElement.GetProperty(root).GetRawText(); // string allocation 
		return JsonSerializer.Deserialize<T>(json);
	}
}

Proposed API

The functionality might be implemented by adding a property like RootElement to JsonSerializerOptions:

JsonSerializerOptions options = new JsonSerializerOptions { RootElement = "categories" };

Usage Examples

Given the following Json:

{
	"categories": [
		{
			"id": 125,
			"name": "Trade Balance",
			"parent_id": 13
		}
	]
}

This code will produce a List containing one object of the structure shown above:

string json = {...};
JsonSerializerOptions options = new JsonSerializerOptions { RootElement = "categories" };
List<Category> categories = JsonSerializer.Deserialize<Category>(json, options);

Alternative Designs

Please suggest.

Risks

None that I'm aware of.

Author:	sam-wheat
Assignees:	-
Labels:	api-suggestion, area-System.Text.Json, untriaged
Milestone:	-

[eiriktsarpalis]

In the example code below I use a JsonDocument to avoid these steps. However, this results in a costly string allocation and precludes us from using the stream provided by the Download method.

We are planning on releasing a new writeable DOM with dynamic support for .NET 6 which will have built-in support for converting elements into objects. While this uses a different approach to what you are proposing, I think it would ultimately solve the problem you are describing cc @steveharter

[sam-wheat]

@eiriktsarpalis Would you mind showing a brief example of how it might be used in the problem I've illustrated?

[eiriktsarpalis]

The API is still being finalized but the rough idea is encapsulated in the design doc under API Walkthrough. Instead of passing the contents of JsonElement.GetRawText() to JsonSerializer.Deserialize you would instead use the JsonNode.GetValue<T>() method which directly converts the node to the specified .NET type.

[eiriktsarpalis]

Addressed by #29690 in .NET 6