Resolve approach for JSON path and dots in key names

[dlvenable]

Is your feature request related to a problem? Please describe.

How do we want to support both of these requirements:

• Providing a JSON-path structure for accessing fields within events
• Allowing fields to contain dots in the name e.g. my.field.

Raised by this question:
https://github.com/opensearch-project/data-prepper/pull/435/files#r732111116

[cmanning09]

I would like to propose support both by allowing users to escape the dot notation with a backslash.

For example setting foo\.bar to 42 would result in:

{
  "foo.bar":42
}

and foo.bar would look like:

{
  "foo": {
    "bar": 42
  }
}

[laneholloway]

Do we properly handle accessing an item in an array? e.g. foo.bar[5] = "f"

{
  "foo": {
    "bar": ["a", "b", "c", "d", "e", "f"]
  }
}

[cmanning09]

Other options proposed by @dlvenable in the original PR are:

• Use the Jackson JsonPointer syntax (/) instead of dot notation inspired by JsonPath (.)
• Support both methods like getPath vs. getKey to differentiate the two.

[cmanning09]

Do we properly handle accessing an item in an array? e.g. foo.bar[5] = "f"

{
  "foo": {
    "bar": ["a", "b", "c", "d", "e", "f"]
  }
}

Not at this moment, but it is a good point and something we should consider when making this decision.

[dlvenable]

Do we want to create our own syntax or try to use an existing one such as JsonPointer, JsonPath? We may be very close to Json Pointer already.

[cmanning09]

Do we want to create our own syntax or try to use an existing one such as JsonPointer, JsonPath? We may be very close to Json Pointer already.

That's a good question and one I have been contemplating.

Regarding creating our own, I am leaning less towards this direction at the moment. I would prefer to leverage an existing syntax over creating our own. Creating our own would give us more flexibility and wouldnt couople our implementation to our key structure but its going to introduce other challenges in maintainence, validations, translation to JsonPointer, supporting lists vs objects (as @laneholloway pointed out), etc.

Aligning with JsonPointer would be easier on implementation because of the underlying implementation is Jackson. However one could consider this coupling our key syntax to our implementation of an event. I do think we are moving that direction regardless if we are considering JsonPointer or JsonPath. I am leaning towards using JsonPointer as you (@dlvenable) suggested.

[laneholloway]

As long as we can index into arrays or lists in a complex document, i'm good with either. I'm more familiar with JsonPath, but I won't let my bias push one over the other.

[dlvenable]

I'm fine with either Json Pointer or Json Path. My understanding is that Json Path is more robust, but that robustness is primarily useful on the query side. And Json Pointer we have out-of-the-box support for with Jackson.

I will note that Json path can support dots in the path by way of ['my.single.key']. I find this a nice approach.

[laneholloway]

Then I think we should move forward with JsonPath.

[dlvenable]

I mistyped a sentence above in my comment yesterday.

Jackson has support for JsonPointer. It does not have support for JsonPath.

I corrected my original comment which said it supports JsonPath.

[dlvenable]

We could conceivably support both JsonPointer and JsonPath since they start differently. JsonPointer starts with a slash and JsonPath often starts with a dollar sign.
So we it would be reasonable to start by using JsonPointer. Then if users requested it, we could add support for JsonPath. And we'd know the difference based on how they start.

However, in either case, I'd like to support setting keys without a leading slash or dollar sign. What would we consider this to be? Just a raw key name with no path?

[laneholloway]

I think we can provide some semantics around that and say that a dotted notation key name (e.g. a.b.c.d) will be translated into a "path" in the underlying implementation as $a.b.c.d or /a/b/c/d based on either JsonPath or JsonPointer.

[cmanning09]

Supporting both at some point sounds nice, I know we will have to consider the scenario where a user's JsonPointer key contains a dot and they want to use JsonPath at some point. There may be other edge cases too.

That being said I would like us to pick one path forward for now. We can add support for the other if users request it like @dlvenable suggested. I am onboard with moving forward with JsonPointer.

[dlvenable]

I'm also onboard with moving forward with JsonPointer for now. Due to the possibility of supporting both, I think this is a two-way door.

[laneholloway]

Consensus reached, JsonPointer for initial implementation.