Columnar json writer for arrow-json

[adriangb]

To get an output like:

{
  "a": [1, 2, 3],
  "b": ["foo", "bar", null]
}

The idea is that I can attach a schema to this and it will be much more compact (and possibly more performant to serializer / deserialize?).

Does this sounds like a good idea that would be accepted to the package?

[adriangb]

cc @tustvold

[tustvold]

How would this encode nested types like ListArray, StructArray or MapArray?

This would also not lend itself to streaming reads, which is normally important to bound memory usage

[adriangb]

I'm not claiming to have thought it all the way through but {"list": [["a","b"],[],["c"]]} and such?

[tustvold]

That would only work for a list of primitives, a list of structs would need to encode the structs as list records to preserve the multiple levels of nullability, at which point you're back to effectively the current format, just exploded by one level

I think given:

• There would be little ability to share code between the two formats
• It would not be compatible with other arrow implementations
• It is unlikely to perform significantly differently, the major overhead of JSON is tokenising and integer/float parsing which would be unchanged
• There are open questions about supporting nested types
• It can't be decoded a batch at a time

It is hard for me to recommend including it in this repository.

Perhaps we could take a step back and ascertain what the desired outcome is? If it is just to reduce the size, running the current JSON format through lz4 will likely yield far greater returns for very little additional overhead compared to the costs of JSON parsing

[adriangb]

Closing this as wonfix