[META] Derived Fields

[qreshi]

Please describe the end goal of this project

"Schema on reads" is a mechanism where query result data can be defined at runtime of the query. This is useful for users who may not be aware of the structure of the data and want to explore its usage before choosing what and how to index it. More simply, this allows users to have additional attributes to query upon post-ingest. Derived fields are an enhancement to OpenSearch that provides this "schema on read" functionality.

Supporting References

The RFC for this feature request can be found here #1133

Issues

• [Derived Fields] Add DerivedFieldMapper #12506
• [Derived Fields] Add parsing support for derived fields in mappings #12507
• [Derived Fields] Add basic query support with derived fields #12508
• [Derived Fields] Add Aggregation support for Derived Fields #14590
• Scoring support
• Sorting support

I will update this section with created issues with more details. For the time being I wanted to create the META issue for tracking.

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@qreshi Thanks for filing, it would be useful to to create subtasks with additional details for some of those items, looking forward to seeing progress here.

[rishabhmaurya]

List of all tasks associated -

• Derived fields building blocks: DerivedFieldQuery, DerivedFieldScript and DerivedFieldValueFetcher - [DerivedFields] PR1 - DerivedFieldScript and query execution logic #12746
• Derived fields supported field types implementation - boolean, keyword, long, double, ip, geo_point, date - [DerivedFields] PR2 - Implementation for all supported types and DerivedFieldType #12808
• Derived fields Mapper definition and integration through index mapping way - Add DerivedFieldMapper and support parsing it in mappings #12569
• Support for adding multiple values from derived field scripts - [Derived Fields] Add support for emitting multiple values in DerivedFieldScripts #12837
• Support defining derived field through search requests and rest client. Also adds support for derived fields in fetch and highlight phase - [Derived Fields] PR4: Capability to define derived fields in search request #12850
• Support for object type in derived fields - [Feature Request] Support for object type in Derived Fields #13143
• • [Derived Field] Dynamic FieldType inference based on random sampling of documents #13592
• • [DerivedField] object type support in mappings and new settings for derived field [DerivedField] object type support in mappings and new settings for derived field  #13717
• • [DerivedField] Support DerivedField object type and integration with Mapper #13720
• • [Derived Field] Integration tests for derived fields #13721
• Documentation - [DOC] Document the Derived Field feature documentation-website#6943
• Support in dashboards - [Derived Fields] Derived fields integration with Dashboards OpenSearch-Dashboards#6817
• Performance benchmark
• Can it interfere with field level security?

Some additional sub features -

• Allow removal of derived fields from index mappings
• Support with concurrent segment search. LeafLookup instance shared among segments within a slice could be problem, assess if its a problem and how to fix it.

[jayaddison]

I'd like to be able to sort using derived fields in OpenSearch, and began attempting to prototype an implementation locally. It seems that determining the datatype of each derived field at query runtime (that will be required for subsequent comparator operations, I believe) is a challenge.

[rishabhmaurya]

@jayaddison could you point to your changes?

It seems that determining the datatype of each derived field at query runtime (that will be required for subsequent comparator operations, I believe) is a challenge.

We do that already when derived fields are defined in query. Check https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/mapper/DefaultDerivedFieldResolver.java#L99

[jayaddison]

Thanks for taking an interest @rishabhmaurya - I should mention that I've hardly made any progress on it, and it's been many years since I coded in Java. Even so, the commit/line where I'd run into this is:

https://github.com/openculinary/OpenSearch/blob/31a65207600568630dacd0cf66649ba9e3487c39/server/src/test/java/org/opensearch/search/query/QueryPhaseTests.java#L775

My plan is to develop test case(s) to expose where the gap in sort functionality exists, and then to get those to pass (initially perhaps with a naive solution, and then to optimize).