Provide a migration path to using Events for Trace data

[dlvenable]

Is your feature request related to a problem? Please describe.

Data Prepper's existing Trace Analytics source and processors operate on ExportTraceServiceRequest records. Each of these ExportTraceServiceRequest objects contains multiple spans. With the new Event model, the Trace Analytics source and processors should operate on spans instead.

The current work to migrate to spans changes the way that the buffer behaves. With this change, the buffer will need to have many more records in it. Each record is smaller and thus the overall buffer size in MB should be the same. But, the buffer count does need to change.

If Data Prepper releases with the new Event model, then existing pipelines will experience a performance degradation which we have measured to be about half the throughput. Pipeline authors must reconfigure their pipeline to get back the original throughput.

Because this is somewhat of a breaking change, Data Prepper should have a migration path toward using the new model.

Describe the solution you'd like

Provide a migration path for existing Data Prepper pipelines to move to the new model.

• Add a new otel_trace_raw processor which operates on the new Event model.
• Keep otel_trace_raw_prepper on the old model.
• Add a record_type field to otel_trace_source to allow it to send out Events instead of the old model.

Here is a conceptual pipeline (shortened for brevity):

otel-trace-pipeline:
  source:
    otel_trace_source:
      record_type: event
  buffer:
    bounded_blocking:
       # Original recommendation of 512, multiplied by 20
       buffer_size: 10240
  sink:
    - pipeline:
        name: "raw-pipeline"
    - pipeline:
        name: "service-map-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "otel-trace-pipeline"
  buffer:
      bounded_blocking:
         # Configure the same value as in otel-trace-pipeline
         buffer_size: 10_240
  processor:
      # This used to be the otel_trace_raw_prepper.
    - otel_trace_raw:
  sink:
    ...

Describe alternatives you've considered (Optional)

1. Require pipeline authors update their pipeline. This would not be a good experience and I don't think we should take this approach.
2. Force the migration in Data Prepper 2.0. Pipeline authors would need to make changes to support Data Prepper 2.0. I think having a migration path is more ideal.

Additional context

Related issue: #546

Existing work: https://github.com/opensearch-project/data-prepper/tree/maint/546-migrate-trace-analytics-to-event-model

[chenqi0805]

Modules to be migrated from migration branch to main branch step by step:

• event model: MAINT: cherry-pick changes on event model from trace migration branch #1216
• otel-trace-raw-processor and otel-trace-group-processor: from corresponding preppers in migration branch: MAINT: add OTelTraceRawProcessor from trace ingestion migration branch #1223 , MAINT: add OTelTraceGroupProcessor from trace ingestion migration branch #1224
• otel-proto-common (codec): can be done in parallel with the above step, MAINT: add otel-proto-common from trace ingestion migration branch #1220
• service-map-stateful: accept either Record or Record, output Record, MAINT: migrate service map stateful to accept both Event and ExportTraceServiceRequest as record data type #1237
• otel-trace-source: buffer accept either Record or Record, MAINT: adjust otel-trace-source from trace event migration branch #1241
• peer-forwarder: accept either Record or Record, MAINT: peer forwarder from trace event migration branch #1239
• add e2e test to cover Event model trace pipeline compatibility with current main branch pipeline

[dlvenable]

@chenqi0805 , Thank you for putting these tasks together. Please add them in the Issue Description. If you do, GitHub will pick them up and report on them within boards.

Also, you said that we could migrate from the migration branch to main branch in different steps. Why not just merge in that branch and then start making the other enhancements as PRs directly into main?

[chenqi0805]

@dlvenable Thanks for the comment.

Why not just merge in that branch and then start making the other enhancements as PRs directly into main?

Essentially it is merging in that branch. But I am just doing it module by module to make sure the current trace ingestion flow does not break. Meanwhile, I can modify certain modules to meet the requirements of this issue. (e.g. otel-trace-source, service-map, peer-forwarder) If we forcefully merge that branch into main, there will be a lot of clean-ups to resolve conflicts and there will also be a lot of rework afterwards. (e.g. otel-trace-raw-prepper from migration branch still has to be renamed/refactored to coexist with the counterpart in main branch)