[BUG] Exception in substitute string processor shuts down processor work but not pipeline

[engechas]

Describe the bug
When the substitute string processor encounters an exception while processing, it throws that exception and shuts down the process worker thread. The rest of the pipeline remains active but unable to make progress on incoming data. Example exception that shut down a process worker thread:

2023-06-30T07:11:01.606 [s3-pipeline-processor-worker-1-thread-1] ERROR org.opensearch.dataprepper.pipeline.ProcessWorker - Encountered exception during pipeline s3-pipeline processing
java.lang.IllegalArgumentException: key _<redacted> must contain only alphanumeric chars with .-_ and must follow JsonPointer (ie. 'field/to/key')

To Reproduce
Steps to reproduce the behavior:

1. Create a pipeline with the substitute string processor
2. Ingest data that causes an exception to be thrown, example: invalid JSON pointer

Expected behavior
The entire pipeline should shut down rather than remaining partially alive but unable to process data

[asifsmohammed]

@engechas @graytaylor0 Will this PR fix this bug? #2945

[graytaylor0]

@asifsmohammed No that PR will just make the characters ._- valid at the start and end of the keys. There could still be other invalid characters in keys that would throw this exception

[kkondaka]

@graytaylor0 @asifsmohammed @dlvenable This is happening if the pipeline config has keys with invalid characters. I think we should check this in the config validation. For example, the following config caused this error

sample-pipeline:                                                                                                                                       
  source:                                                                                                                                              
    http:                                                                                                                                              
  processor:                                                                                                                                           
    - substitute_string:                                                                                                                               
        entries:                                                                                                                                       
          - source: "mes&sage"                                                                                                                         
            from: "A"                                                                                                                            
            to: "B"                                                                                                                                     
  sink:                                                                                                                                                
   - stdout:   

In the above config the source name containing invalid characters.

[dlvenable]

@kkondaka , I have a proposal in #1916 that will help with this solution by getting the key during processor start-up. It would also help with performance. If we can move toward that, we can have a good solution for a couple of issues we've been having.