[HUDI-7351] Implement partition pushdown for glue

[parisni]

Change Logs

This is a follow up of #10572

While the mentioned PR fixed the runtime error, it did not implement the logic for pushdown which was silently no returning partitions.

The current PR does so by:

• refactor to enable one implementation of pushdown per sync client
• provide a glue expression (diverge from hms)
• reorder the partitions in case they are returned misordered by the metastore
• optimize partition retrieval by missing the columns details
• UT has been copied and adapted from hiveSync
• IT has been introduced with moto to simulate aws env

Impact

Using this feature has proven faster metastore sync in case of large number of partitions, from couple of minutes to seconds.

Risk level (write none, low medium or high below)

Integration testing in aws environment has been done

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[parisni]

@danny0405 you were involved in the previous PR

[danny0405]

There are no UT, can you ensure the functionality by offline e2e tests?

[parisni]

I e2e tested locally. We will land this patch next week in production so let me confirm it's all right then.
BTW I added a refacto let me know if it's better

[parisni]

BTW @danny0405 I am working on a POC for IT test with moto to improve the reliability of this module #10614

[hudi-bot]

CI report:

• 8299f34 UNKNOWN
• c63161e Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[parisni]

@danny0405 Introduced IT tests for hudi-aws with dockerized moto here