[BUG] No of file list calls increased while moving jobs from databricks to OSS

[mani-sethu]

Bug

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Describe the problem

Steps to reproduce

Observed results

The number of list files on gcs have increased while moving the same code to OSS. This is across all the workloads, no specific code

Expected results

Ideally it should be same

Further details

Environment information

• Delta Lake version:
• Spark version:
• Scala version:

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

• Yes. I can contribute a fix for this bug independently.
• Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
• No. I cannot contribute a bug fix at this time.