Release arrow-rs / parquet minor version 53.4.0 (Jan 2025)

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Our release schedule is here https://github.com/apache/arrow-rs?tab=readme-ov-file#release-versioning-and-schedule

• We plan Release arrow-rs / parquet major version 54.0.0 (December 2024) #6342 in Dec

However, upstream in DataFusion we are planning to release without picking up arrow 54 until after DataFusion 44

• Release DataFusion 44.0.0 datafusion#13334
  To minimize downstream disruptions

However, there are several fixes / features (related to Utf8View support for example) that we would like to make available downstream before we release DataFusion 45 and thus I would like to propose another release from the 44 branch

Describe the solution you'd like

Create a 53.4.0 release:

• Release notes and bump version: [53.0.0_maintance] Prepare for 53.4.0 release -- version and changelog #6982
• Vote thread: https://lists.apache.org/thread/s6tqn3fw3dykb6d6glwxq9nyqzv0ggt6
• Release to crates.io / approval: https://lists.apache.org/thread/5nf1wnnxwsl85w1t9201fj6z8pk19043

Describe alternatives you've considered

I think @andygrove would like to make

apache/datafusion#13334 (comment)

I am +1 for releasing DF 44 without the next major version of Arrow. However, there are some bug fixes and performance improvements in Arrow that I would like to use so would like to see a new minor release (and am happy to help with this if needed).

Process:

I will create the release from the https://github.com/apache/arrow-rs/tree/53.0.0_maintenance branch

To include patches in this release, please create PRs targeting the 53.0.0_maintenance branch. Once that branch has the desired content we can cut the release

Additional context

Planned content (please add more if necessary)

• perf: Use Cow in get_format_string in FFI_ArrowSchema #6853
• fix: decimal conversion looses value on lower precision #6836
• fix: Encoding of List offsets was incorrect when slice offsets begin with zero #6805
• [arrow-cast] Support cast numeric to string view (alternate) #6816
• Enable matching temporal as from_type to Utf8View #6872
• [arrow-cast] Support cast boolean from/to string view #6822

Possibly (if someone makes backport PRs):

• add buffered data_pages to parquet column writer total bytes estimation #6862
• Handling nullable DictionaryArray in CSV parser #6830
• Add Array::shrink_to_fit(&mut self) #6790
• Support shrink to empty #6817
• fix: make GetCatalogsBuilder sort catalog names  #6864

here is the complete list of commits in 53.3.3 --> 54.0.0 for consideration:

• 53.3.0...54.0.0

[andygrove]

@himadripal I added #6836 for consideration for this point release

[alamb]

Update:

• There is now a maintenance branch where we will make the release from: https://github.com/apache/arrow-rs/tree/53.0.0_maintenance
• I added a process section above

Next steps are to start creating PRs (one or many) with the desired changes cherry-picked / backported targeting the 53.0.0_maintenance branch

[alamb]

Update here is we have begun making backports of select PRS to the 53.0.0_maintenance branch, for example:

• [53.0.0_maintenance] perf: Use Cow in get_format_string in FFI_ArrowSchema (#6853) #6937
• [53.0.0_maintenance] fix: decimal conversion looses value on lower precision (#6836) #6936

I personally plan to create PRs for string view related functionality and what I view as important bugs (in the "planned content" heading above). If anyone else would like something else, please make a PR targeting the 53.0.0 branch

I plan to make a release candidate around Jan 15, 2025

[alamb]

I have completed backporting the PRs I have planned.

If anyone else would like to make a backport PR please do; I plan to cut a RC next week as planned

[emilk]

I would love to get this in if I can:

• Add Array::shrink_to_fit(&mut self) to 53.4.0 (#6790) (#6817) #6962
  🙏

[alamb]

Also suggests:

• Add support StringView / BinaryView in interleave kernel #6779

[alamb]

Also suggests:

• Add support StringView / BinaryView in interleave kernel #6779

I don't think I will be able to get this one in in time, so I will begin work on the RC now

[alamb]

Here is a PR to update version + changlog:

• [53.0.0_maintance] Prepare for 53.4.0 release -- version and changelog #6982

[alamb]

I made an RC but it currently fails like this:

failures:

---- bad_data::test_invalid_files stdout ----
thread 'bad_data::test_invalid_files' panicked at parquet/tests/arrow_reader/bad_data.rs:65:9:
Found new file in bad_data, please add test: ARROW-GH-45185.parquet
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    bad_data::test_invalid_files

test result: FAILED. 52 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.73s

error: test failed, to rerun pass `-p parquet --test arrow_reader`
+ cleanup
+ '[' no = yes ']'
+ echo 'Failed to verify release candidate. See /var/folders/1l/tg68jc6550gg8xqf1hr4mlwr0000gn/T/arrow-53.4.0.XXXXX.0Fd6xzcI2S for details.'
Failed to verify release candidate. See /var/folders/1l/tg68jc6550gg8xqf1hr4mlwr0000gn/T/arrow-53.4.0.XXXXX.0Fd6xzcI2S for details.

I need to update the test. I will do so tomorrow

[alamb]

I made a PR to fix the tests

• [53.0.0_maintenance] Update parquet-testing pin, add tests for new invalid data #6985

[alamb]

We now have a vote thread outstanding:

• https://lists.apache.org/thread/s6tqn3fw3dykb6d6glwxq9nyqzv0ggt6

[alamb]

The release was approved and published
https://lists.apache.org/thread/5nf1wnnxwsl85w1t9201fj6z8pk19043

[alamb]

label_issue.py automatically added labels {'arrow'} from #6956

[alamb]

label_issue.py automatically added labels {'parquet'} from #6985

[alamb]

label_issue.py automatically added labels {'documentation'} from #6896

[alamb]

label_issue.py automatically added labels {'arrow-flight'} from #6964

[alamb]

label_issue.py automatically added labels {'object-store'} from #6964