[BUG] Parquet chunked reader can throw an 'unexpected short subpass' exception under certain conditions. #18043
Comments
nvdbaranec
added
bug
3 tasks
raydouglass
pushed a commit
that referenced
this issue
Feb 20, 2025
…18019) Fixes #18043 An incorrect computation in the subpass generation code would come to the conclusion that there weren't enough rows to decode for list columns under certain circumstances. This PR fixes the issue and I did a little bit of variable naming cleanup around the area. Ultimately the true source of the bug was poorly named variables causing them to be used incorrectly. Edit: I've disabled various checks in the chunked reader tests that expect specific chunk counts being returned from chunking operations. Changes to decompression temporary memory usage can make this unreliable. We will need a smarter solution down the road.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This was discovered as a byproduct of changes to nvcomp temporary memory usage for decompression. The change caused us to produce a slightly different set of chunks, exposing the underlying bug in the chunked reader itself (nvcomp was not doing anything wrong). Spark Rapids customers have experienced this as well, under difficult-to-reproduce conditions, so having a clean repro case here is nice.
To reproduce, build cudf using nvcomp 4.2.0.11 (#18042) and run the tests. Two of the list tests,
ParquetChunkedReaderInputLimitConstrainedTest.MixedColumnsandParquetChunkedReaderInputLimitTest.Listwill throw the exception.The text was updated successfully, but these errors were encountered: