"Error in python': free(): invalid next size (normal)" when reading boolean columns from Parquet files into tfio.IODataset`

[bguisard]

Hi,

I am running into a corrupted memory issue when I try to read boolean columns from parquet files.

This colab notebook reproduces the error. I'm also attaching my local error log for convenience since Colab doesn't seem to expose the error that caused it to crash:

*** Error in `python': free(): invalid next size (normal): 0x00007f8648009640 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f87e96db7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f87e96e437a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f87e96e853c]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(+0x1dc8b61)[0x7f8701e02b61]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(_ZN5arrow10PoolBufferD0Ev+0x2f)[0x7f8701ceccef]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x56)[0x7f8700ba8eb6]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(_ZN7parquet21TypedColumnReaderImplINS_12PhysicalTypeILNS_4Type4typeE0EEEE4SkipEl+0x28f)[0x7f8701d59a3f]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(+0x17f3a44)[0x7f870182da44]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(+0x17f0110)[0x7f870182a110]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/../libtensorflow_framework.so.2(+0x10a0961)[0x7f87c3899961]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(+0x956ef91)[0x7f87cdc6ff91]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/../libtensorflow_framework.so.2(+0xd7c250)[0x7f87c3575250]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN5Eigen15ThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x4b1)[0x7f87c7a0e141]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x43)[0x7f87c7a0ae33]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/../libtensorflow_framework.so.2(+0x1158e45)[0x7f87c3951e45]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f87e9a356ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f87e976b41d]
======= Memory map: ========
200000000-2400000000 ---p 00000000 00:00 0
... 
7ffde0221000-7ffde0249000 rw-p 00000000 00:00 0                          [stack]
7ffde031d000-7ffde0320000 r--p 00000000 00:00 0                          [vvar]
7ffde0320000-7ffde0322000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
[1]    3860 abort (core dumped)  python dataset_benchmark.py

[yongtang]

@bguisard Wondering if you can provide the pyarrow version, is it 3.0.0 or 2.0.0?

[bguisard]

@yashk2810

@bguisard Wondering if you can provide the pyarrow version, is it 3.0.0 or 2.0.0?

I didn't install pyarrow on my local environment. It's only used on the Colab notebook mentioned above, to generate parquet files with random data, so we can easily reproduce the error.

I'm guessing that the issue is not with pyarrow, but with the C++ ops used by tensorflow-io to parse parquet files.

[yongtang]

@bguisard PR #1285 updates the arrow C++ library to 3.0.0, with is also the library used for parsing parquet files. Can you give the latest nightly pip install tensorflow-io-nightly==0.17.0.dev20210127161146 a try?

[bguisard]

Thanks for the quick turn around @yongtang.

Unfortunately, the problem doesn't seem to go away after upgrading to tensorflow-io-nightly==0.17.0.dev20210127161146. The issue persists both in my local environment and on the Colab notebook referenced above.

*** Error in `python': free(): invalid next size (normal): 0x00007f62800093c0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f642298a7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f642299337a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f642299753c]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(+0x1d8bc71)[0x7f63100a9c71]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(_ZN5arrow10PoolBufferD0Ev+0x2f)[0x7f630ff9340f]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x56)[0x7f630ee45f56]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(+0x1ce3d26)[0x7f6310001d26]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(+0x17acb44)[0x7f630facab44]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow_io/core/python/ops/libtensorflow_io.so(+0x17a9210)[0x7f630fac7210]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/../libtensorflow_framework.so.2(+0x10a0961)[0x7f63fcb48961]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(+0x956ef91)[0x7f6406f1ef91]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/../libtensorflow_framework.so.2(+0xd7c250)[0x7f63fc824250]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN5Eigen15ThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x4b1)[0x7f6400cbd141]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x43)[0x7f6400cb9e33]
/mnt/env/anaconda3/envs/tf2-3/lib/python3.8/site-packages/tensorflow/python/../libtensorflow_framework.so.2(+0x1158e45)[0x7f63fcc00e45]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f6422ce46ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f6422a1a41d]

FYI:

❯ pip freeze | grep tensorflow
tensorflow==2.4.1
tensorflow-estimator==2.4.0
tensorflow-io-nightly==0.17.0.dev20210127161146
tensorflow-probability==0.12.1

[diggerk]

This is caused by a bug in the arrow library, see https://issues.apache.org/jira/browse/ARROW-11518 and apache/arrow#9432.

[yongtang]

@bguisard I have applied @diggerk 's patch (thanks 👍 ) to tensorflow-io and a new version of tensorflow-io-nightly is available:
https://pypi.org/project/tensorflow-io-nightly/0.17.0.dev20210208020836/#files

Can you give it a try?

[bguisard]

Hi @yongtang, I was able to read boolean columns successfully after you've pulled the fix in the arrow library. Thanks @diggerk for the quick fix.