Add tutorial for avro dataset API

[burgerkingeater]

This PR Iincludes a tutorial for avro dataset API(https://github.com/tensorflow/io/blob/98f52b669496602757ee801afa6b91fab041482b/tensorflow_io/core/kernels/20190307-avro-dataset.md).

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[burgerkingeater]

@yongtang @terrytangyuan mind taking a look?

[burgerkingeater]

@yongtang kindly ping

[kvignesh1420]

@burgerkingeater thanks for the PR and sorry for the delayed response.

I see that you are committing the mnist dataset in avro format in this PR. Maybe you can use the already available avro data located here ?

[burgerkingeater]

@kvignesh1420 thanks for the reply. I see this avro file is being used by other test:

io/tests/test_serialization_eager.py

Line 100 in 6579047

os.path.dirname(os.path.abspath(__file__)), "test_avro", "weather.avro"

I would like to have different dataset for different purpose. But I reuploaded a dataset of much smaller size.

[burgerkingeater]

@kvignesh1420 could you take a look?

[kvignesh1420]

@burgerkingeater few things to note:

1. Can we replace the images by adding cells in the notebooks which output the schema and a sample mnist record?
2. Maybe we can pick up a few scenarios from the test cases and put it in this tutorial?

[burgerkingeater]

@kvignesh1420 thanks for the feedback,

1. Can we replace the images by adding cells in the notebooks which output the schema and a sample mnist record?
   updated
2. Maybe we can pick up a few scenarios from the test cases and put it in this tutorial?
   I did considered adding more cases from unit tests to the tutorial, but the examples in these unit tests require largely heterogeneous testing datasets, which is better to be added iteratively. This PR could be treated a starting point and I'm planing to enhance it step by step. However I just added one more example, and more explanation in this tutorial.

Let me know if you have additional feedback. Thanks.

[yongtang]

@burgerkingeater I think you may want to expand Overview section to give readers some background information about Avro format and its usage (e.g., where Avro is used, its integration with Kafka, etc). Not every reader in machine learning/tensorflow community are familiar with Avro format so background information may be needed.

You may also expand the Usage section with more details about the data file you used and how they are generated (Download a sample Avro file: part and Download the schema file corresponding for the sample Avro data: part)

If you are reusing the mnist data for training, I assume you are following some other similar tutorials but replacing tfrecord format with avro format? If that is the case, it might be better to provide the link of the other tutorial so that readers can easily do a comparison and following through different routes (tfrecord format vs avro format).

[burgerkingeater]

@yongtang replied inline:
@burgerkingeater I think you may want to expand Overview section to give readers some background information about Avro format and its usage (e.g., where Avro is used, its integration with Kafka, etc). Not every reader in machine learning/tensorflow community are familiar with Avro format so background information may be needed.
- updated

You may also expand the Usage section with more details about the data file you used and how they are generated (Download a sample Avro file: part and Download the schema file corresponding for the sample Avro data: part)
- updated

If you are reusing the mnist data for training, I assume you are following some other similar tutorials but replacing tfrecord format with avro format? If that is the case, it might be better to provide the link of the other tutorial so that readers can easily do a comparison and following through different routes (tfrecord format vs avro format).
_- we are not following other tutorials. we wrote some script to convert mnist from tfrecord to avro, and additionally trimmed the data for simplicity. As you can see, the mnist avro data we are using here only has 4 records, and the some fields are trimmed as well.

[burgerkingeater]

@yongtang @kvignesh1420 , updated the tutorial per feedback. thanks.

[yongtang]

@burgerkingeater Some additional background about the mnist format data you are using would be good. For example, the motivation (e.g., mnist is widely used as the beginner's guide in ml, etc), how is the avro mnist data file generated, the schema of the avro mnist and the matching comparison to normal mnist, etc.

[burgerkingeater]

@yongtang updated. Note although the avro file is originated from mnist, it has been trimmed and customized so it's significantly different from mnist. So I renamed mnist.avro to test.avro to avoid any confusion.

[burgerkingeater]

@yongtang @kvignesh1420 updated per feedback, added more background info. Let me know if you have any additional feedback. Thanks.

[burgerkingeater]

@yongtang @kvignesh1420 kindly ping..

[burgerkingeater]

@kvignesh1420 updated the tutorial per feedback. Please let me know how it looks. Thanks.

[kvignesh1420]

Please check the comments for minor edits.

[burgerkingeater]

@kvignesh1420 thanks for your quick response. Updated.

[kvignesh1420]

LGTM. @yongtang Please take a look.

[burgerkingeater]

@kvignesh1420 thanks, can we resolve the change request? @yongtang please let me know how it looks. thanks.

[yongtang]

@burgerkingeater Can you move testing data (test.avro, test.avsc, training.avro, training.avsc) to another PR? Once the other PR is merged, it will be possible to live test this PR (with URLs of testing data already in place).

[burgerkingeater]

@yongtang i just checked, the two files are shown in this PR as deleted files as they are no longer needed in this tutorial.

[kvignesh1420]

@burgerkingeater so you are using only test.avsc and test.avro in this tutorial? If that is the case then it might be confusing for the readers when they see that they are training the models on test.avro. I think it would be better to rename the files to train.avro and train.avsc ( since there is no testing/inference phase in the tutorial ).

Also, any idea why there were intermittent kernel crashes?

[burgerkingeater]

@kvignesh1420 thanks, i created another PR: #1278 for the data file renaming change.

[kvignesh1420]

@burgerkingeater #1278 has been merged. Please rebase so that only the notebook shows up as a changed file in this PR.

[burgerkingeater]

@kvignesh1420 Still not sure about the intermittent kernel crashes, I'm unable to reproduce it from my local machine, and there's not yet enough output from failed kernel log.

[burgerkingeater]

@kvignesh1420 done, thanks.

[MarkDaoust]

One way you may get more detailed logs form the crash is to run the notebook in a subprocess.

!curl -O https://raw.githubusercontent.com/tensorflow/io/0747f5d99d479c4c3e07268c06f70dc5ad76c953/docs/tutorials/avro.ipynb
!jupyter nbconvert --execute avro.ipynb

That way when the subprocess dies the Colab kernel is still up and you can see all the logs.

[kvignesh1420]

@MarkDaoust can we go ahead and merge this tutorial then? Let me know.

[MarkDaoust]

The current version did fail again for me in Colab just now. After 4 or 5 "restart and run all".

It would be good to get it fixed before publishing.

[burgerkingeater]

@MarkDaoust can you shed more lights? This is the output, and I didn't see any process running afterwards.

[NbConvertApp] Converting notebook avro.ipynb to html
[NbConvertApp] Executing notebook with kernel: python3
2021-01-19 11:25:32.664165: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-19 11:25:32.665549: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-19 11:25:33.313405: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: AVX2 FMA
2021-01-19 11:25:34.491664: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
[NbConvertApp] Writing 322486 bytes to avro.html

[kvignesh1420]

@MarkDaoust as per your suggestion, I ran the following in colab:

!curl -O https://raw.githubusercontent.com/tensorflow/io/0747f5d99d479c4c3e07268c06f70dc5ad76c953/docs/tutorials/avro.ipynb
!jupyter nbconvert --execute avro.ipynb

The output was as follows:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20850  100 20850    0     0   108k      0 --:--:-- --:--:-- --:--:--  108k
[NbConvertApp] Converting notebook avro.ipynb to html
[NbConvertApp] Executing notebook with kernel: python3
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
2021-01-19 19:33:25.670952: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
[NbConvertApp] ERROR | unhandled iopub msg: colab_request
2021-01-19 19:33:30.669364: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-19 19:33:30.670455: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-01-19 19:33:30.681147: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2021-01-19 19:33:30.681201: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (bec4304dada5): /proc/driver/nvidia/version does not exist
2021-01-19 19:33:30.681789: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-19 19:33:30.787381: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: AVX2 FMA
2021-01-19 19:33:32.256142: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-01-19 19:33:32.256615: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2299995000 Hz
[NbConvertApp] Writing 322930 bytes to avro.html

I did a "restart and run" over 8 times and checked the avro.html file. Surprisingly, I did not see any session crash messages.
Am I missing something?
cc: @burgerkingeater

[MarkDaoust]

That's an example of the process completing successfully.

Below I've attached one where it fails (with the debug flag), but that doesn't seem to give any additional information. Just which cell it failed in, and we already knew that.

Anyone have other ideas?

---

!jupyter nbconvert --execute --debug avro.ipynb

[NbConvertApp] Executing cell:
features = {
    'features[*]': tfio.experimental.columnar.VarLenFeatureWithRank(dtype=tf.int32),
    'label': tf.io.FixedLenFeature(shape=[], dtype=tf.int32, default_value=-100),
    'dataType': tf.io.FixedLenFeature(shape=[], dtype=tf.string)
}

schema = tf.io.gfile.GFile('train.avsc').read()

dataset = tfio.experimental.columnar.make_avro_record_dataset(file_pattern=['train.avro'],
                                                              reader_schema=schema,
                                                              features=features,
                                                              shuffle=False,
                                                              batch_size=3,
                                                              num_epochs=1)

for record in dataset:
    print(record['features[*]'])
    print(record['label'])
    print(record['dataType'])
    print("--------------------")

2021-01-19 19:35:37.754502: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set

... [deleted lines]...

2021-01-19 19:35:39.878253: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-01-19 19:35:39.878756: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2000144999 Hz
[NbConvertApp] ERROR | Kernel died while waiting for execute reply.
Traceback (most recent call last):
  File "/usr/local/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/nbconvertapp.py", line 338, in start
    self.convert_notebooks()

... [deleted lines] ...

  File "/usr/local/lib/python2.7/dist-packages/nbconvert/preprocessors/execute.py", line 473, in _wait_for_reply
    self._check_alive()
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/preprocessors/execute.py", line 456, in _check_alive
    raise DeadKernelError("Kernel died")
nbconvert.preprocessors.execute.DeadKernelError: Kernel died

[burgerkingeater]

@MarkDaoust thanks. Any way to see why kernel crashes? I run it locally and it never fails for me..

[kvignesh1420]

Related discussion: jupyter/nbconvert#1035
Resource constraints seem to affect nbconvert and give out this vague message.

[MarkDaoust]

No I'm running out of ideas here. I was expecting to see a segfault, core dumped or something more detailed from those subprocess logs.

Are there any debugging techniques you would have run locally that you could apply in Colab?

Colab does compile TensorFlow using tcmalloc, and I've seen that difference trip up extensions a few of times, but I have no way of knowing if that's what's going wrong here.

Tip: instead of doing a "restart and run all" you can copy/paste the !jupyter nbconvert --execute --debug avro.ipynb a number of times and do a single "run all" but that isn't getting us any more information.

If we're really at a dead end we could just publish it, a 1/10 intermittent failure on a non-core tutorial isn't a huge deal.

[burgerkingeater]

@MarkDaoust @kvignesh1420 it shouldn't be related to memory constraint as the training dataset is very small as colab has 8G mem. I'm running out of idea too.. I'm ok with publishing the tutorial given the chance of crash.

[burgerkingeater]

@MarkDaoust @yongtang @kvignesh1420 thanks!