Inquiry on the Implementation and Support for vlm_datasets Calibration Data Class

[aptsunny]

Hello,

I am currently working with the vlm_datasets calibration data class as outlined in the documentation here. I noticed that there seems to be some aspects of the implementation that might not be fully developed or documented.

Here are the details I've gathered from the YAML configuration:

model:
  type: Llava
  path: model path
  tokenizer_mode: slow
  torch_dtype: auto

calib:
  name: vlm_datasets
  type: img_txt
  download: False
  path: datasets path
  n_samples: 32
  bs: 1
  seq_len: 512
  preproc: vlm_general
  padding: True
  seed: *seed

I have a few questions regarding the vlm_datasets:

1. Is the vlm_datasets calibration data class fully implemented and supported?
2. Are there any known limitations or issues with the current implementation?
3. If it's not fully implemented, is there a timeline for when these features might be completed?

I would appreciate any guidance or clarification on this matter. If there is additional documentation or resources available, please let me know.

Thank you for your time and support.

Best regards

[gushiqiao]

@chengtao-lv

[helloyongyang]

general_custom_data_examples.zip
@aptsunny

[aptsunny]

general_custom_data_examples.zip @aptsunny

thx

[sasha-hailo]

Hello,
Besides @aptsunny 's question, I suspect there's another issue.
The YAML files contain preproc: vlm_general preprocessing method,
But vlm_general does not have any implementation or reference in the Python code.
Did you by chance forget to add this implementation?

Another question of understanding:
Your documentation page specifies a format of a calibration dataset.
Is it a standard format of VQA datasets? In other words, can I configure to use any relevant VQA dataset from HF, or does it need adaptation?

Thanks in advance!