Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dependency for pdf2image and OCR processing #421

Merged
merged 9 commits into from
Aug 21, 2024

Conversation

ZailiWang
Copy link
Collaborator

Description

Fix #414 as well as adding another tesseract-ocr dependency.

Issues

#414

Type of change

List the type of change like below. Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds new functionality)
  • Breaking change (fix or feature that would break existing design and interface)
  • Others (enhancement, documentation, validation, etc.)

Dependencies

poppler-utils
tesseract-ocr

Tests

Tested locally with langchain docker image.

@ZailiWang
Copy link
Collaborator Author

Testing with the file used in #414, the error is fixed.

Progress: |██████████████████████████████████████████████████| 100.0% CompleteDone preprocessing. Created  343  chunks of the original pdf
[ ingest chunks ] file name: nke-10k-2023.pdf
[ ingest chunks ] Current batch: 0

@ZailiWang
Copy link
Collaborator Author

@lvliang-intel would you kindly help review the PR? Thanks!

@ZailiWang ZailiWang requested a review from lvliang-intel August 7, 2024 06:45
@lvliang-intel
Copy link
Collaborator

@ZailiWang,
Please also add the dependency libs into the BOM.

@ZailiWang
Copy link
Collaborator Author

Would you tell where's the BOM file of this repo? I didn't find it.

lkk12014402 pushed a commit that referenced this pull request Aug 8, 2024
Signed-off-by: Yue, Wenjiao <[email protected]>
@chensuyue
Copy link
Collaborator

Would you tell where's the BOM file of this repo? I didn't find it.

Please ping me by teams, I will send you the link.

@chensuyue chensuyue merged commit 9397522 into opea-project:main Aug 21, 2024
11 checks passed
chensuyue pushed a commit that referenced this pull request Aug 21, 2024
tileintel pushed a commit to siddhivelankar23/GenAIComps that referenced this pull request Aug 22, 2024
srinarayan-srikanthan pushed a commit to siddhivelankar23/GenAIComps that referenced this pull request Sep 1, 2024
sharanshirodkar7 pushed a commit to predictionguard/pg-GenAIComps that referenced this pull request Sep 3, 2024
yinghu5 pushed a commit that referenced this pull request Sep 4, 2024
…e for v0.9 (#538)

* clip embedding support

Signed-off-by: srinarayan-srikanthan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <[email protected]>

* test script for embedding

Signed-off-by: srinarayan-srikanthan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <[email protected]>

* fix freeze workflow (#522)

Signed-off-by: Sun, Xuehao <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Fix Dataprep Potential Error in get_file (#540)

* fix get file error & refine logs

Signed-off-by: letonghan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Support SearchedDoc input type in LLM for No Rerank Pipeline (#541)

Signed-off-by: letonghan <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add dependency for pdf2image and OCR processing (#421)

Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add local_embedding return 768 length to align with chatqna example (#313)

Signed-off-by: Chendi.Xue <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* add telemetry doc (#536)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add video-llama LVM microservice under lvms  (#495)

Signed-off-by: BaoHuiling <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Fix the data load issue for structured files (#505)

Signed-off-by: XuhuiRen <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add finetuning component (#502)

Signed-off-by: Xinyu Ye <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <[email protected]>
Co-authored-by: test <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Letong Han <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* add torchvision into requirements (#546)

Signed-off-by: chensuyue <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Use Gaudi base images from Dockerhub (#526)

* Use Gaudi base images from Dockerhub

Signed-off-by: Abolfazl Shahbazi <[email protected]>

* Fixing the malformed tag

Signed-off-by: Abolfazl Shahbazi <[email protected]>

* fix another malformed tag

Signed-off-by: Abolfazl Shahbazi <[email protected]>

---------

Signed-off-by: Abolfazl Shahbazi <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* Add toxicity detection microservice (#338)

* Add toxicity detection microservice

Signed-off-by: Qun Gao <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Modification to toxicity plugin PR  (#432)

* changed microservice to use Service.GUARDRAILS and input/output to TextDoc

Signed-off-by: Tyler Wilbers <[email protected]>

* simplify dockerfile to use langchain

Signed-off-by: Tyler Wilbers <[email protected]>

* sort requirements

Signed-off-by: Tyler Wilbers <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Tyler Wilbers <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Minor SPDX header update (#434)

Signed-off-by: Abolfazl Shahbazi <[email protected]>

* Remove 'langsmith' per code review (#534)

Signed-off-by: Abolfazl Shahbazi <[email protected]>

* Add toxicity detection microservices with E2E testing

Signed-off-by: Qun Gao <[email protected]>

---------

Signed-off-by: Qun Gao <[email protected]>
Signed-off-by: Tyler Wilbers <[email protected]>
Signed-off-by: Abolfazl Shahbazi <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <[email protected]>
Co-authored-by: Tyler W <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* rename script and use 5xxx

Signed-off-by: BaoHuiling <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* add proxy for build

Signed-off-by: BaoHuiling <[email protected]>
Signed-off-by: srinarayan-srikanthan <[email protected]>

* fixed commit issues

Signed-off-by: srinarayan-srikanthan <[email protected]>

* Fix docarray constraint

Signed-off-by: srinarayan-srikanthan <[email protected]>

* updated docarray

Signed-off-by: srinarayan-srikanthan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rm telemetry which cause error in mega

Signed-off-by: BaoHuiling <[email protected]>

* renamed dirs

Signed-off-by: srinarayan-srikanthan <[email protected]>

* renamed test

Signed-off-by: srinarayan-srikanthan <[email protected]>

---------

Signed-off-by: srinarayan-srikanthan <[email protected]>
Signed-off-by: Sun, Xuehao <[email protected]>
Signed-off-by: letonghan <[email protected]>
Signed-off-by: Chendi.Xue <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
Signed-off-by: XuhuiRen <[email protected]>
Signed-off-by: Xinyu Ye <[email protected]>
Signed-off-by: chensuyue <[email protected]>
Signed-off-by: Abolfazl Shahbazi <[email protected]>
Signed-off-by: Qun Gao <[email protected]>
Signed-off-by: Tyler Wilbers <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sun, Xuehao <[email protected]>
Co-authored-by: Letong Han <[email protected]>
Co-authored-by: Zaili Wang <[email protected]>
Co-authored-by: Chendi.Xue <[email protected]>
Co-authored-by: Sihan Chen <[email protected]>
Co-authored-by: Huiling Bao <[email protected]>
Co-authored-by: XuhuiRen <[email protected]>
Co-authored-by: XinyuYe-Intel <[email protected]>
Co-authored-by: lkk <[email protected]>
Co-authored-by: test <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: chen, suyue <[email protected]>
Co-authored-by: Abolfazl Shahbazi <[email protected]>
Co-authored-by: qgao007 <[email protected]>
Co-authored-by: Tyler W <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

dataprep-redis: failed when uploading pdf with process_table=true
3 participants