Add WebSearch Retriever Microservice #209

Spycsh · 2024-06-18T08:42:23Z

Description

Add Web Retriever Microservice

Issues

opea-project/GenAIExamples#306

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)

Dependencies

chromadb
google-api-python-client>=2.100.0
bs4
html2text

Tests

tests/test_web_retrievers_langchain.sh

Signed-off-by: Spycsh <[email protected]>

for more information, see https://pre-commit.ci

Spycsh · 2024-06-18T08:43:48Z

CI environment should have

export GOOGLE_API_KEY=xxx
export GOOGLE_CSE_ID=xxx

before test is triggered.

This PR should not be merged until SearchQnA Example PR is ready.

codecov · 2024-06-18T08:44:15Z

Codecov Report

Attention: Patch coverage is 20.00000% with 16 lines in your changes missing coverage. Please review.

Files	Coverage Δ
comps/cores/mega/constants.py	`98.11% <100.00%> (ø)`
comps/cores/mega/gateway.py	`26.48% <15.78%> (ø)`

for more information, see https://pre-commit.ci

chensuyue · 2024-06-20T02:58:49Z

Codecov Report

Attention: Patch coverage is 20.00000% with 16 lines in your changes missing coverage. Please review.

Files Coverage Δ
comps/cores/mega/constants.py 98.03% <100.00%> (ø)
comps/cores/mega/gateway.py 26.43% <15.78%> (ø)

@Spycsh need to enhance ut coverage, patch coverage should large than 70% for this release.

…arch

for more information, see https://pre-commit.ci

* add gateway Signed-off-by: Spycsh <[email protected]> * Add web retriever comp * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add entry * add persistent dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add correct delete * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test_web_retrievers_langchain.sh * Update test_web_retrievers_langchain.sh * revert * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Spycsh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: chen, suyue <[email protected]> Signed-off-by: Wu, Xiaochang <[email protected]>

* add gateway Signed-off-by: Spycsh <[email protected]> * Add web retriever comp * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add entry * add persistent dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add correct delete * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test_web_retrievers_langchain.sh * Update test_web_retrievers_langchain.sh * revert * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Spycsh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: chen, suyue <[email protected]> Signed-off-by: sharanshirodkar7 <[email protected]>

* add gateway Signed-off-by: Spycsh <[email protected]> * Add web retriever comp * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add entry * add persistent dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add correct delete * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test_web_retrievers_langchain.sh * Update test_web_retrievers_langchain.sh * revert * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Spycsh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: chen, suyue <[email protected]> Signed-off-by: Yogesh Pandey <[email protected]>

* add gateway Signed-off-by: Spycsh <[email protected]> * Add web retriever comp * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add entry * add persistent dir * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add correct delete * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test_web_retrievers_langchain.sh * Update test_web_retrievers_langchain.sh * revert * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Spycsh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: chen, suyue <[email protected]> Signed-off-by: Daniel Whitenack <[email protected]>

Signed-off-by: lvliang-intel <[email protected]>

Spycsh and others added 3 commits June 17, 2024 00:54

add gateway

9a02ac2

Signed-off-by: Spycsh <[email protected]>

Add web retriever comp

f2de971

[pre-commit.ci] auto fixes from pre-commit.com hooks

2de11c4

for more information, see https://pre-commit.ci

add entry

8681297

Spycsh force-pushed the search branch from 9f282ec to 8681297 Compare June 19, 2024 05:36

Spycsh and others added 6 commits June 18, 2024 23:41

add persistent dir

8dda3bc

[pre-commit.ci] auto fixes from pre-commit.com hooks

5a21c7a

for more information, see https://pre-commit.ci

add correct delete

445da8f

add correct delete

c9a550f

[pre-commit.ci] auto fixes from pre-commit.com hooks

6e26d4c

for more information, see https://pre-commit.ci

Merge branch 'main' into search

5657724

Spycsh mentioned this pull request Jun 19, 2024

Add SearchQnA with GenAIComps opea-project/GenAIExamples#306

Merged

3 tasks

chensuyue mentioned this pull request Jun 20, 2024

add secrets in ci test #220

Merged

3 tasks

chensuyue and others added 3 commits June 20, 2024 11:14

Merge branch 'main' into search

dfd3fab

fix test

8d83616

Merge branch 'search' of https://github.com/Spycsh/GenAIComps into se…

9fbb90d

…arch

Spycsh requested review from lvliang-intel and chensuyue June 20, 2024 06:50

Spycsh and others added 2 commits June 20, 2024 21:45

Merge branch 'main' into search

b39d716

[pre-commit.ci] auto fixes from pre-commit.com hooks

72de4fe

for more information, see https://pre-commit.ci

Spycsh linked an issue Jun 20, 2024 that may be closed by this pull request

SearchQnA opea-project/GenAIExamples#295

Closed

lvliang-intel approved these changes Jun 21, 2024

View reviewed changes

lvliang-intel requested review from letonghan and zehao-intel June 21, 2024 02:08

Spycsh and others added 3 commits June 24, 2024 09:48

Merge branch 'main' into search

6a7fc9e

[pre-commit.ci] auto fixes from pre-commit.com hooks

9b93545

for more information, see https://pre-commit.ci

Update test_web_retrievers_langchain.sh

e599088

zehao-intel approved these changes Jun 24, 2024

View reviewed changes

Spycsh and others added 4 commits June 24, 2024 11:40

Update test_web_retrievers_langchain.sh

eae5d0c

Merge remote-tracking branch 'source/main' into search

be5aadb

revert

2fe5d23

[pre-commit.ci] auto fixes from pre-commit.com hooks

a1a11d0

for more information, see https://pre-commit.ci

Spycsh merged commit 6557688 into opea-project:main Jun 24, 2024
9 checks passed

lkk12014402 pushed a commit that referenced this pull request Aug 8, 2024

Remove model info in curl request (#209)

ee6debe

Signed-off-by: lvliang-intel <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add WebSearch Retriever Microservice #209

Add WebSearch Retriever Microservice #209

Spycsh commented Jun 18, 2024 •

edited

Loading

Spycsh commented Jun 18, 2024 •

edited

Loading

codecov bot commented Jun 18, 2024 •

edited

Loading

chensuyue commented Jun 20, 2024

Codecov Report

Add WebSearch Retriever Microservice #209

Add WebSearch Retriever Microservice #209

Conversation

Spycsh commented Jun 18, 2024 • edited Loading

Description

Issues

Type of change

Dependencies

Tests

Spycsh commented Jun 18, 2024 • edited Loading

codecov bot commented Jun 18, 2024 • edited Loading

Codecov Report

chensuyue commented Jun 20, 2024

Codecov Report

Spycsh commented Jun 18, 2024 •

edited

Loading

Spycsh commented Jun 18, 2024 •

edited

Loading

codecov bot commented Jun 18, 2024 •

edited

Loading