Increase Total tokens to 128K (currently 4K)

[Padmaapparao]

For Doc Sum example as we will upload 100's of files, we need the input token length to be large and same with the output. Currently it is fixed at 4096 total, so if we upload even 1 large file, output token length for summarization will be only 32 tokens which is very very small for a summary.

need total tokens 128K, so we can get at least 16K-32K summary.
These values are hardcoded in compose.yaml. Need them to be parametrizable.

[lvliang-intel]

We are considering adding these parameters and making them configurable in compose.yaml to support flexible setups. A PR will be created for this, and we will update the details here once the PR is ready.

However, the models themselves currently don't support a 256K context length. And some hardware also have limitation to support large input token length and max output token length. We recommend exploring alternative approaches, such as chunking files or using recursive summarization techniques, to achieve optimal results within the current technical limitations.

[Padmaapparao]

Hi all Sorry for this question: not too familiar yet with the PR and issue filing. How do I find the PR associated with an issue Are they kinda linked together. Any easy search mechanism? Cheers -Padma From: Liang Lv ***@***.***> Sent: Tuesday, November 26, 2024 11:27 PM To: opea-project/GenAIExamples ***@***.***> Cc: Apparao, Padma ***@***.***>; Author ***@***.***> Subject: Re: [opea-project/GenAIExamples] Increase Total tokens to 256K (currently 4K) (Issue #1186) We are considering adding these parameters and making them configurable in compose.yaml to support flexible setups. A PR will be created for this, and we will update the details here once the PR is ready. However, the models themselves currently don't support a 256K context length. And some hardware also have limitation to support large input token length and max output token length. We recommend exploring alternative approaches, such as chunking files or using recursive summarization techniques, to achieve optimal results within the current technical limitations. — Reply to this email directly, view it on GitHub<#1186 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AL3WNHWUFBHQ6PJKC76SCKL2CVX3PAVCNFSM6AAAAABSLGXBI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBTGEYDKOBWGM>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[yongfengdu]

@Padmaapparao Once a PR was created and referred to/mentioned this issue, it will link automatically.

[eero-t]

(Already merged) PRs for supporting longer documents (with current small token amounts are):

• Support Long context for DocSum GenAIComps#981
• DocSum Long Context add auto mode GenAIComps#1046
• Support Long context for DocSum #1255

[joshuayao]

Hi @Padmaapparao, as @eero-t mentioned, OPEA now offers multiple strategies to support long contexts for DocSum, including auto, stuff, truncate, map_reduce, and refine. Please refer to the section MegaService with long context of the doc for more details. Could we proceed with closing this issue if these PRs align with your requirements?

[joshuayao]

Closed for no active responses in the recent 30 days. Please feel free to reopen it if the PRs do not resolve the issue.