Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ChatFireworks doesn't retry on API errors (max_retries parameter is fake) #29843

Open
5 tasks done
aimestereo opened this issue Feb 17, 2025 · 1 comment
Open
5 tasks done

ChatFireworks doesn't retry on API errors (max_retries parameter is fake) #29843

aimestereo opened this issue Feb 17, 2025 · 1 comment
Labels
🤖:bug Related to a bug, vulnerability, unexpected error with an existing feature investigate Flagged for investigation.

Comments

@aimestereo
Copy link

Checked other resources

  • I added a very descriptive title to this issue.
  • I searched the LangChain documentation with the integrated search.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

nothing custom, just copy/paste example from ChatFireworks documentation (notice specified parameter max_retries ):

import getpass
import os

if "FIREWORKS_API_KEY" not in os.environ:
    os.environ["FIREWORKS_API_KEY"] = getpass.getpass("Enter your Fireworks API key: ")
from langchain_fireworks import ChatFireworks

llm = ChatFireworks(
    model="accounts/fireworks/models/llama-v3-70b-instruct",
    temperature=0,
    max_tokens=None,
    timeout=None,
    max_retries=2,
    # other params...
)
messages = [
    (
        "system",
        "You are a helpful assistant that translates English to French. Translate the user sentence.",
    ),
    ("human", "I love programming."),
]
ai_msg = llm.invoke(messages)
print(ai_msg.content)

Error Message and Stack Trace (if applicable)

httpx.ReadError:

email-writer-9   | Traceback (most recent call last):
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
email-writer-9   |     yield
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
email-writer-9   |     resp = await self._pool.handle_async_request(req)
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
email-writer-9   |     raise exc from None
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
email-writer-9   |     response = await connection.handle_async_request(
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
email-writer-9   |     return await self._connection.handle_async_request(request)
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
email-writer-9   |     raise exc
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
email-writer-9   |     ) = await self._receive_response_headers(**kwargs)
email-writer-9   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
email-writer-9   |     event = await self._receive_event(timeout=timeout)
email-writer-9   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
email-writer-9   |     data = await self._network_stream.read(
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 32, in read
email-writer-9   |     with map_exceptions(exc_map):
email-writer-9   |          ^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
email-writer-9   |     self.gen.throw(value)
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
email-writer-9   |     raise to_exc(exc) from exc
email-writer-9   | httpcore.ReadError
email-writer-9   |
email-writer-9   | The above exception was the direct cause of the following exception:
email-writer-9   |
email-writer-9   | Traceback (most recent call last):
email-writer-9   |   File "/app/email_writer/flow.py", line 68, in run
email-writer-9   |     await handle_task(task, querier)
email-writer-9   |   File "/app/email_writer/worker.py", line 157, in handle_task
email-writer-9   |     result = await generate_email(agent_input)
email-writer-9   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/app/email_writer/llm/workflow.py", line 160, in generate_email
email-writer-9   |     state = await agent_graph.ainvoke(State(), config)
email-writer-9   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2122, in ainvoke
email-writer-9   |     async for chunk in self.astream(
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2007, in astream
email-writer-9   |     async for _ in runner.atick(
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 444, in atick
email-writer-9   |     await arun_with_retry(
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 128, in arun_with_retry
email-writer-9   |     return await task.proc.ainvoke(task.input, config)
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 543, in ainvoke
email-writer-9   |     input = await step.ainvoke(input, config, **kwargs)
email-writer-9   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 331, in ainvoke
email-writer-9   |     ret = await asyncio.create_task(coro, context=context)
email-writer-9   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/app/email_writer/llm/nodes/base.py", line 125, in __call__
email-writer-9   |     result = await self.runnable.ainvoke(state.model_dump())
email-writer-9   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 3058, in ainvoke
email-writer-9   |     input = await asyncio.create_task(part(), context=context)  # type: ignore
email-writer-9   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 305, in ainvoke
email-writer-9   |     llm_result = await self.agenerate_prompt(
email-writer-9   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 870, in agenerate_prompt
email-writer-9   |     return await self.agenerate(
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 830, in agenerate
email-writer-9   |     raise exceptions[0]
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 998, in _agenerate_with_cache
email-writer-9   |     result = await self._agenerate(
email-writer-9   |              ^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_fireworks/chat_models.py", line 603, in _agenerate
email-writer-9   |     response = await self.async_client.acreate(messages=message_dicts, **params)
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/base_completion.py", line 217, in _acreate_non_streaming
email-writer-9   |     response = await client.post_request_async_non_streaming(
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/api_client.py", line 188, in post_request_async_non_streaming
email-writer-9   |     response = await self._async_client.post(
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1859, in post
email-writer-9   |     return await self.request(
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1540, in request
email-writer-9   |     return await self.send(request, auth=auth, follow_redirects=follow_redirects)
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
email-writer-9   |     response = await self._send_handling_auth(
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
email-writer-9   |     response = await self._send_handling_redirects(
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
email-writer-9   |     response = await self._send_single_request(request)
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
email-writer-9   |     response = await transport.handle_async_request(request)
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
email-writer-9   |     with map_httpcore_exceptions():
email-writer-9   |          ^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
email-writer-9   |     self.gen.throw(value)
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
email-writer-9   |     raise mapped_exc(message) from exc
email-writer-9   | httpx.ReadError

httpx.ConnectError:

email-writer-9   | Traceback (most recent call last):
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
email-writer-9   |     yield
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
email-writer-9   |     resp = await self._pool.handle_async_request(req)
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
email-writer-9   |     raise exc from None
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
email-writer-9   |     response = await connection.handle_async_request(
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
email-writer-9   |     raise exc
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request
email-writer-9   |     stream = await self._connect(request)
email-writer-9   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection.py", line 156, in _connect
email-writer-9   |     stream = await stream.start_tls(**kwargs)
email-writer-9   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 67, in start_tls
email-writer-9   |     with map_exceptions(exc_map):
email-writer-9   |          ^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
email-writer-9   |     self.gen.throw(value)
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
email-writer-9   |     raise to_exc(exc) from exc
email-writer-9   | httpcore.ConnectError
email-writer-9   |
email-writer-9   | The above exception was the direct cause of the following exception:
email-writer-9   |
email-writer-9   | Traceback (most recent call last):
email-writer-9   |   File "/app/email_writer/flow.py", line 68, in run
email-writer-9   |     await handle_task(task, querier)
email-writer-9   |   File "/app/email_writer/worker.py", line 157, in handle_task
email-writer-9   |     result = await generate_email(agent_input)
email-writer-9   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/app/email_writer/llm/workflow.py", line 160, in generate_email
email-writer-9   |     state = await agent_graph.ainvoke(State(), config)
email-writer-9   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2122, in ainvoke
email-writer-9   |     async for chunk in self.astream(
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2007, in astream
email-writer-9   |     async for _ in runner.atick(
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 444, in atick
email-writer-9   |     await arun_with_retry(
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 128, in arun_with_retry
email-writer-9   |     return await task.proc.ainvoke(task.input, config)
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 543, in ainvoke
email-writer-9   |     input = await step.ainvoke(input, config, **kwargs)
email-writer-9   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 331, in ainvoke
email-writer-9   |     ret = await asyncio.create_task(coro, context=context)
email-writer-9   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/app/email_writer/llm/nodes/base.py", line 125, in __call__
email-writer-9   |     result = await self.runnable.ainvoke(state.model_dump())
email-writer-9   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 3058, in ainvoke
email-writer-9   |     input = await asyncio.create_task(part(), context=context)  # type: ignore
email-writer-9   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 305, in ainvoke
email-writer-9   |     llm_result = await self.agenerate_prompt(
email-writer-9   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 870, in agenerate_prompt
email-writer-9   |     return await self.agenerate(
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 830, in agenerate
email-writer-9   |     raise exceptions[0]
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 998, in _agenerate_with_cache
email-writer-9   |     result = await self._agenerate(
email-writer-9   |              ^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/langchain_fireworks/chat_models.py", line 603, in _agenerate
email-writer-9   |     response = await self.async_client.acreate(messages=message_dicts, **params)
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/base_completion.py", line 217, in _acreate_non_streaming
email-writer-9   |     response = await client.post_request_async_non_streaming(
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/api_client.py", line 188, in post_request_async_non_streaming
email-writer-9   |     response = await self._async_client.post(
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1859, in post
email-writer-9   |     return await self.request(
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1540, in request
email-writer-9   |     return await self.send(request, auth=auth, follow_redirects=follow_redirects)
email-writer-9   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
email-writer-9   |     response = await self._send_handling_auth(
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
email-writer-9   |     response = await self._send_handling_redirects(
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
email-writer-9   |     response = await self._send_single_request(request)
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
email-writer-9   |     response = await transport.handle_async_request(request)
email-writer-9   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
email-writer-9   |     with map_httpcore_exceptions():
email-writer-9   |          ^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-9   |   File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
email-writer-9   |     self.gen.throw(value)
email-writer-9   |   File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
email-writer-9   |     raise mapped_exc(message) from exc
email-writer-9   | httpx.ConnectError

BadGatewayError:

email-writer-4   | Traceback (most recent call last):
email-writer-4   |   File "/app/email_writer/flow.py", line 68, in run
email-writer-4   |     await handle_task(task, querier)
email-writer-4   |   File "/app/email_writer/worker.py", line 157, in handle_task
email-writer-4   |     result = await generate_email(agent_input)
email-writer-4   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/app/email_writer/llm/workflow.py", line 160, in generate_email
email-writer-4   |     state = await agent_graph.ainvoke(State(), config)
email-writer-4   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2122, in ainvoke
email-writer-4   |     async for chunk in self.astream(
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2007, in astream
email-writer-4   |     async for _ in runner.atick(
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 444, in atick
email-writer-4   |     await arun_with_retry(
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 128, in arun_with_retry
email-writer-4   |     return await task.proc.ainvoke(task.input, config)
email-writer-4   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 543, in ainvoke
email-writer-4   |     input = await step.ainvoke(input, config, **kwargs)
email-writer-4   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 331, in ainvoke
email-writer-4   |     ret = await asyncio.create_task(coro, context=context)
email-writer-4   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/app/email_writer/llm/nodes/base.py", line 125, in __call__
email-writer-4   |     result = await self.runnable.ainvoke(state.model_dump())
email-writer-4   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 3058, in ainvoke
email-writer-4   |     input = await asyncio.create_task(part(), context=context)  # type: ignore
email-writer-4   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 305, in ainvoke
email-writer-4   |     llm_result = await self.agenerate_prompt(
email-writer-4   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 870, in agenerate_prompt
email-writer-4   |     return await self.agenerate(
email-writer-4   |            ^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 830, in agenerate
email-writer-4   |     raise exceptions[0]
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 998, in _agenerate_with_cache
email-writer-4   |     result = await self._agenerate(
email-writer-4   |              ^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/langchain_fireworks/chat_models.py", line 603, in _agenerate
email-writer-4   |     response = await self.async_client.acreate(messages=message_dicts, **params)
email-writer-4   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/base_completion.py", line 217, in _acreate_non_streaming
email-writer-4   |     response = await client.post_request_async_non_streaming(
email-writer-4   |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/api_client.py", line 191, in post_request_async_non_streaming
email-writer-4   |     await self._async_error_handling(response)
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/api_client.py", line 119, in _async_error_handling
email-writer-4   |     self._raise_for_status(resp)
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/api_client.py", line 113, in _raise_for_status
email-writer-4   |     self._raise_for(response.status_code, get_error_message)
email-writer-4   |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/api_client.py", line 87, in _raise_for
email-writer-4   |     raise BadGatewayError(error_message())

RateLimitError:

email-writer-10  | Traceback (most recent call last):
email-writer-10  |   File "/app/email_writer/flow.py", line 68, in run
email-writer-10  |     await handle_task(task, querier)
email-writer-10  |   File "/app/email_writer/worker.py", line 157, in handle_task
email-writer-10  |     result = await generate_email(agent_input)
email-writer-10  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/app/email_writer/llm/workflow.py", line 160, in generate_email
email-writer-10  |     state = await agent_graph.ainvoke(State(), config)
email-writer-10  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2122, in ainvoke
email-writer-10  |     async for chunk in self.astream(
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2007, in astream
email-writer-10  |     async for _ in runner.atick(
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 444, in atick
email-writer-10  |     await arun_with_retry(
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 128, in arun_with_retry
email-writer-10  |     return await task.proc.ainvoke(task.input, config)
email-writer-10  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 543, in ainvoke
email-writer-10  |     input = await step.ainvoke(input, config, **kwargs)
email-writer-10  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 331, in ainvoke
email-writer-10  |     ret = await asyncio.create_task(coro, context=context)
email-writer-10  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/app/email_writer/llm/nodes/base.py", line 125, in __call__
email-writer-10  |     result = await self.runnable.ainvoke(state.model_dump())
email-writer-10  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 3058, in ainvoke
email-writer-10  |     input = await asyncio.create_task(part(), context=context)  # type: ignore
email-writer-10  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 305, in ainvoke
email-writer-10  |     llm_result = await self.agenerate_prompt(
email-writer-10  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 870, in agenerate_prompt
email-writer-10  |     return await self.agenerate(
email-writer-10  |            ^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 830, in agenerate
email-writer-10  |     raise exceptions[0]
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 998, in _agenerate_with_cache
email-writer-10  |     result = await self._agenerate(
email-writer-10  |              ^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/langchain_fireworks/chat_models.py", line 603, in _agenerate
email-writer-10  |     response = await self.async_client.acreate(messages=message_dicts, **params)
email-writer-10  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/base_completion.py", line 217, in _acreate_non_streaming
email-writer-10  |     response = await client.post_request_async_non_streaming(
email-writer-10  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/api_client.py", line 191, in post_request_async_non_streaming
email-writer-10  |     await self._async_error_handling(response)
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/api_client.py", line 119, in _async_error_handling
email-writer-10  |     self._raise_for_status(resp)
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/api_client.py", line 113, in _raise_for_status
email-writer-10  |     self._raise_for(response.status_code, get_error_message)
email-writer-10  |   File "/usr/local/lib/python3.12/site-packages/fireworks/client/api_client.py", line 83, in _raise_for
email-writer-10  |     raise RateLimitError(error_message())

Description

what I expect:

  • model client to handle API errors (including rate limit errors) by retrying them with some strategy

what I see:

  • a noticeable amount of workflows fails due to API errors:
    • BadGatewayError
    • httpx.ConnectError
    • httpx.ReadError
    • RateLimitError

ChatFireworks documentation specify parameter max_retries , example from documentation:

from langchain_fireworks import ChatFireworks

llm = ChatFireworks(
    model="accounts/fireworks/models/llama-v3-70b-instruct",
    temperature=0,
    max_tokens=None,
    timeout=None,
    max_retries=2,
    # other params...
)

Class ChatFireworks has this parameter, but it's not used, to be specific it:

  1. passes it to client wrappers:
  • Fireworks(BaseFireworks)
  • AsyncFireworks(BaseFireworks)
  1. then client wrappers don't use it when creating actual clients:
  • FireworksClient
  • ImageInference(FireworksClient)
  1. FireworksClient don't have/receive this parameter, but it also doesn't have any retry specific logic

Interim solution for those who experience issues with Fireworks models: add your own retry logic, for example as wrapper around model calls

System Info

System Information
------------------
> OS:  Darwin
> OS Version:  Darwin Kernel Version 23.5.0: Wed May  1 20:17:33 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6031
> Python Version:  3.12.8 (main, Dec  3 2024, 18:42:41) [Clang 19.1.7 ]

Package Information
-------------------
> langchain_core: 0.3.35
> langchain: 0.3.18
> langchain_community: 0.3.17
> langsmith: 0.3.8
> langchain_fireworks: 0.2.7
> langchain_openai: 0.3.5
> langchain_postgres: 0.0.13
> langchain_text_splitters: 0.3.6
> langgraph_sdk: 0.1.51

Optional packages not installed
-------------------------------
> langserve

Other Dependencies
------------------
> aiohttp: 3.11.12
> aiohttp<4.0.0,>=3.8.3: Installed. No version info available.
> async-timeout<5.0.0,>=4.0.0;: Installed. No version info available.
> dataclasses-json<0.7,>=0.5.7: Installed. No version info available.
> fireworks-ai: 0.15.12
> httpx: 0.28.1
> httpx-sse<1.0.0,>=0.4.0: Installed. No version info available.
> jsonpatch<2.0,>=1.33: Installed. No version info available.
> langchain-anthropic;: Installed. No version info available.
> langchain-aws;: Installed. No version info available.
> langchain-cohere;: Installed. No version info available.
> langchain-community;: Installed. No version info available.
> langchain-core<1.0.0,>=0.3.34: Installed. No version info available.
> langchain-deepseek;: Installed. No version info available.
> langchain-fireworks;: Installed. No version info available.
> langchain-google-genai;: Installed. No version info available.
> langchain-google-vertexai;: Installed. No version info available.
> langchain-groq;: Installed. No version info available.
> langchain-huggingface;: Installed. No version info available.
> langchain-mistralai;: Installed. No version info available.
> langchain-ollama;: Installed. No version info available.
> langchain-openai;: Installed. No version info available.
> langchain-text-splitters<1.0.0,>=0.3.6: Installed. No version info available.
> langchain-together;: Installed. No version info available.
> langchain<1.0.0,>=0.3.18: Installed. No version info available.
> langsmith-pyo3: Installed. No version info available.
> langsmith<0.4,>=0.1.125: Installed. No version info available.
> langsmith<0.4,>=0.1.17: Installed. No version info available.
> numpy: 1.26.4
> numpy<2,>=1.26.4;: Installed. No version info available.
> numpy<3,>=1.26.2;: Installed. No version info available.
> openai: 1.61.1
> openai<2.0.0,>=1.58.1: Installed. No version info available.
> orjson: 3.10.15
> packaging<25,>=23.2: Installed. No version info available.
> pgvector: 0.3.6
> psycopg: 3.2.4
> psycopg-pool: 3.2.4
> pydantic: 2.10.6
> pydantic-settings<3.0.0,>=2.4.0: Installed. No version info available.
> pydantic<3.0.0,>=2.5.2;: Installed. No version info available.
> pydantic<3.0.0,>=2.7.4: Installed. No version info available.
> pydantic<3.0.0,>=2.7.4;: Installed. No version info available.
> pytest: 8.3.4
> PyYAML>=5.3: Installed. No version info available.
> requests: 2.32.3
> requests-toolbelt: 1.0.0
> requests<3,>=2: Installed. No version info available.
> rich: 13.9.4
> sqlalchemy: 2.0.38
> SQLAlchemy<3,>=1.4: Installed. No version info available.
> tenacity!=8.4.0,<10,>=8.1.0: Installed. No version info available.
> tenacity!=8.4.0,<10.0.0,>=8.1.0: Installed. No version info available.
> tiktoken<1,>=0.7: Installed. No version info available.
> typing-extensions>=4.7: Installed. No version info available.
> zstandard: 0.23.0
@langcarl langcarl bot added the investigate Flagged for investigation. label Feb 17, 2025
@dosubot dosubot bot added the 🤖:bug Related to a bug, vulnerability, unexpected error with an existing feature label Feb 17, 2025
@aimestereo
Copy link
Author

aimestereo commented Feb 17, 2025 •
edited
Loading

Interim solution for those who experience issues with Fireworks models: add your own retry logic, for example as wrapper around model calls

I have a better solution for it and probably this bug is not longer valid. According to fireworks.AI documentation it has compatibility with OpenAI client.

You can use OpenAI Python client library to interact with Fireworks. This makes migration of existing applications already using OpenAI particularly easy.

That means, that langchain fireworks client can be marked as deprecated? (considering lack functionality) and client for OpenAI can be used instead.

In my case I used ChatOpenAI with api_key and base_url for fireworks.ai - works like a charm.

@keenborder786 keenborder786 mentioned this issue Feb 18, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🤖:bug Related to a bug, vulnerability, unexpected error with an existing feature investigate Flagged for investigation.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@aimestereo