I frequently get API timeout errors, deeply frustrated
I have tried various settings like:
client = OpenAI(api_key = os.getenv(โOPENAI_API_KEYโ),timeout=20.0, max_retries=3)
or also:
granular_timeout = httpx.Timeout(25.0, connect=3.0, read=20.0, write=2.0)
client = OpenAI(api_key = os.getenv(โOPENAI_API_KEYโ),timeout=granular_timeout, max_retries=2)
What am I doing wrong can anyone help?? Some requests are taking more then a minute as shown in the picture and I get timeout error that should not be happening.
Solved by _j in post #2 The problem is likely that you are not giving time for the AI to respond. A 20 second timeout terminates a non-streaming response before you receive any text, even if the AI is still generating. Then that setting retries uses the internal openai python library retry method to do the same thing (at โฆ
I have streaming enabled already. Should I keep timeout to 60.0 ??
or more granular timeout like:
granular_timeout = httpx.Timeout(60.0, connect=3.0, read=20.0, write=2.0)
2 months later
Until lately, I was using 30 seconds and five retries. My prompts are becoming increasingly complex, which made me get a lot of timeouts, too. This brought me to 60 seconds and one retry, which I am testing nowโno streaming needed for my use cases.
Related topics
Topic list, column headers with buttons are sortable.
Topic | Replies | Views | Activity |
981 | |||
820 | |||
369 | |||
1.3k | |||
103 |
This is most likely the correct answer, Iโm at 60 seconds for timeout and have no issues. Now if youโre in a production environment with customers, even 60 seconds would not be acceptable. You might want to try streaming responses.