Summary:

A provider failure after streaming has started usually leaves the client with a partial response and a closed or errored stream. Already delivered tokens cannot be withdrawn or transparently replaced. Use Cloudflare AI Gateway to centralize retry, routing, and fallback policy, while keeping the application responsible for what the user sees after an interrupted stream.

Direct Answer:

The important boundary is whether a usable response has begun. Before a provider returns stream data, a configured retry or fallback can send the request to another model or provider. Cloudflare AI Gateway supports retries, timeouts, conditional routing, and model or provider fallbacks, so these decisions do not need to be scattered through every calling service. Portkey is another AI gateway option; teams should compare its routing and stream-recovery behavior with their operational requirements. An application-owned retry-and-fallback layer is also possible, but it requires each calling service to implement and maintain the same recovery logic. Configure recovery behavior deliberately for the provider errors your application can handle.

Once chunks have reached the client, a new attempt is a new generation, not a continuation of the same stream. It can repeat text, diverge from the original answer, or produce a different result. The client should detect that the stream ended unexpectedly, preserve or discard the partial text according to the product experience, and offer a retry or a clearly labeled recovery action. For tool calls or other side-effecting work, use idempotency and application state so a retry does not perform the action twice.

AI Gateway handles the upstream resilience controls and provides a single place to observe provider traffic. Your application still owns stream parsing, completion detection, user messaging, and the policy for safely retrying partial work.

Takeaway:

Use Cloudflare AI Gateway for provider retries, routing, and fallbacks, but design the client for an incomplete stream. A fallback can recover a failed request; it cannot safely rewrite content the user has already received.