One app, several model providers

One internal shape, two adapters

Pick a shape your app uses everywhere, independent of any provider. A reply is text, a list of tool calls, why it stopped, and token counts; a tool call is an id, a name and a dictionary of arguments. The conversation history is a list of plain dictionaries: user turns, assistant turns (with any tool calls), and tool results.

history = [
    {"role": "user", "text": "Will Fern Hollow be dry on Saturday?"},
    {"role": "assistant", "text": "", "tool_calls": [{"id": "c1", "name": "forecast", "args": {"trailhead": "fern-hollow"}}]},
    {"role": "tool", "id": "c1", "result": "rain chance 10% before noon"},
]

Then each provider gets two functions: one that turns the internal request into that provider's JSON, and one that turns its response back into a reply. Nothing else in the app ever sees a provider's format.

import json

def from_messages(data):
    calls = [{"id": b["id"], "name": b["name"], "args": b["input"]} for b in data["content"] if b["type"] == "tool_use"]
    return {"text": "".join(b["text"] for b in data["content"] if b["type"] == "text"),
            "tool_calls": calls,
            "stop": {"end_turn": "done", "tool_use": "tool", "max_tokens": "length"}.get(data["stop_reason"], data["stop_reason"]),
            "tokens": (data["usage"]["input_tokens"], data["usage"]["output_tokens"])}

def to_messages(system, history, tools, max_tokens=500):
    messages = []
    for turn in history:
        if turn["role"] == "user":
            messages.append({"role": "user", "content": turn["text"]})
        elif turn["role"] == "assistant":
            blocks = [{"type": "text", "text": turn["text"]}] if turn["text"] else []
            blocks += [{"type": "tool_use", "id": c["id"], "name": c["name"], "input": c["args"]} for c in turn.get("tool_calls", [])]
            messages.append({"role": "assistant", "content": blocks})
        else:   # tool results travel inside one user message
            result = {"type": "tool_result", "tool_use_id": turn["id"], "content": turn["result"]}
            if messages and messages[-1]["role"] == "user" and isinstance(messages[-1]["content"], list):
                messages[-1]["content"].append(result)
            else:
                messages.append({"role": "user", "content": [result]})
    return {"system": system, "messages": messages, "max_tokens": max_tokens,
            "tools": [{"name": t["name"], "description": t["description"], "input_schema": t["schema"]} for t in tools]}

reply = {"content": [{"type": "text", "text": "Checking."}, {"type": "tool_use", "id": "toolu_1", "name": "forecast", "input": {"trailhead": "fern-hollow"}}],
         "stop_reason": "tool_use", "usage": {"input_tokens": 120, "output_tokens": 18}}
print(from_messages(reply))
history = [{"role": "user", "text": "Dry on Saturday?"},
           {"role": "assistant", "text": "", "tool_calls": [{"id": "toolu_1", "name": "forecast", "args": {"trailhead": "fern-hollow"}}]},
           {"role": "tool", "id": "toolu_1", "result": "rain chance 10%"}]
print(json.dumps(to_messages("Use the tools.", history, [])["messages"], indent=1)[:420])

The internal reply says "stop": "tool", not tool_use or tool_calls: the app's loop checks one word, whatever the provider. A reason the adapter doesn't recognize, such as a refusal, passes through unchanged rather than pretending to be done. Notice the tool result: in this format it rides inside a user message, so several results in a row share one message.

import json


def from_chat(data):
    message = data["choices"][0]["message"]
    return {"text": message["content"], "tool_calls": message.get("tool_calls", []), "stop": data["choices"][0]["finish_reason"], "tokens": None}


reply = {"choices": [{"message": {"role": "assistant", "content": None, "tool_calls": [
            {"id": "call_1", "type": "function", "function": {"name": "forecast", "arguments": '{"trailhead": "fern-hollow"}'}}]},
          "finish_reason": "tool_calls"}],
         "usage": {"prompt_tokens": 120, "completion_tokens": 18, "total_tokens": 138}}
print(from_chat(reply))

What adapters can't hide

Formats are the easy part. Some features exist on one provider and not the other, or work differently: prompt-caching markers, thinking, citations from documents, structured-output modes. Even parameter names drift: some newer chat-completions models expect a "developer" message instead of "system", and max_completion_tokens instead of max_tokens. For each, an adapter has three honest choices. Map it when the other provider has an equivalent. Leave it out when it's only an optimization: a cache marker the other side doesn't understand costs money, not correctness. Raise an error when dropping it would change the answer: a request that needs citations shouldn't quietly come back without any. Never drop something silently.

Streaming differs too. One API sends typed events (a block starts, text arrives, the block stops); the other sends chunks carrying small pieces of text or of a tool call's arguments. A streaming adapter turns both into the same internal events, such as "text arrived" and "tool call finished", so the screen code doesn't care. And model names belong to providers: keep one table that maps your app's own names, like "fast" and "careful", to each provider's current model ids, so trying a new model is a one-line change.

A gateway that falls back

With adapters in place, a gateway is a short function: try the first provider, and if it fails in a way another provider might not, try the next. Which failures count? A timeout, a rate limit (429) or an overloaded or broken server (500 and up) are about that provider right now: worth trying elsewhere. Other 4xx codes, such as 400, 401 or 413, mean your request is wrong (its format, its key, its size), and the next provider will most likely reject it too, so raise it and fix the bug. One exception is worth routing: a 400 because the conversation is too long for this model's context window is a reason to send it to a model with a bigger one. (Retrying the same provider once or twice, with a growing wait between tries, fits inside each provider's attempt.)

import time

class ProviderError(Exception):
    def __init__(self, status, message):
        super().__init__(f"{status}: {message}")
        self.status = status

def complete(request, providers):
    """providers: a list of (name, send) pairs; send(request) returns an internal reply or raises."""
    errors = []
    for name, send in providers:
        start = time.perf_counter()
        try:
            reply = send(request)
        except ProviderError as e:
            if 400 <= e.status < 500 and e.status != 429:
                raise                       # our request is wrong: another provider won't fix it
            errors.append(f"{name}: {e}")
            continue
        except TimeoutError:                # send() turns requests' own Timeout into TimeoutError
            errors.append(f"{name}: timed out")
            continue
        return {**reply, "provider": name, "seconds": round(time.perf_counter() - start, 3), "skipped": errors}
    raise RuntimeError("every provider failed: " + "; ".join(errors))

def overloaded(request):
    raise ProviderError(529, "overloaded")

def backup(request):
    return {"text": "Dry until noon.", "tool_calls": [], "stop": "done", "tokens": (60, 6)}

print(complete({"text": "Dry Saturday?"}, [("primary", overloaded), ("backup", backup)]))

Every reply records which provider answered and what was skipped, so your logs show how often the fallback ran. A gateway is also where routing lives: send short, easy requests to a cheaper model and long documents to one with a bigger context window, as the cost lesson suggested.

Rerun the evals before you switch

Adapters make switching possible; they don't make it safe. The same prompt behaves differently on another model: it may call tools more or less often, follow formatting rules more loosely, refuse different things, or write twice as much. So before moving traffic, run your eval set from the evals lesson through each provider via the same adapters, and compare like with like: the pass rate, the cost per case (each provider's token prices times its token counts), and the p95 latency. Expect to adjust prompts per provider; keep those variants in one place, next to the adapter, rather than scattering if provider == ... through the app.