Cloud for AI engineers

Compute: machines, containers, functions

You can run code three main ways. A virtual machine is a whole computer you manage, billed while it's on. A container service runs your Docker image and handles machines for you; many restart and scale it automatically. Serverless functions run a small piece of code per request, cost nothing while idle, and suit short, bursty work, but they start slowly after a quiet spell and usually have a time limit.

For AI work there's one more question: does it need a GPU? Calling a hosted model doesn't, since the provider's GPUs do the work. Serving your own open model, or fine-tuning one, does, and GPU machines are the most expensive thing in most cloud bills.

Managed model endpoints

Each big cloud also sells access to frontier models through its own account: Amazon Bedrock, Google Cloud's Vertex AI and Microsoft's Azure AI Foundry, for example, offer Claude among others. The models are the same, but the plumbing changes. You authenticate with the cloud's IAM instead of a separate API key, usage lands on the same bill as everything else, and you can keep traffic and data inside a chosen region, which some companies' data rules require.

Two things catch people out. Each region has quotas, a cap on tokens or requests per minute, which you may need to ask to raise before launch. And model names and available versions differ slightly per provider, so keep the model id in configuration rather than scattered through the code.

Secrets without long-lived keys

A secret manager stores credentials encrypted, controls who may read each one, logs every read and can rotate them on a schedule. Your service reads a secret at start-up, so changing it never means rebuilding the image.

Better still, avoid long-lived keys altogether. Clouds can give a running service its own identity, often called workload identity, and hand it short-lived credentials automatically. There's then no key to leak at all, and access is still checked by the same IAM policies.

Costs and budgets

Cloud bills grow quietly: a forgotten GPU machine, logs nobody deletes, or data flowing out to the internet, which providers charge for as egress. Model calls add a fourth meter, priced per token. Estimate before you build:

hours = 24 * 30
container = 2 * hours * 0.05        # two small containers at $0.05 an hour
storage = 200 * 0.02                # 200 GB at $0.02 per GB-month
egress = 50 * 0.09                  # 50 GB out at $0.09 per GB
tokens = 30_000_000 / 1_000_000 * 4 # 30 million input tokens at $4 per million
print(round(container + storage + egress + tokens, 2))

Then set a budget with alerts, say at 50%, 90% and 100% of what you expect, sent to people who will act on them. Budgets warn rather than stop spending, so the provider's own spending caps on model usage are worth setting too.

spend = [30, 45, 60, 80, 95, 120]   # spend so far, by day
budget = 100
crossed = [day for day, total in enumerate(spend, start=1) if total >= 0.9 * budget]
print(crossed[0] if crossed else None)

def alerts(spend_by_day, budget, thresholds=(0.5, 0.9, 1.0)):
    return []


print(alerts([10, 12, 15, 14, 20, 18, 25], 100))

A checklist for your first cloud project

• One project or account per environment (development, production), so a test can't touch real data.
• Service accounts with least privilege, and no personal logins inside apps.
• Secrets in the secret manager, ideally replaced by workload identity.
• Private buckets, presigned URLs for sharing, lifecycle rules for old data.
• A region chosen for your users and your data rules, with model quotas checked before launch.
• A budget with alerts at 50%, 90% and 100%, and spending caps on model usage.
• Logs and metrics turned on from day one, so the first incident isn't also the first time you look.

POLICY = [
    {"effect": "Allow", "action": "storage:read", "resource": "bucket/docs/*"},
    {"effect": "Allow", "action": "secrets:read", "resource": "secret/model-key"},
    {"effect": "Allow", "action": "model:invoke", "resource": "model/claude-opus-5-5"},
]
BUDGET = {"amount": 50, "alerts": [0.5, 0.9, 1.0], "notify": "ops@yourteam.example"}


def review(policy, budget):
    notes = []
    for s in policy:
        if s["effect"] == "Allow" and ("*" in s["action"] or s["resource"] in ("*", "bucket/*")):
            notes.append(f"Broad permission: {s['action']} on {s['resource']}. Can it be narrower?")
    if not budget.get("amount"):
        notes.append("Set a budget amount")
    if sorted(budget.get("alerts", [])) != [0.5, 0.9, 1.0]:
        notes.append("Use alerts at 50%, 90% and 100%")
    if "@" not in budget.get("notify", ""):
        notes.append("Send the alerts to an address someone reads")
    return notes


for note in review(POLICY, BUDGET) or ["Nothing to flag. Ask the tutor to review the deployment."]:
    print("-", note)