January 2, 2023 8 min read

GPT-3.5 on Azure OpenAI: What You Get Before text-davinci-003

Azure OpenAI OpenAI GPT-3.5 Python Cost Optimization

Since ChatGPT launched, “GPT-3.5” has become the label everyone asks for, and I keep getting the same question: can we have that on Azure? The honest answer is “partly”, because the model names on the OpenAI API and in the Azure OpenAI catalogue don’t line up one to one, and if you build against a name rather than a capability, you will rework your code the day the catalogue changes. The fix is to treat the model as configuration, not code, so the inevitable model swap is a settings change rather than a release.

What “GPT-3.5” actually refers to

GPT-3.5 isn’t one model. OpenAI uses it for a series of models trained on a blend of text and code, finishing training in early 2022. As of today there are three of them on the OpenAI API, and OpenAI’s model index for researchers (archived copy; OpenAI has since reorganised its docs) sets out how they relate:

Model	What it is	Max request (tokens)
`code-davinci-002`	Codex base model, tuned for code completion	8,001
`text-davinci-002`	InstructGPT-style model built on `code-davinci-002`	4,097
`text-davinci-003`	Improved `text-davinci-002`, released by OpenAI on 28 November 2022	4,097

ChatGPT itself is, in OpenAI’s words, “fine-tuned from a model in the GPT-3.5 series”. There is no ChatGPT API today, on OpenAI or on Azure. If someone is offering you a ChatGPT API today, ask what it calls. It’s likely to be automation over the web UI, which breaks OpenAI’s terms and is not something to put in front of customers.

The other thing worth getting straight: text-curie-001, text-babbage-001 and text-ada-001 are GPT-3 models, not GPT-3.5. I see them lumped into “GPT-3.5” in a lot of slide decks. They are older, smaller, cheaper, and have half the context window (2,049 tokens). That doesn’t make them useless. It means you should choose them on purpose.

What Azure OpenAI offers right now

Microsoft’s December 2022 update marks Azure OpenAI Service as generally available and calls API version 2022-12-01 the GA API. Access is still gated: you apply under Microsoft’s Limited Access policy, and production use needs a use-case review. If your subscription hasn’t been approved, you can’t create the resource at all, so start that process before you start designing.

Once you’re in, the GPT-3.5 models you can deploy are text-davinci-002 and code-davinci-002. text-davinci-003 is on the OpenAI API but is not in the Azure catalogue yet. New OpenAI models have so far reached Azure some time after their OpenAI debut, and there is no published date for this one.

Alongside those you have the older GPT-3 InstructGPT models (text-davinci-001, text-curie-001, text-babbage-001, text-ada-001), the smaller code-cushman-001 Codex model, and embedding models. The token limits in this post are OpenAI’s published limits for the same model versions, which Azure serves unchanged.

There are also practical constraints you’ll hit in the first week. You can only create the resource in East US, South Central US or West Europe, which matters if your data has to stay in a particular geography. Each deployment is rate-limited by requests per second (20 for Davinci models, 50 for the rest after the December 2022 update), and you can have at most two resources per region, so a batch job that fires requests in parallel will start getting HTTP 429 responses well before you expect it. The quotas and limits include one more rule that shapes your design: you can’t deploy the same model to more than one deployment in a resource.

Content filtering is temporarily off by default (Microsoft says it returns, on by default, in Q1 2023). When it’s on, an inappropriate prompt returns HTTP 400 and a filtered completion comes back as HTTP 200 with finish_reason set to content_filter and no text, so design for both now rather than discovering them after the switch flips.

Azure bills per 1,000 tokens, prompt and completion combined, at the same rate per model family, as listed on the Azure OpenAI Service pricing page. The table covers the text models; Codex models are priced separately and aren’t shown here:

Family	Price per 1K tokens	Relative to Ada
Davinci text models (incl. `text-davinci-002`)	$0.02	50×
Curie	$0.002	5×
Babbage	$0.0005	1.25×
Ada	$0.0004	1×

That 10× jump from Curie to Davinci is the whole reason model choice matters. A workload that sends 2,000 tokens per request, 50,000 times a month, costs roughly $2,000 a month on Davinci and $200 on Curie.

How I choose between them

My rule of thumb: prototype on text-davinci-002, then try to move down a tier for every task that turns out to be narrow.

Davinci is where you start because it follows instructions without needing many examples, and you want to know whether the task is possible before you worry about cost. Once the prompt is stable, re-run your test set against Curie. For classification, sentiment, simple extraction into a fixed schema, and routing, Curie with three or four good examples in the prompt is often close enough, and you’ve cut the bill by 90%.

Where I would not step down:

Multi-step instructions. “Summarise this, then list the risks, then draft a reply” falls apart on smaller models.
Long inputs. Curie’s 2,049-token window is shared between prompt and completion. A one-page document plus instructions plus a useful answer won’t fit.
Anything resembling reasoning over the input. Davinci-class models are noticeably better at following a chain of logic through a document.

And when should you not use any of these? When a deterministic tool already does the job. Azure Cognitive Service for Language handles named entity recognition, key phrase extraction and language detection with predictable output and a proper schema. A completion model that returns JSON most of the time is a worse entity extractor than a service that always returns JSON.

Design for the model swap

Whatever you build this month will run on a different model within a year. text-davinci-003 will land on Azure. Something chat-shaped probably will too. If the model name is hard-coded through your codebase, each of those becomes a code change.

On Azure this is easier than on the OpenAI API, because you never call a model directly. You call a deployment that you named, and the deployment points at a model. The catch is that one resource can’t hold two deployments of the same model, so you can’t create a summarise deployment and a classify deployment that both point at text-davinci-002. One deployment per model, named plainly (davinci, curie), is what the service allows today.

So I put the indirection one layer up: tasks map to deployment names in configuration, not in code. Today summarise and classify both resolve to the davinci deployment. When you want to test a task on Curie, you deploy text-curie-001 in the same resource (a different model is allowed) and change one setting. When text-davinci-003 arrives, it’s a different model too, so it can sit beside text-davinci-002 in the same resource. If you ever need two configurations of the same model side by side, that means a second resource, and you only get two per region.

Here’s a small client that keeps the task-to-deployment mapping in environment variables, retries with backoff when per-deployment request limits throttle it, treats a filtered completion as a failure rather than an empty answer, and records token usage on every call. It uses the openai Python package (0.25.x) against API version 2022-12-01, which returns a usage block on completions:

import logging
import os
import time

import openai

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("aoai")

openai.api_type = "azure"
openai.api_base = os.environ["AZURE_OPENAI_ENDPOINT"]  # https://<your-resource-name>.openai.azure.com/
openai.api_version = "2022-12-01"
openai.api_key = os.environ["AZURE_OPENAI_KEY"]

# Map tasks to deployment names in configuration. A resource can't hold two
# deployments of the same model, so several tasks can share one deployment.
DEPLOYMENTS = {
    "summarise": os.getenv("AOAI_DEPLOYMENT_SUMMARISE", "davinci"),
    "classify": os.getenv("AOAI_DEPLOYMENT_CLASSIFY", "davinci"),
}


class ContentFilteredError(Exception):
    """The completion was withheld by the content filter."""


def complete(
    task: str,
    prompt: str,
    max_tokens: int = 256,
    temperature: float = 0.2,
    max_retries: int = 4,
) -> str:
    deployment = DEPLOYMENTS[task]
    for attempt in range(max_retries + 1):
        try:
            response = openai.Completion.create(
                engine=deployment,
                prompt=prompt,
                max_tokens=max_tokens,
                temperature=temperature,
            )
            break
        except openai.error.RateLimitError:
            if attempt == max_retries:
                raise
            delay = 2 ** attempt
            logger.warning("task=%s deployment=%s throttled, retrying in %ss", task, deployment, delay)
            time.sleep(delay)
        except openai.error.InvalidRequestError:
            # Over-length prompts and filtered prompts (HTTP 400) land here.
            # Retrying won't help.
            logger.exception("task=%s deployment=%s request rejected", task, deployment)
            raise

    usage = response["usage"]
    logger.info(
        "task=%s deployment=%s prompt_tokens=%s completion_tokens=%s",
        task,
        deployment,
        usage["prompt_tokens"],
        usage["completion_tokens"],
    )

    choice = response["choices"][0]
    if choice["finish_reason"] == "content_filter":
        # Filtered completions return HTTP 200 with no text.
        logger.warning("task=%s deployment=%s completion filtered", task, deployment)
        raise ContentFilteredError(f"{task}: completion withheld by content filter")
    return choice["text"].strip()


if __name__ == "__main__":
    ticket = "My invoice for December shows two charges for the same subscription."
    label = complete(
        "classify",
        "Classify the support ticket as Billing, Technical or Account.\n\n"
        f"Ticket: {ticket}\nCategory:",
        max_tokens=5,
        temperature=0,
    )
    print(label)

In production, send these log records to Application Insights or whatever telemetry you already run. The point is that every request records which deployment served it and how many tokens it used. Without that, you can’t answer “what would this cost on Curie?” with data.

Budget tokens before you send them

The 4,097-token limit on text-davinci-002 covers the prompt and the completion. If your prompt uses 3,900 tokens, you have 197 left for the answer, and the API will reject a request that asks for more. Count tokens client-side so you fail early and estimate cost.

OpenAI’s tiktoken library was released last month and includes the encodings these models use: p50k_base for the Davinci GPT-3.5 and Codex models, r50k_base for the GPT-3 family. Version 0.1.1 has no model-name lookup, so pick the encoding explicitly:

import tiktoken

# Model -> (encoding, max request tokens, USD per 1K tokens)
MODELS = {
    "text-davinci-002": ("p50k_base", 4097, 0.02),
    "text-curie-001": ("r50k_base", 2049, 0.002),
}


def plan_request(model: str, prompt: str, max_tokens: int) -> dict:
    encoding_name, limit, price_per_1k = MODELS[model]
    prompt_tokens = len(tiktoken.get_encoding(encoding_name).encode(prompt))
    total = prompt_tokens + max_tokens
    if total > limit:
        raise ValueError(
            f"{model}: {prompt_tokens} prompt + {max_tokens} completion tokens "
            f"exceeds the {limit}-token limit"
        )
    return {
        "prompt_tokens": prompt_tokens,
        "worst_case_cost_usd": round(total / 1000 * price_per_1k, 5),
    }


prompt = "Summarise the following incident report for an executive audience:\n\n" + "Text " * 500
for model in MODELS:
    try:
        print(model, plan_request(model, prompt, max_tokens=300))
    except ValueError as err:
        print(err)

The cost here is a worst case, because you’re billed for the tokens actually generated, not for max_tokens. It’s still the right number to budget with, and the habit of counting before you send is what stops a long document from failing in production.

What to do this month

If you’re planning a GPT-3.5 workload on Azure in early 2023, this is the sequence I’d follow:

Apply for access now. Approval is the longest lead time in the project.
Build on text-davinci-002, and don’t wait for text-davinci-003. Prompts that work on 002 will generally work on 003, and you’ll want a baseline to compare against when it arrives.
Map tasks to deployments in configuration. One deployment per model is the rule today, so let several tasks share the davinci deployment and change the mapping, not the code, when a new model lands.
Log tokens per deployment from day one, then test each narrow task against Curie.
Don’t promise “ChatGPT on Azure” to stakeholders. What exists is the completion API over GPT-3.5 models, which is plenty to build on, but it isn’t a chat product.

If you want the background on the GPT-3 family and Codex on Azure, I covered those in GPT-3 Models on Azure. The short version of this post: GPT-3.5 on Azure today means text-davinci-002 and code-davinci-002, and the design decisions that pay off are the ones that make the next model easy to adopt.