Serverless Inference

Serverless inference applies the serverless computing paradigm — pioneered by AWS Lambda for general-purpose functions — to machine learning model serving. The core principle is the same: the developer provides the code (or in this case, the model), and the platform handles all infrastructure concerns.

In a serverless inference platform, the lifecycle of a request looks like this:

1. Request arrives: Your application sends an inference request (prompt, parameters) to the platform's API endpoint.
2. Scheduling: The platform routes the request to a warm GPU that already has the model loaded in memory. If no warm GPUs are available, the platform provisions one (this is the "cold start" scenario).
3. Inference: The model processes the request on the GPU, generating tokens. The platform may batch multiple concurrent requests to improve throughput.
4. Response: The generated tokens are streamed or returned to your application.
5. Scale down: After a configurable idle period (typically 30 seconds to 5 minutes), the platform deallocates the GPU resources. If no more requests arrive, you scale to zero and stop paying.

The major serverless inference platforms each have distinct characteristics:

• AWS Bedrock: Amazon's managed LLM service. Offers proprietary models (Claude, Llama, Mistral, Cohere, Titan) on AWS infrastructure. Pricing is per-token with no minimum commitment. Deep integration with AWS services (IAM, VPC, CloudWatch, S3). Best for teams already invested in the AWS ecosystem.
• Google Vertex AI: Google's ML platform. Offers Gemini models natively, plus open models via Model Garden. Per-token pricing for Gemini; per-node-hour for custom model endpoints. Integration with Google Cloud services.
• Replicate: Developer-focused platform for running open-source models. Pay per second of GPU compute. Extensive model catalog with one-click deployment. Popular for experimentation and prototyping.
• Modal: Python-first serverless compute platform. Define your inference function in Python, and Modal handles containerization, GPU provisioning, and scaling. Per-second billing with fast cold starts. Excellent developer experience.
• Together AI: Focuses on open-source LLM inference. Per-token pricing competitive with self-hosting. Offers API-compatible endpoints for Llama, Mistral, Qwen, and other open models. Supports fine-tuned model deployment.
• Fireworks AI: High-performance inference platform. Offers some of the lowest per-token rates for open models, with emphasis on speed (low latency, high throughput). Custom model hosting available.
• Groq: Hardware-differentiated platform using custom LPU (Language Processing Unit) chips instead of NVIDIA GPUs. Offers extremely fast inference at competitive per-token prices. Limited model selection but very fast for supported models.

Each platform makes different trade-offs between price, performance, model selection, and ecosystem integration. CostHawk integrates with all major serverless inference platforms to provide unified cost tracking and comparison.

Choosing between serverless inference, dedicated GPU instances, and managed APIs requires evaluating multiple dimensions beyond just per-token cost. Here is a comprehensive comparison:

The optimal strategy evolves as your workload grows. Many teams follow this progression:

1. Prototype phase: Use managed APIs for fastest iteration and best model quality.
2. Growth phase: Migrate non-frontier workloads to serverless inference for 50–70% cost reduction.
3. Scale phase: Move highest-volume, most stable workloads to dedicated GPUs for an additional 40–60% reduction. Keep bursty and low-volume workloads on serverless.

Serverless inference platforms use several pricing models, each with distinct economics. Understanding these models is essential for cost prediction and optimization:

1. Per-token pricing: Charged by the number of input and output tokens processed, identical to how OpenAI and Anthropic price their APIs. This is the most transparent model for LLM workloads because cost scales linearly with usage.

2. Per-second GPU billing: Charged by the number of seconds your model runs on GPU hardware. This model is used by Replicate, Modal, and some other platforms. The advantage is full transparency into hardware utilization; the complexity is that cost per token varies depending on model efficiency and batch size.

3. Hybrid pricing (provisioned + per-token): Some platforms offer a reserved baseline capacity at a lower per-token rate, with on-demand scaling at a higher rate for bursts. AWS Bedrock's Provisioned Throughput is an example: you commit to a fixed throughput level (measured in model units) for a term, getting guaranteed capacity and lower rates. This model suits workloads with a predictable baseline plus occasional spikes.

4. Free tiers and credits: Most platforms offer free tiers or new-user credits. Together AI provides $25 in free credits. Replicate offers free CPU inference and credits for new accounts. Google Vertex AI includes a free tier for Gemini models. These are useful for experimentation but rarely sufficient for production workloads.

Cost prediction tips:

• For per-token platforms, cost prediction is straightforward: multiply your estimated token volume by the per-token rate.
• For per-second platforms, benchmark your model's throughput (tokens/second) on the target GPU, then divide your token volume by throughput to estimate GPU-seconds needed.
• Always account for cold starts in per-second billing — loading a 70B model takes 30–60 seconds of GPU time, which is billed even though no inference is happening.

Cold start is the single biggest operational challenge in serverless inference. When a request arrives and no GPU has the model loaded in memory, the platform must provision a GPU, load the model from storage, and initialize the inference engine before processing the first token. This delay — the cold start latency — directly impacts user experience and cost.

Cold start durations by model size:

Cold start latency comes from three phases:

1. GPU provisioning: Allocating a GPU from the platform's pool. Typically 2–10 seconds depending on availability.
2. Model loading: Transferring model weights from storage (typically NVMe or network storage) to GPU VRAM. This is the dominant component — a 70B FP16 model is 140 GB, and even at 10 GB/s transfer speed, loading takes 14 seconds. Quantized models load proportionally faster.
3. Engine initialization: Starting the inference engine (vLLM, TGI, etc.), compiling CUDA kernels, and warming up caches. Typically 2–5 seconds.

Mitigation strategies:

• Keep-warm configuration: Most platforms let you configure a minimum number of warm replicas — GPUs that stay provisioned with the model loaded, even during idle periods. This eliminates cold starts but reintroduces idle cost. The trade-off is worth it for latency-sensitive applications. Modal, for example, lets you set keep_warm=1 to maintain one warm GPU at all times.
• Predictive scaling: Some platforms analyze traffic patterns and pre-warm GPUs before anticipated demand increases. If your traffic predictably rises at 9 AM, the platform begins provisioning at 8:55 AM.
• Smaller model variants: A quantized 7B model cold-starts in 5 seconds vs. 60 seconds for a 70B model. If latency is critical and a smaller model meets your quality bar, the cold-start benefit is substantial.
• Request queuing: For non-interactive workloads (batch processing, asynchronous tasks), queue requests and tolerate the cold-start delay. The first request takes 30–60 seconds; subsequent requests on the now-warm GPU are fast.
• Hybrid routing: Route latency-sensitive requests to a keep-warm serverless endpoint or a managed API, and route latency-tolerant requests to a scale-to-zero serverless endpoint. This minimizes both cost and latency.

CostHawk tracks cold-start events in its latency analytics, showing you how frequently cold starts occur, their duration, and their cost impact. This data helps you decide whether to configure keep-warm replicas and how many — balancing the idle cost of warm replicas against the latency cost of cold starts.

Serverless inference is not the cheapest option at every volume level, and it is not the most performant for every workload. It excels in specific scenarios where its unique characteristics — zero idle cost, automatic scaling, and operational simplicity — provide the most value:

1. Variable and unpredictable workloads: If your inference volume fluctuates significantly — high during business hours, low overnight, spiking during marketing campaigns, quiet on weekends — serverless inference's scale-to-zero capability means you only pay for actual usage. A dedicated GPU running at 20% average utilization effectively costs 5x its per-token rate. Serverless eliminates this waste entirely. Calculate your average utilization honestly: if a dedicated GPU would be utilized less than 40% of the time, serverless is almost certainly cheaper.

2. Early-stage and experimental workloads: When you are evaluating models, testing new features, or building prototypes, serverless inference provides instant access without infrastructure setup. Deploy a model, test it with real traffic for a week, analyze the results, and shut it down — total cost might be $50 instead of the $929/month minimum for a dedicated GPU. This low commitment enables faster experimentation and better model selection decisions.

3. Low to medium volume production workloads: The sweet spot for serverless inference is 50M–2B tokens per month. Below 50M, managed APIs are competitive and operationally simpler (no model selection or deployment). Above 2B, dedicated GPUs become cheaper even accounting for operational overhead. In the 50M–2B range, serverless offers the best combination of cost and simplicity.

4. Multi-model architectures: If your application uses multiple specialized models (a small model for classification, a medium model for summarization, a large model for complex reasoning), serverless inference shines because each model scales independently. A classification model handling 1,000 requests per minute and a reasoning model handling 10 requests per minute can each scale to exactly the resources they need. With dedicated GPUs, you would need at least one GPU per model regardless of volume.

5. Geographic distribution: Serverless platforms with multi-region deployment can serve requests from the nearest data center, reducing latency for global applications. Running dedicated GPUs in multiple regions multiplies your infrastructure cost; serverless distributes this across shared platform capacity.

When serverless does NOT make sense:

• Very high, stable volume: If you consistently process 5B+ tokens per month with predictable demand, dedicated GPUs are significantly cheaper. The serverless per-token premium adds up at scale.
• Ultra-low latency requirements: If every request must respond within 200ms including TTFT, cold starts are unacceptable and you need always-warm infrastructure (dedicated GPUs or keep-warm serverless, which approaches dedicated GPU cost).
• Custom infrastructure requirements: If you need custom CUDA kernels, non-standard model architectures, or hardware configurations that serverless platforms do not support, self-hosting is the only option.

Serverless inference introduces unique monitoring challenges compared to both APIs and dedicated GPUs. With APIs, cost is purely a function of tokens — easy to predict and track. With dedicated GPUs, cost is fixed hourly — easy to budget. Serverless inference combines variable per-token costs with cold-start costs, keep-warm costs, and platform-specific surcharges that require more nuanced monitoring.

Key metrics to track:

• Cost per request: The total cost of each inference request, including both inference compute and any cold-start overhead. CostHawk computes this by combining the platform's billing data with per-request token counts and latency measurements.
• Cold start frequency: The percentage of requests that trigger a cold start. If this exceeds 5–10% for a production endpoint, consider adding keep-warm replicas. Each cold start costs GPU-seconds (billed) and adds latency (degrading user experience).
• Cold start cost: The total cost attributable to cold starts — GPU time spent loading models rather than processing tokens. For a 70B model with 60-second cold starts at $0.0014/second (A100 rate), each cold start costs $0.084. At 100 cold starts per day, that is $8.40/day or $252/month in pure overhead.
• Effective cost per token: The blended cost including inference time, cold-start overhead, and keep-warm costs, divided by total tokens processed. This is the metric to compare against API and dedicated GPU alternatives.
• Utilization patterns: Time-series view of inference volume showing daily, weekly, and seasonal patterns. This data informs decisions about keep-warm configuration, predictive scaling, and potential migration to dedicated GPUs for stable base-load.
• Per-model cost breakdown: If you run multiple models on serverless, track each independently. A small classification model at $0.10/MTok and a large reasoning model at $0.54/MTok have very different economics, and optimization strategies differ.

CostHawk serverless monitoring:

CostHawk integrates with serverless inference platforms via API to ingest billing data, request logs, and performance metrics. The dashboard provides:

• Unified cost view across all serverless platforms, APIs, and GPU infrastructure
• Cold-start analysis with frequency, duration, and cost attribution
• Savings recommendations: "Your Together AI Llama 3 70B endpoint processed 3.2B tokens last month at $1,728. A dedicated A100 on Lambda Labs would cost $929/month — a potential 46% saving."
• Anomaly detection tuned for serverless patterns — distinguishing between normal traffic variability and genuine cost anomalies
• Budget alerts that account for the variability of serverless pricing — setting thresholds based on trailing averages rather than fixed amounts

The monitoring strategy should be tailored to your deployment model. For per-token platforms (Together, Fireworks), monitor token volume and cost per token. For per-second platforms (Replicate, Modal), monitor GPU-seconds consumed and effective tokens per GPU-second. CostHawk normalizes these different billing models into a consistent cost-per-token metric, enabling direct comparison across platforms and deployment models. This normalization is essential for identifying the most cost-effective platform for each workload and for tracking the impact of optimization efforts over time.