OpenTelemetry

OpenTelemetry is the merger of two earlier CNCF projects — OpenTracing and OpenCensus — unified in 2019 to create a single, definitive observability standard. It reached general availability for traces in 2023 and has since become the second most active CNCF project after Kubernetes, with contributions from Google, Microsoft, Amazon, Splunk, Datadog, and hundreds of other organizations.

OpenTelemetry provides four core components:

1. API: A set of interfaces for creating telemetry in your application code. The API is intentionally thin — calling it with no SDK configured produces no overhead (a "no-op" implementation). This means library authors can instrument their code with the OTel API without forcing users to adopt any specific observability stack.
2. SDK: The implementation that processes telemetry created via the API. The SDK handles sampling, batching, and exporting. You configure it at application startup with your desired exporters, sampling rate, and resource attributes.
3. OTLP (OpenTelemetry Protocol): A standardized wire protocol for transmitting telemetry data over gRPC or HTTP. OTLP is supported natively by all major observability platforms, eliminating the need for vendor-specific exporters in most cases.
4. Collector: An optional standalone service that receives telemetry data, processes it (filtering, sampling, enriching, transforming), and exports it to one or more backends. The Collector is deployed as a sidecar, DaemonSet, or standalone service and acts as a telemetry pipeline between your applications and your observability backends.

The three signal types that OpenTelemetry supports are:

• Traces: Distributed traces that follow a request through multiple services and operations. Each trace consists of spans — units of work with a start time, duration, status, and attributes. For LLM applications, a span typically represents one inference call.
• Metrics: Numerical measurements collected over time — counters (total tokens consumed), histograms (latency distribution), and gauges (current active requests). Metrics are lower-cardinality than traces and are ideal for dashboards and alerting.
• Logs: Structured log records that can be correlated with traces via trace IDs and span IDs. OTel logs provide the narrative context that explains what happened during a trace.

For LLM applications, all three signals are relevant: traces capture per-request detail (model, tokens, cost, latency), metrics capture aggregate trends (total spend, average latency, error rate), and logs capture prompt/response content and evaluation results. Together, they provide complete observability for AI-powered applications.

Instrumenting LLM API calls with OpenTelemetry involves creating spans that capture the full context of each inference request. The instrumentation can be done manually, using auto-instrumentation libraries, or through LLM-specific OTel libraries like OpenLLMetry and Traceloop.

Manual instrumentation example (TypeScript with OpenAI):

import { trace, SpanKind, SpanStatusCode } from '@opentelemetry/api'
import OpenAI from 'openai'

const tracer = trace.getTracer('llm-service', '1.0.0')
const openai = new OpenAI()

async function chatCompletion(prompt: string) {
  return tracer.startActiveSpan('gen_ai.chat', {
    kind: SpanKind.CLIENT,
    attributes: {
      'gen_ai.system': 'openai',
      'gen_ai.request.model': 'gpt-4o',
      'gen_ai.request.max_tokens': 1000,
      'gen_ai.request.temperature': 0.7,
    }
  }, async (span) => {
    try {
      const response = await openai.chat.completions.create({
        model: 'gpt-4o',
        messages: [{ role: 'user', content: prompt }],
        max_tokens: 1000,
        temperature: 0.7,
      })

      const usage = response.usage!
      span.setAttributes({
        'gen_ai.response.model': response.model,
        'gen_ai.usage.input_tokens': usage.prompt_tokens,
        'gen_ai.usage.output_tokens': usage.completion_tokens,
        'gen_ai.usage.total_tokens': usage.total_tokens,
        'gen_ai.response.finish_reason':
          response.choices[0].finish_reason,
      })
      span.setStatus({ code: SpanStatusCode.OK })
      return response
    } catch (error) {
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: (error as Error).message
      })
      throw error
    } finally {
      span.end()
    }
  })
}

This instrumentation creates a span for every LLM call with standardized attributes for the model, token counts, and finish reason. The span is automatically correlated with the parent trace, so you can see the LLM call in the context of the broader request that triggered it.

Auto-instrumentation with OpenLLMetry:

For teams that want instrumentation without modifying application code, the OpenLLMetry project provides drop-in auto-instrumentation for popular LLM libraries:

import { OpenLLMetry } from 'openllmetry'

// Initialize once at application startup
OpenLLMetry.init({
  exporter: new OTLPTraceExporter({
    url: 'https://otel.costhawk.com/v1/traces'
  }),
  appName: 'my-ai-app',
})

OpenLLMetry automatically patches the OpenAI, Anthropic, Google, Cohere, and other LLM client libraries to emit OTel spans with the correct gen_ai.* attributes. No code changes are needed beyond the initialization call.

Key instrumentation best practices for LLM applications:

• Always capture gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — these are the foundation for cost calculation
• Add custom attributes for business context: app.feature, app.user_id, app.project — these enable cost attribution
• Record the actual model returned by the API (not just the requested model), since providers may route to different model versions
• Use span events to record prompt and response content when needed for debugging, but be mindful of payload size and PII considerations

Semantic conventions are standardized attribute names and values that ensure telemetry data is consistent and interoperable across different instrumentation libraries and observability backends. The OpenTelemetry project maintains an evolving set of semantic conventions for Generative AI under the gen_ai.* namespace.

As of early 2026, the key semantic conventions for LLM observability are:

These conventions are complemented by cost-specific attributes that CostHawk and other cost monitoring tools recognize:

The standardization these conventions provide is transformative for the LLM observability ecosystem. When every instrumentation library records gen_ai.usage.input_tokens with the same semantics, any observability backend can build dashboards, alerts, and analytics on top of that data without custom parsing or normalization. If you instrument your application with these conventions today, you can switch or add observability backends tomorrow without reinstrumenting.

The conventions are still evolving — the GenAI working group within the OTel project meets regularly to refine and expand the attribute set. Notable proposals in progress include attributes for tool/function calling (gen_ai.tool.name, gen_ai.tool.parameters), multi-turn conversation tracking (gen_ai.conversation.id), and quality evaluation scores (gen_ai.eval.score). CostHawk tracks these conventions as they evolve and updates its ingestion pipeline to support new attributes as they are standardized.

Teams building LLM applications face a choice between instrumenting with OpenTelemetry (open standard) or with proprietary SDKs from observability vendors. Here is a detailed comparison:

The recommendation for most teams in 2026 is to adopt a hybrid approach: use OpenTelemetry as the core instrumentation layer for all telemetry, then layer on specialized tools for LLM-specific capabilities that OTel does not yet cover (prompt management, evaluation suites, fine-tuning workflows). This gives you the vendor neutrality and correlation benefits of OTel while still accessing specialized LLM tooling.

For cost monitoring specifically, OpenTelemetry is the clear winner. Cost data needs to be correlated with application context (which feature? which user? which team?) and aggregated across multiple providers. OTel's distributed tracing and flexible attribute system make this natural, while proprietary SDKs typically only see their own platform's data.

Using OpenTelemetry to track LLM costs requires capturing token usage per request, enriching it with pricing data, and exporting the results to a cost monitoring backend. Here is a complete implementation pattern:

Step 1: Configure the OTel SDK with cost-aware span processing.

import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { Resource } from '@opentelemetry/resources'
import {
  ATTR_SERVICE_NAME,
  ATTR_SERVICE_VERSION,
} from '@opentelemetry/semantic-conventions'

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'my-ai-service',
    [ATTR_SERVICE_VERSION]: '2.1.0',
    'deployment.environment': 'production',
    'team.name': 'ml-platform',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'https://otel.costhawk.com/v1/traces',
    headers: {
      'x-costhawk-api-key': process.env.COSTHAWK_API_KEY!,
    },
  }),
})

sdk.start()

Step 2: Create a cost-enriching span processor.

import { SpanProcessor, ReadableSpan } from '@opentelemetry/sdk-trace-base'

const MODEL_PRICING: Record<string, { input: number; output: number }> = {
  'gpt-4o':              { input: 2.50,  output: 10.00 },
  'gpt-4o-mini':         { input: 0.15,  output: 0.60  },
  'claude-3.5-sonnet':   { input: 3.00,  output: 15.00 },
  'claude-3.5-haiku':    { input: 0.80,  output: 4.00  },
  'gemini-2.0-flash':    { input: 0.10,  output: 0.40  },
}

class CostEnrichmentProcessor implements SpanProcessor {
  onEnd(span: ReadableSpan): void {
    const model = span.attributes['gen_ai.response.model'] as string
    const inputTokens = span.attributes['gen_ai.usage.input_tokens'] as number
    const outputTokens = span.attributes['gen_ai.usage.output_tokens'] as number

    if (model && inputTokens && outputTokens) {
      const pricing = MODEL_PRICING[model]
      if (pricing) {
        const inputCost = (inputTokens / 1_000_000) * pricing.input
        const outputCost = (outputTokens / 1_000_000) * pricing.output
        // Note: attributes set in onEnd may require a custom exporter
        // to include in the exported span. Some implementations use
        // onStart + deferred enrichment instead.
      }
    }
  }
  onStart(): void {}
  forceFlush(): Promise<void> { return Promise.resolve() }
  shutdown(): Promise<void> { return Promise.resolve() }
}

Step 3: Add business context attributes to enable cost attribution.

The real power of OTel for cost tracking emerges when you add application-level attributes that enable grouping and attribution:

span.setAttributes({
  'app.feature': 'document-summarization',
  'app.customer_id': 'cust_abc123',
  'app.project': 'enterprise-chatbot',
  'app.environment': 'production',
  'app.api_key_id': 'key_xyz789',
})

These attributes let CostHawk slice cost data by feature, customer, project, and environment — answering questions like "Which feature costs the most?" and "Which customer is driving the biggest spend increase?" that provider dashboards cannot answer.

Step 4: Deploy an OTel Collector for pipeline flexibility.

For production deployments, route telemetry through an OTel Collector rather than exporting directly from your application. The Collector can sample, filter, batch, and fan out telemetry to multiple backends. A typical Collector configuration sends traces to both Jaeger (for debugging) and CostHawk (for cost monitoring) while filtering out low-value spans to control ingestion volume and cost.

CostHawk provides native OpenTelemetry ingestion, accepting OTLP data over both gRPC and HTTP. This means any application instrumented with OpenTelemetry can send cost telemetry to CostHawk without a custom integration — just configure the OTLP exporter to point to CostHawk's endpoint.

Integration architecture:

┌──────────────────┐      OTLP/HTTP       ┌──────────────────┐
│  Your Application │  ─────────────────▶  │  CostHawk OTLP   │
│  (OTel SDK)       │                      │  Endpoint         │
└──────────────────┘                      └────────┬─────────┘
                                                    │
           ┌────────────────────────────────────────┤
           │                                        │
           ▼                                        ▼
┌──────────────────┐                    ┌──────────────────┐
│  Cost Attribution │                    │  Anomaly          │
│  Engine           │                    │  Detection        │
│  (per-feature,    │                    │  (baseline +      │
│   per-customer,   │                    │   deviation)      │
│   per-model)      │                    │                   │
└────────┬─────────┘                    └────────┬─────────┘
         │                                        │
         ▼                                        ▼
┌──────────────────┐                    ┌──────────────────┐
│  Dashboard &      │                    │  Webhook Alerts   │
│  Reports          │                    │  (Slack, PD, etc) │
└──────────────────┘                    └──────────────────┘

What CostHawk extracts from OTel data:

• Token usage: gen_ai.usage.input_tokens and gen_ai.usage.output_tokens are used to calculate per-request cost using CostHawk's pricing database (which tracks the latest rates for 200+ models across all major providers).
• Model attribution: gen_ai.response.model identifies which model was used, enabling cost-per-model breakdowns and model routing optimization recommendations.
• Business context: Custom attributes like app.feature, app.project, app.customer_id, and app.environment enable multi-dimensional cost attribution that provider dashboards cannot provide.
• Latency: Span duration reveals per-request latency, enabling cost-vs-latency analysis (are you paying more for faster responses? is a cheaper model acceptable given the latency requirements?).
• Error rates: Span status codes identify failed requests that consumed tokens but did not deliver value — wasted spend that should be minimized.

Dual-path monitoring: CostHawk supports both OTel-based monitoring and wrapped-key-based monitoring simultaneously. Teams can use wrapped keys for immediate, zero-instrumentation cost tracking and add OTel instrumentation incrementally for deeper attribution and correlation. The two data sources are merged in the CostHawk dashboard, providing a unified view regardless of how the data was collected.

Collector-based deployment: For teams already running an OTel Collector, add CostHawk as an additional exporter in the Collector configuration. This requires zero changes to application code — the Collector fans out existing telemetry to CostHawk alongside your existing observability backends. This is the lowest-friction integration path for organizations that already have OTel infrastructure in place.