Setting Up LLM Observability Without a Vendor
Build a vendor-neutral observability stack for LLM applications using traces, metrics, structured logs, OpenTelemetry, and your own dashboards.An LLM application fails in production.
The API returned 200 OK.
Your infrastructure dashboard is green.
CPU looks normal.
Memory looks normal.
Yet the user waited 18 seconds and received a poor answer.
What happened?
Maybe the model was slow.
Maybe the prompt became enormous.
Maybe retrieval returned irrelevant context.
Maybe an agent called the same tool four times.
Maybe a retry silently doubled the cost.
Traditional application monitoring doesn't always answer these questions.
That's where LLM observability comes in.
And you don't necessarily need to send every prompt, response, and trace to a specialized observability vendor to get started.
In this guide, we'll build a practical LLM observability setup around open telemetry concepts so that we can understand:
requests
model calls
latency
tokens
estimated cost
errors
retrieval
tool calls
end-to-end traces
while keeping control over where our telemetry goes.
What Is LLM Observability?
LLM observability is the ability to understand what happens inside an LLM-powered application by collecting and correlating telemetry.
For a basic application:
User
│
▼
Application
│
▼
LLM
│
▼
Response
we may want to know:
Which model handled the request?
How long did generation take?
How many input tokens were consumed?
How many output tokens were generated?
Did the request fail?
How much did it approximately cost?
For an agent or RAG application, the execution path becomes more complicated:
User
│
▼
Agent
│
├── LLM Call
│
├── Vector Search
│
├── LLM Call
│
├── Tool Call
│
├── Tool Result
│
└── Final LLM Call
│
▼
Response
Now observability must explain the entire execution, not just the final API request.
Monitoring vs. Observability
These terms are related but aren't identical.
Monitoring tells you:
p95 latency = 4.8 seconds
error rate = 2.1%
requests/minute = 1,200
Observability helps answer:
Why did p95 latency increase?
Perhaps traces reveal:
Retrieval 180 ms
First LLM call 1,420 ms
Tool call 3,900 ms ← bottleneck
Second LLM call 1,610 ms
The metric tells us something is wrong.
The trace helps explain where it happened.
A useful LLM stack therefore combines:
Metrics + Traces + Logs + Evaluation Signals
Why Build It Without a Specialized Vendor?
Specialized LLM observability platforms can be useful.
But building the basic telemetry layer yourself has several advantages.
Portability
Your instrumentation doesn't have to depend on one dashboard provider.
Data control
Prompts and responses may contain:
customer information
source code
internal documents
financial information
credentials accidentally supplied by users
personal data
Keeping telemetry within infrastructure you control may simplify some privacy and security requirements.
Existing infrastructure
Your organization may already operate:
OpenTelemetry
Prometheus
Grafana
Jaeger
Tempo
Loki
Elasticsearch
Adding LLM signals to the same stack may be easier than creating another monitoring silo.
Understanding
Building the first version yourself forces you to decide which signals actually matter.
That's valuable even if you eventually adopt a specialized platform.
The Architecture We're Building
Our initial architecture will look like this:
LLM Application
│
│ telemetry
▼
OpenTelemetry SDK
│
▼
OTel Collector
/ | \
/ | \
▼ ▼ ▼
Traces Metrics Logs
│ │ │
▼ ▼ ▼
Tempo Prometheus Loki
\ | /
\ | /
▼ ▼ ▼
Grafana
You don't have to use these exact backends.
That's one of the advantages of the architecture.
The application emits standardized telemetry.
The backend is replaceable.
Step 1: Decide What You Need to Observe
Don't start by logging everything.
Start with questions.
For example:
Why is this request slow?
Which model is consuming the most tokens?
Which prompt version causes the most failures?
How often are tools failing?
How much does one successful request cost?
Which retrieval step adds the most latency?
Those questions define your telemetry requirements.
For a first version, collect:
Request ID
Trace ID
Model
Operation
Latency
Input tokens
Output tokens
Finish reason
Status
Error type
Prompt version
Application version
Environment
For agents, add:
Tool name
Tool duration
Tool result status
Agent turns
Retry count
For RAG:
Retriever
Retrieval duration
Number of documents
Embedding model
Top-k
Context size
That's already enough to investigate many production problems.
Step 2: Don't Log Full Prompts by Default
This deserves its own step.
The easiest observability implementation is:
logger.info(prompt)
logger.info(response)
It's also potentially dangerous.
Prompts may contain:
PII
customer records
internal documents
source code
access tokens
medical information
financial information
Instead, begin with metadata:
{
"trace_id": "8c94...",
"model": "example-model",
"prompt_version": "support-v12",
"input_tokens": 812,
"output_tokens": 184,
"duration_ms": 1420,
"status": "ok"
}
If you later decide that prompt capture is necessary, make it an explicit feature with:
redaction
sampling
access controls
retention policies
encryption
audit logging
Observability shouldn't become a new data-leakage system.
Step 3: Start With OpenTelemetry
OpenTelemetry gives us a vendor-neutral model for telemetry.
Instead of wiring our application directly to one backend:
Application
│
▼
Vendor-specific SDK
│
▼
Vendor
we can use:
Application
│
▼
OpenTelemetry
│
▼
OTLP
│
├── Backend A
├── Backend B
└── Self-hosted stack
This separation is one of the most important design choices in a vendor-neutral observability system.
Step 4: Instrument the Application
Let's use Python for a minimal example.
Install OpenTelemetry packages:
pip install \
opentelemetry-api \
opentelemetry-sdk \
opentelemetry-exporter-otlp
Create a tracer:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import (
BatchSpanProcessor
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
OTLPSpanExporter
)
resource = Resource.create({
"service.name": "llm-api",
"service.version": "1.0.0",
"deployment.environment": "production"
})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint="http://otel-collector:4317",
insecure=True
)
)
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm-observability")
Now the application can create spans.
Step 5: Trace an LLM Call
Wrap model execution in a span.
def call_llm(client, model, messages):
with tracer.start_as_current_span(
"llm.generate"
) as span:
span.set_attribute(
"gen_ai.request.model",
model
)
response = client.generate(
model=model,
messages=messages
)
span.set_attribute(
"gen_ai.usage.input_tokens",
response.usage.input_tokens
)
span.set_attribute(
"gen_ai.usage.output_tokens",
response.usage.output_tokens
)
return response
Now a model request isn't just:
POST /chat
It becomes part of a distributed trace.
Step 6: Trace the Whole Request
The model call should normally be a child of the user request.
def handle_request(question):
with tracer.start_as_current_span(
"answer_question"
):
context = retrieve_context(question)
answer = call_llm(
client,
MODEL,
build_messages(
question,
context
)
)
return answer
The resulting trace might look like:
answer_question 2.83s
│
├── retrieve_context 0.31s
│
└── llm.generate 2.47s
This is where tracing becomes much more useful than disconnected logs.
You can see where the time went.
Step 7: Instrument Retrieval
For a RAG system:
def retrieve_context(query):
with tracer.start_as_current_span(
"rag.retrieve"
) as span:
results = vector_store.search(
query,
top_k=5
)
span.set_attribute(
"rag.top_k",
5
)
span.set_attribute(
"rag.documents.returned",
len(results)
)
return results
Now the trace becomes:
answer_question
│
├── rag.retrieve
│
└── llm.generate
Later you might add:
embedding generation
reranking
document filtering
context construction
as separate spans.
Step 8: Instrument Tool Calls
Agent applications need another layer.
Suppose the model calls:
get_order_status
Trace it separately:
def execute_tool(name, arguments):
with tracer.start_as_current_span(
f"tool.{name}"
) as span:
span.set_attribute(
"tool.name",
name
)
try:
result = tool_registry[name](
**arguments
)
span.set_attribute(
"tool.status",
"success"
)
return result
except Exception as error:
span.record_exception(error)
span.set_attribute(
"tool.status",
"error"
)
raise
Now an agent trace might show:
agent.run 7.4s
│
├── llm.generate 1.8s
│
├── tool.get_order_status 3.2s
│
└── llm.generate 2.1s
Immediately we know the model wasn't responsible for most of the delay.
The internal tool was.
Step 9: Add Metrics
Traces explain individual requests.
Metrics reveal patterns across thousands of them.
At minimum, track:
LLM request count
LLM request duration
Input tokens
Output tokens
Errors
Tool duration
Tool errors
Useful dashboards might include:
Requests/min
p50 latency
p95 latency
p99 latency
Token usage/min
Average tokens/request
Errors by model
Latency by model
Tool failure rate
For production AI systems, observability becomes part of the broader deployment lifecycle rather than something added only after incidents. This is why a mature generative AI development process typically combines testing, deployment, monitoring, guardrails, and ongoing optimization.
Step 10: Estimate Cost Yourself
If you know a model's current token pricing, cost calculation is straightforward.
Conceptually:
def estimate_cost(
input_tokens,
output_tokens,
input_price,
output_price
):
return (
input_tokens / 1_000_000
* input_price
) + (
output_tokens / 1_000_000
* output_price
)
Then attach the result:
span.set_attribute(
"llm.estimated_cost_usd",
cost
)
You can now answer:
What does one request cost?
Which endpoint costs the most?
Which model consumes the most budget?
Did the latest prompt increase token usage?
Keep pricing configuration outside application code so it can be updated without changing instrumentation.
Step 11: Track Prompt Versions, Not Necessarily Prompt Text
Suppose production currently uses:
support-agent-v17
Record:
span.set_attribute(
"app.prompt.version",
"support-agent-v17"
)
rather than storing the entire system prompt on every request.
Now you can compare:
Prompt v15
p95 latency: 2.8s
avg input: 720 tokens
Prompt v16
p95 latency: 3.0s
avg input: 890 tokens
Prompt v17
p95 latency: 4.4s
avg input: 1,620 tokens
Suddenly a latency regression has an obvious clue.
Prompt versioning gives you useful correlation without automatically storing sensitive content.
Step 12: Add Structured Logs
You still need logs.
Just make them structured.
Instead of:
LLM failed!!!
write:
{
"level": "error",
"event": "llm_request_failed",
"trace_id": "83a21...",
"model": "example-model",
"prompt_version": "support-v17",
"error_type": "timeout",
"retry_count": 2
}
The most important field is often:
trace_id
It connects logs with traces.
A useful debugging workflow becomes:
Alert
↓
Metric
↓
Trace
↓
Span
↓
Correlated logs
Step 13: Deploy an OpenTelemetry Collector
Don't have every application export directly to storage.
Put a collector between them.
Applications
│
│ OTLP
▼
+---------------------+
| OpenTelemetry |
| Collector |
+---------------------+
│
├── traces
├── metrics
└── logs
The collector gives you a central place for:
batching
filtering
sampling
redaction
routing
retries
export
Your applications only need to know where the collector lives.
Step 14: Configure the Collector
A simplified configuration might look like:
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
exporters:
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers:
- otlp
processors:
- batch
exporters:
- otlp/tempo
You can later add separate pipelines for:
metrics
logs
and additional processors for filtering or sampling.
Step 15: Build a Local Stack
A useful self-hosted stack is:
OpenTelemetry Collector
│
├── Tempo → traces
├── Prometheus → metrics
└── Loki → logs
│
▼
Grafana
Another option is:
OpenTelemetry
│
▼
Jaeger
for a simpler tracing-focused setup.
The important part isn't the exact backend.
It's keeping instrumentation separate from storage.
Step 16: Create Your First Dashboard
Don't build 40 charts.
Start with a small operational dashboard.
Traffic
Requests/minute
Requests by model
Latency
p50
p95
p99
Tokens
Input tokens/min
Output tokens/min
Tokens/request
Reliability
Error rate
Timeout rate
Retry rate
Cost
Estimated cost/hour
Estimated cost/request
Cost by model
Agent Tools
Tool calls
Tool latency
Tool failures
That's enough to answer many operational questions.
Step 17: Add Alerts
Dashboards require someone to look at them.
Alerts tell you when something changes.
For example:
p95 latency > 8 seconds
for 10 minutes
or:
LLM error rate > 5%
or:
Tool failure rate > 10%
Token anomalies can also be useful:
Average input tokens increased
more than 40% from baseline
This can detect:
prompt changes
retrieval explosions
conversation-history growth
agent loops
before the cost increase becomes obvious on the monthly bill.
Step 18: Add Sampling Before Volume Explodes
Keeping every trace forever is rarely necessary.
A simple strategy might be:
Successful request → sample 5%
Error → keep 100%
Very slow request → keep 100%
Critical workflow → keep 100%
This is much more useful than blindly sampling every request at the same rate.
The interesting traces are often the unusual ones.
Step 19: Redact at the Collector
Suppose telemetry accidentally contains:
user.email
authorization.header
customer.phone
You don't necessarily want those values reaching storage.
A useful architecture is:
Application
│
▼
OTel Collector
│
├── Remove sensitive attributes
├── Redact values
├── Sample
└── Route
│
▼
Telemetry Storage
This creates a centralized privacy control point.
Still, prevention at the application layer is better than relying exclusively on downstream redaction.
Step 20: Observe RAG as a Pipeline
A common mistake is treating RAG as one LLM request.
It's actually a pipeline:
Question
│
▼
Embedding
│
▼
Vector Search
│
▼
Reranking
│
▼
Context Construction
│
▼
LLM
Trace these separately:
rag.request
│
├── embedding.create
├── vector.search
├── reranker.rank
├── context.build
└── llm.generate
Now when RAG quality deteriorates, you can investigate the individual stages.
Step 21: Observe Agents as Trees
Agent executions are even more interesting.
A trace could look like:
agent.run
│
├── llm.generate
│
├── tool.search_customer
│
├── llm.generate
│
├── tool.get_orders
│
├── llm.generate
│
└── response
From that single trace you can calculate:
number of LLM calls
number of tool calls
total tokens
total latency
tool latency
retries
estimated cost
This makes traces especially valuable for debugging agentic systems.
Step 22: Detect Agent Loops
Imagine this trace:
agent.run
│
├── llm.generate
├── tool.search
├── llm.generate
├── tool.search
├── llm.generate
├── tool.search
├── llm.generate
├── tool.search
├── ...
Your application is probably stuck.
Create metrics such as:
agent_turns
tool_calls_per_run
llm_calls_per_run
Then alert on abnormal values.
For example:
agent_turns > 12
or:
same tool called > 5 times
Observability can reveal problems that ordinary API monitoring completely misses.
Step 23: Add Quality Signals
Infrastructure telemetry answers:
Was it fast?
Did it fail?
How many tokens did it use?
But an LLM can be:
fast
cheap
error-free
and still give a terrible answer.
So eventually add evaluation signals:
groundedness
correctness
user feedback
task completion
hallucination rate
retrieval quality
Then correlate them with traces.
For example:
Trace ID: abc123
Latency: 2.8s
Cost: $0.009
Tokens: 1,240
Groundedness: 0.91
User rating: positive
Now you're observing both system performance and AI behavior.
Step 24: Define LLM SLOs
Traditional systems have SLOs.
LLM systems should too.
For example:
Reliability
99.5% successful model calls
Latency
95% of requests complete < 6 seconds
Cost
95% of requests cost < $0.03
Agent behavior
99% complete within 8 turns
Quality
groundedness score > agreed threshold
The exact values depend on the application.
The important part is defining acceptable behavior before an incident.
For larger production AI systems, this operating layer should be planned alongside model deployment, security, evaluation, and lifecycle management. SDLC Corp's AI development services similarly describe deployment and monitoring as part of the broader AI lifecycle rather than treating monitoring as a separate afterthought.
Step 25: Build a Debugging Workflow
The biggest benefit of observability isn't the dashboard.
It's reducing the time between:
Something is wrong.
and:
We know why.
A useful workflow looks like:
Alert
│
▼
Find affected metric
│
▼
Filter by model/version
│
▼
Open representative trace
│
▼
Find slow/failing span
│
▼
Inspect correlated logs
│
▼
Identify root cause
For example:
ALERT:
p95 latency > 8s
↓
FILTER:
prompt_version = support-v24
↓
TRACE:
rag.retrieve = 5.4s
↓
ROOT CAUSE:
vector search latency regression
That's observability doing useful engineering work.
What Should You Avoid Collecting?
Your telemetry policy should explicitly define what should not be collected.
Usually that includes:
Passwords
API keys
Authorization headers
Access tokens
Raw secrets
Payment details
Unnecessary PII
Sensitive documents
For prompt and completion content, make a deliberate risk-based decision.
Don't enable content capture simply because the instrumentation supports it.
A Practical Production Architecture
A mature setup could eventually look like:
LLM Application
│
▼
OpenTelemetry SDK
│
▼
OTel Collector
│
┌───────────┼───────────┐
│ │ │
▼ ▼ ▼
Metrics Traces Logs
│ │ │
▼ ▼ ▼
Prometheus Tempo Loki
│ │ │
└───────────┼───────────┘
│
▼
Grafana
│
┌───────────┼───────────┐
▼ ▼ ▼
Alerts Dashboards Debugging
Then add:
Evaluation results
User feedback
Cost data
Release metadata
Prompt versions
Model versions
as your system matures.
LLM Observability Setup Checklist
Before calling the setup complete, verify:
Instrumentation
End-to-end requests have trace IDs
LLM calls are individual spans
Retrieval operations are traced
Tool calls are traced
Errors are recorded
Metrics
Request volume
Latency
Input tokens
Output tokens
Error rate
Retry rate
Estimated cost
Metadata
Model
Prompt version
Application version
Environment
Privacy
Secrets are excluded
PII policy exists
Prompt capture is intentional
Retention is defined
Telemetry access is restricted
Operations
Dashboard exists
Alerts exist
Sampling is configured
Engineers know how to find a trace from an incident
Common LLM Observability Mistakes
Logging Everything
More data isn't automatically more observability.
Collect information that helps answer operational questions.
Capturing Prompts Without a Privacy Plan
Prompt logging can expose sensitive information.
Start metadata-first.
Tracking Only Latency
A fast LLM request can still be expensive or incorrect.
Track:
latency + tokens + cost + errors + quality
Tracking Only the LLM
For RAG and agents, the model is only one part of the system.
Trace:
retrieval
tools
databases
external APIs
model calls
together.
Ignoring Versions
Always record enough information to correlate regressions with changes:
model version
prompt version
application version
retrieval configuration
Building Vendor-Specific Instrumentation Too Early
If possible, instrument around open telemetry standards and adapt at the export layer.
That keeps future backend changes much easier.
What You Actually Need to Start
You don't need a huge LLMOps platform on day one.
Start with:
OpenTelemetry SDK
↓
OTel Collector
↓
Traces + Metrics
↓
Dashboard
Instrument:
LLM calls
retrieval
tools
and record:
latency
tokens
errors
model
prompt version
trace ID
That's already enough to answer a surprising number of production questions.
Add complexity only when a real problem requires it.
Final Takeaway
A good LLM observability setup isn't about collecting every possible detail from every prompt.
It's about being able to answer:
What happened, where did it happen, why did it happen, and what changed?
Build around open telemetry rather than a specific dashboard.
Start with metadata rather than sensitive content.
Trace the entire application rather than only the model call.
Measure tokens and cost alongside traditional reliability metrics.
Correlate prompts, models, retrieval, and application releases through version metadata.
Then gradually connect operational telemetry with evaluation results and user feedback.
The final goal isn't a beautiful dashboard.
It's reaching the point where an LLM application behaves strangely in production and your team can explain why without guessing.