What is Tracing? - Think of tracing as having superpowers that let you see through your agent’s execution! Every conversation, every tool call, every decision - it’s all captured in beautiful detail. No more guessing what went wrong or why something is slow!
What Gets Captured?
Execution Timeline
Complete timeline with nested spans showing exactly when each operation happened
LLM Interactions
Every request and response, including prompts, completions, and model details
Tool Executions
All tool calls with their parameters, results, and execution time
Memory Operations
Vector searches, context retrieval, and memory storage operations
Token Usage
Detailed token counts and costs for every LLM interaction
Performance Metrics
Bottlenecks, slow operations, and optimization opportunities
Viewing Your Traces
Web Interface - The Visual Experience!
Fire up your browser and navigate tohttp://your-app.test/vizra/traces for the full visual experience!
Interactive Timeline
Zoom, pan, and explore your execution flow visually
Span Details
Click any span to see inputs, outputs, and metadata
Waterfall Charts
Visualize performance bottlenecks at a glance
Token Analytics
See exactly where your tokens are being spent
CLI Commands - For the Terminal Warriors!
Prefer the command line? We’ve got you covered with powerful CLI tools!Terminal
Understanding Trace Structure
Traces are organized in a beautiful tree structure, just like a family tree! Each operation has parent and child relationships that show you exactly how your agent thinks.Typical Trace Structure
| Span Type | Description |
|---|---|
| Root Span | The main execution |
| Parent Spans | Major operations |
| Child Spans | Detailed steps |
Programmatic Tracing
Accessing Traces in Code
Want to build your own debugging tools? Access trace data directly in your code! It’s like having a debugger API at your fingertips.TraceAccess.php
Creating Custom Spans
Add your own spans to trace custom operations! Perfect for tracking database queries, API calls, or any complex logic.CustomTool.php
Analyzing Your Traces
Performance Analysis
Find those pesky bottlenecks and optimize like a pro! Let’s hunt down slow operations and expensive API calls.PerformanceAnalysis.php
Quick Wins
Look for operations over 500ms - they’re usually the easiest to optimize!
Token Tip
High token usage often means verbose prompts - try to be more concise!
Error Tracking
Errors happen to the best of us! Track them down and squash those bugs with precision.ErrorTracking.php
- Check error patterns - repeated errors often have the same root cause
- Look at parent spans - context matters!
- Review input parameters - bad data in = errors out
- Monitor error rates over time to catch regressions early
Configuring Tracing
Configuration Options
Fine-tune your tracing setup to match your needs! Control what gets traced, how long to keep data, and more.config/vizra-adk.php
Environment Variables
Set these in your.env file to control tracing behavior across environments!
.env
Development Tip
Always enable tracing in development for maximum visibility!
Production Tip
Consider sampling in production to balance insights with performance.
Data Retention & Cleanup
Keep your database lean and mean! Set up automatic cleanup to remove old traces.Terminal
How Tracing Works Behind the Scenes
Automatic Span Creation
The ADK works its magic by automatically creating spans for key operations. No manual instrumentation needed!Agent Execution
Root span created when agent runs
BaseLlmAgent::handle()LLM Calls
Every AI interaction tracked
BaseLlmAgent::callLlm()Tool Calls
Tool execution monitoring
BaseLlmAgent::executeTool()Sub-agent Calls
Delegation trackingWhen delegating to sub-agents
Manual Span Creation
Need more control? Create your own spans for custom operations! Perfect for tracking specific business logic.CustomOperation.php
- Keep span names descriptive and consistent
- Add relevant metadata for better filtering
- Always handle errors to mark spans as failed
- Use nested spans for complex operations
Debugging Like a Detective
Common Issues & Solutions
Let’s solve those mysteries! Here are the most common issues and how to track them down.Slow LLM Calls
Check prompt size and model selectionTip: Smaller prompts = faster responses!
Failed Tool Calls
Review parameters and error messagesTip: Check for missing required params!
High Token Usage
Optimize prompts and contextTip: Use system prompts wisely!
Memory Misses
Verify vector search configurationTip: Check embedding model consistency!
Timeout Errors
Identify bottlenecks in executionTip: Look for sequential operations!
Retry Loops
Check for infinite retry patternsTip: Add exponential backoff!
Tracing Best Practices
Development
- Enable tracing for all requests - visibility is king!
- Add custom spans for business logic
- Use trace viewer frequently during development
Production
- Use sampling to control costs and performance
- Set up alerts for anomalies and errors
- Regularly clean up old traces
- Review traces regularly - Make it a habit!
- Export important traces - Keep them for post-mortems
- Share traces with your team - Collective debugging is powerful
- Learn from patterns - Similar issues often have similar traces