Skip to main content
What is Tracing? - Think of tracing as having superpowers that let you see through your agent’s execution! Every conversation, every tool call, every decision - it’s all captured in beautiful detail. No more guessing what went wrong or why something is slow!

What Gets Captured?

Execution Timeline

Complete timeline with nested spans showing exactly when each operation happened

LLM Interactions

Every request and response, including prompts, completions, and model details

Tool Executions

All tool calls with their parameters, results, and execution time

Memory Operations

Vector searches, context retrieval, and memory storage operations

Token Usage

Detailed token counts and costs for every LLM interaction

Performance Metrics

Bottlenecks, slow operations, and optimization opportunities

Viewing Your Traces

Web Interface - The Visual Experience!

Fire up your browser and navigate to http://your-app.test/vizra/traces for the full visual experience!

Interactive Timeline

Zoom, pan, and explore your execution flow visually

Span Details

Click any span to see inputs, outputs, and metadata

Waterfall Charts

Visualize performance bottlenecks at a glance

Token Analytics

See exactly where your tokens are being spent

CLI Commands - For the Terminal Warriors!

Prefer the command line? We’ve got you covered with powerful CLI tools!
Pro Tip: Every agent interaction creates a unique session ID (like abc123). Use this session ID to view all the execution traces from that conversation! You can find the session ID in your agent responses or logs.
Terminal

Understanding Trace Structure

Traces are organized in a beautiful tree structure, just like a family tree! Each operation has parent and child relationships that show you exactly how your agent thinks.
Typical Trace Structure

Programmatic Tracing

Accessing Traces in Code

Want to build your own debugging tools? Access trace data directly in your code! It’s like having a debugger API at your fingertips.
TraceAccess.php

Creating Custom Spans

Add your own spans to trace custom operations! Perfect for tracking database queries, API calls, or any complex logic.
CustomTool.php
The Tracer::span() method automatically handles timing, error capturing, and span hierarchy. Just wrap your code and let it do the magic!

Analyzing Your Traces

Performance Analysis

Find those pesky bottlenecks and optimize like a pro! Let’s hunt down slow operations and expensive API calls.
PerformanceAnalysis.php

Quick Wins

Look for operations over 500ms - they’re usually the easiest to optimize!

Token Tip

High token usage often means verbose prompts - try to be more concise!

Error Tracking

Errors happen to the best of us! Track them down and squash those bugs with precision.
ErrorTracking.php
Error Recovery Tips:
  • Check error patterns - repeated errors often have the same root cause
  • Look at parent spans - context matters!
  • Review input parameters - bad data in = errors out
  • Monitor error rates over time to catch regressions early

Configuring Tracing

Configuration Options

Fine-tune your tracing setup to match your needs! Control what gets traced, how long to keep data, and more.
config/vizra-adk.php

Environment Variables

Set these in your .env file to control tracing behavior across environments!
.env

Development Tip

Always enable tracing in development for maximum visibility!

Production Tip

Consider sampling in production to balance insights with performance.

Data Retention & Cleanup

Keep your database lean and mean! Set up automatic cleanup to remove old traces.
Terminal
Pro Tip: Schedule It! - Add this to your Laravel scheduler for automatic cleanup:
app/Console/Kernel.php

How Tracing Works Behind the Scenes

Automatic Span Creation

The ADK works its magic by automatically creating spans for key operations. No manual instrumentation needed!

Agent Execution

Root span created when agent runsBaseLlmAgent::handle()

LLM Calls

Every AI interaction trackedBaseLlmAgent::callLlm()

Tool Calls

Tool execution monitoringBaseLlmAgent::executeTool()

Sub-agent Calls

Delegation trackingWhen delegating to sub-agents

Manual Span Creation

Need more control? Create your own spans for custom operations! Perfect for tracking specific business logic.
CustomOperation.php
Span Best Practices:
  • Keep span names descriptive and consistent
  • Add relevant metadata for better filtering
  • Always handle errors to mark spans as failed
  • Use nested spans for complex operations

Debugging Like a Detective

Common Issues & Solutions

Let’s solve those mysteries! Here are the most common issues and how to track them down.

Slow LLM Calls

Check prompt size and model selectionTip: Smaller prompts = faster responses!

Failed Tool Calls

Review parameters and error messagesTip: Check for missing required params!

High Token Usage

Optimize prompts and contextTip: Use system prompts wisely!

Memory Misses

Verify vector search configurationTip: Check embedding model consistency!

Timeout Errors

Identify bottlenecks in executionTip: Look for sequential operations!

Retry Loops

Check for infinite retry patternsTip: Add exponential backoff!

Tracing Best Practices

Development

  • Enable tracing for all requests - visibility is king!
  • Add custom spans for business logic
  • Use trace viewer frequently during development

Production

  • Use sampling to control costs and performance
  • Set up alerts for anomalies and errors
  • Regularly clean up old traces
Golden Rules:
  1. Review traces regularly - Make it a habit!
  2. Export important traces - Keep them for post-mortems
  3. Share traces with your team - Collective debugging is powerful
  4. Learn from patterns - Similar issues often have similar traces

You’re Now a Tracing Expert!

With the power of tracing, you can see through your agents like never before. No more mysteries, no more guessing - just pure visibility and control! Remember: Great debugging starts with great tracing. Use it liberally, learn from it constantly, and let it guide you to building amazing AI agents!