Skip to main content
What is Tracing? - Think of tracing as having superpowers that let you see through your agent’s execution! Every conversation, every tool call, every decision - it’s all captured in beautiful detail. No more guessing what went wrong or why something is slow!

What Gets Captured?

Execution Timeline

Complete timeline with nested spans showing exactly when each operation happened

LLM Interactions

Every request and response, including prompts, completions, and model details

Tool Executions

All tool calls with their parameters, results, and execution time

Memory Operations

Vector searches, context retrieval, and memory storage operations

Token Usage

Detailed token counts and costs for every LLM interaction

Performance Metrics

Bottlenecks, slow operations, and optimization opportunities

Viewing Your Traces

Web Interface - The Visual Experience!

Fire up your browser and navigate to http://your-app.test/vizra/traces for the full visual experience!

Interactive Timeline

Zoom, pan, and explore your execution flow visually

Span Details

Click any span to see inputs, outputs, and metadata

Waterfall Charts

Visualize performance bottlenecks at a glance

Token Analytics

See exactly where your tokens are being spent

CLI Commands - For the Terminal Warriors!

Prefer the command line? We’ve got you covered with powerful CLI tools!
Pro Tip: Every agent interaction creates a unique session ID (like abc123). Use this session ID to view all the execution traces from that conversation! You can find the session ID in your agent responses or logs.
Terminal
# View traces for a session
php artisan vizra:trace {session_id}

# Example: View all traces for session abc123
php artisan vizra:trace abc123

# View trace with additional options
php artisan vizra:trace abc123 --format=tree --show-input --show-output

# View a specific trace ID instead of all session traces
php artisan vizra:trace any-session --trace-id=specific-trace-id

# Clean up old traces
php artisan vizra:trace:cleanup --days=30

Understanding Trace Structure

Traces are organized in a beautiful tree structure, just like a family tree! Each operation has parent and child relationships that show you exactly how your agent thinks.
Typical Trace Structure
Agent Execution (root span)
├── Session Initialization
├── Memory Retrieval
│   ├── Vector Search
│   └── Context Building
├── LLM Request
│   ├── Prompt Construction
│   ├── API Call
│   └── Response Processing
├── Tool Execution
│   ├── Tool: order_lookup
│   └── Tool: send_email
└── Memory Storage
Span TypeDescription
Root SpanThe main execution
Parent SpansMajor operations
Child SpansDetailed steps

Programmatic Tracing

Accessing Traces in Code

Want to build your own debugging tools? Access trace data directly in your code! It’s like having a debugger API at your fingertips.
TraceAccess.php
use Vizra\ADK\Tracing\TraceManager;

// Get trace for a session
$trace = TraceManager::forSession($sessionId);

// Loop through all spans
foreach ($trace->spans as $span) {
    echo $span->name . ': ' . $span->duration . "ms\n";
}

// Get performance metrics
$metrics = $trace->getMetrics();
echo "Total Duration: " . $metrics['total_duration'] . "ms\n";
echo "Token Usage: " . $metrics['total_tokens'] . "\n";

Creating Custom Spans

Add your own spans to trace custom operations! Perfect for tracking database queries, API calls, or any complex logic.
CustomTool.php
use Vizra\ADK\Tracing\Tracer;

class CustomTool extends BaseTool
{
    public function execute(array $parameters): ToolResult
    {
        return Tracer::span('custom_operation', function() use ($parameters) {
            // Add span attributes
            Tracer::addAttribute('operation.type', 'database');
            Tracer::addAttribute('query.complexity', 'high');

            // Your operation
            $result = $this->performOperation($parameters);

            // Add result to span
            Tracer::addEvent('operation_completed', [
                'records_processed' => count($result),
            ]);

            return ToolResult::success($result);
        });
    }
}
The Tracer::span() method automatically handles timing, error capturing, and span hierarchy. Just wrap your code and let it do the magic!

Analyzing Your Traces

Performance Analysis

Find those pesky bottlenecks and optimize like a pro! Let’s hunt down slow operations and expensive API calls.
PerformanceAnalysis.php
// Find slow operations
$slowSpans = $trace->getSlowSpans(1000); // Spans over 1 second

foreach ($slowSpans as $span) {
    echo "Slow operation: {$span->name} took {$span->duration}ms\n";
}

// Analyze token usage
$tokenAnalysis = $trace->analyzeTokenUsage();
echo "Most expensive span: " . $tokenAnalysis['most_expensive']->name;
echo " used " . $tokenAnalysis['most_expensive']->tokens . " tokens\n";

Quick Wins

Look for operations over 500ms - they’re usually the easiest to optimize!

Token Tip

High token usage often means verbose prompts - try to be more concise!

Error Tracking

Errors happen to the best of us! Track them down and squash those bugs with precision.
ErrorTracking.php
// Find failed spans
$errors = $trace->getErrors();

foreach ($errors as $error) {
    echo "Error in {$error->span->name}: {$error->message}\n";
    echo "Stack trace:\n{$error->stackTrace}\n";
}
Error Recovery Tips:
  • Check error patterns - repeated errors often have the same root cause
  • Look at parent spans - context matters!
  • Review input parameters - bad data in = errors out
  • Monitor error rates over time to catch regressions early

Configuring Tracing

Configuration Options

Fine-tune your tracing setup to match your needs! Control what gets traced, how long to keep data, and more.
config/vizra-adk.php
'tracing' => [
    'enabled' => env('VIZRA_ADK_TRACING_ENABLED', true),
    'table' => 'agent_trace_spans',
    'cleanup_days' => env('VIZRA_ADK_TRACING_CLEANUP_DAYS', 30),
];

Environment Variables

Set these in your .env file to control tracing behavior across environments!
.env
# Enable/disable tracing
VIZRA_ADK_TRACING_ENABLED=true

# Days to keep trace data before cleanup
VIZRA_ADK_TRACING_CLEANUP_DAYS=30

Development Tip

Always enable tracing in development for maximum visibility!

Production Tip

Consider sampling in production to balance insights with performance.

Data Retention & Cleanup

Keep your database lean and mean! Set up automatic cleanup to remove old traces.
Terminal
# Clean up old traces (uses config value, default 30 days)
php artisan vizra:trace:cleanup

# Keep last 30 days of traces (delete anything older)
php artisan vizra:trace:cleanup --days=30

# Keep last week of traces (delete anything older)
php artisan vizra:trace:cleanup --days=7

# Keep only today's traces (delete anything from yesterday or earlier)
php artisan vizra:trace:cleanup --days=1

# Delete ALL traces (use with caution!)
php artisan vizra:trace:cleanup --days=0

# Dry run to see what would be deleted
php artisan vizra:trace:cleanup --days=7 --dry-run

# Skip confirmation prompt
php artisan vizra:trace:cleanup --days=7 --force
Pro Tip: Schedule It! - Add this to your Laravel scheduler for automatic cleanup:
app/Console/Kernel.php
$schedule->command('vizra:trace:cleanup')->daily();

How Tracing Works Behind the Scenes

Automatic Span Creation

The ADK works its magic by automatically creating spans for key operations. No manual instrumentation needed!

Agent Execution

Root span created when agent runsBaseLlmAgent::handle()

LLM Calls

Every AI interaction trackedBaseLlmAgent::callLlm()

Tool Calls

Tool execution monitoringBaseLlmAgent::executeTool()

Sub-agent Calls

Delegation trackingWhen delegating to sub-agents

Manual Span Creation

Need more control? Create your own spans for custom operations! Perfect for tracking specific business logic.
CustomOperation.php
use Vizra\VizraADK\Services\Tracer;

class CustomOperation
{
    public function process(Tracer $tracer)
    {
        // Start a custom span
        $spanId = $tracer->startSpan(
            type: 'custom_operation',
            name: 'Process Data',
            input: ['records' => 100],
            metadata: ['batch_id' => 'abc123']
        );

        try {
            // Your operation
            $result = $this->doWork();

            // End span with success
            $tracer->endSpan($spanId, ['processed' => 100]);

        } catch (\Throwable $e) {
            // Mark span as failed
            $tracer->failSpan($spanId, $e);
            throw $e;
        }
    }
}
Span Best Practices:
  • Keep span names descriptive and consistent
  • Add relevant metadata for better filtering
  • Always handle errors to mark spans as failed
  • Use nested spans for complex operations

Debugging Like a Detective

Common Issues & Solutions

Let’s solve those mysteries! Here are the most common issues and how to track them down.

Slow LLM Calls

Check prompt size and model selectionTip: Smaller prompts = faster responses!

Failed Tool Calls

Review parameters and error messagesTip: Check for missing required params!

High Token Usage

Optimize prompts and contextTip: Use system prompts wisely!

Memory Misses

Verify vector search configurationTip: Check embedding model consistency!

Timeout Errors

Identify bottlenecks in executionTip: Look for sequential operations!

Retry Loops

Check for infinite retry patternsTip: Add exponential backoff!

Tracing Best Practices

Development

  • Enable tracing for all requests - visibility is king!
  • Add custom spans for business logic
  • Use trace viewer frequently during development

Production

  • Use sampling to control costs and performance
  • Set up alerts for anomalies and errors
  • Regularly clean up old traces
Golden Rules:
  1. Review traces regularly - Make it a habit!
  2. Export important traces - Keep them for post-mortems
  3. Share traces with your team - Collective debugging is powerful
  4. Learn from patterns - Similar issues often have similar traces

You’re Now a Tracing Expert!

With the power of tracing, you can see through your agents like never before. No more mysteries, no more guessing - just pure visibility and control! Remember: Great debugging starts with great tracing. Use it liberally, learn from it constantly, and let it guide you to building amazing AI agents!