# Vizra ADK Source: https://docs.vizra.ai/adk/index Build incredible intelligent agents with the most developer-friendly Laravel package. Your journey to AI mastery starts here! **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). Welcome to the future of AI development! Build incredible intelligent agents with the most developer-friendly Laravel package on the planet. ## Lightning Quick Start Get your first AI agent running with these simple commands: ```bash Terminal theme={null} # Install the package composer require vizra/vizra-adk # Set up everything with one command php artisan vizra:install # Create your first agent php artisan vizra:make:agent CustomerSupportAgent ``` ## What Makes Vizra ADK Special? Vizra ADK isn't just another package - it's your toolkit for creating intelligent, conversational agents that work in production. OpenAI, Anthropic, Google Gemini - use them all! Give your agents superpowers with custom tools Vector memory & RAG for intelligent conversations Build multi-step processes that just work LLM-as-a-Judge for automated testing See exactly what your agents are thinking ## Your Learning Adventure Pick your path and let's build something incredible together: Get up and running in minutes! Requirements, installation, and your first agent Master agents, tools, sessions, and workflows - the building blocks of intelligence Complete technical reference - every class, method, and command documented Handle exceptions gracefully and build resilient agents that recover from failures ## See the Magic in Action Here's a real agent that can look up orders and process refunds - all in just a few lines of code! ```php app/Agents/CustomerSupportAgent.php theme={null} That's it! This agent can now have intelligent conversations, look up customer orders, and process refunds - all while maintaining context and providing helpful responses. The framework handles all the complex AI orchestration for you! ## Join Our Community Got questions? Need help? Want to share what you're building? Join our growing community of AI developers! Join hundreds of Laravel developers building AI agents. Get help, share ideas, and stay updated on the latest features! ## Ready to Build Something Amazing? Your AI journey starts with a single command! Choose your adventure and let's create agents that will blow your mind. Install, create, and deploy your first agent in minutes! Master the fundamentals of intelligent agent development Deep dive into every class, method, and feature # Agent Queuing Source: https://docs.vizra.ai/advanced/agent-queuing Queue agents for background and async processing with Laravel queues **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). ## What is Agent Queuing? Agent Queuing allows you to execute AI agents asynchronously using Laravel's queue system. This is useful for long-running tasks, background processing, and scenarios where you don't need immediate responses. Run LLM agents and media generation in the background without blocking requests Uses Laravel's native queue system - works with any queue driver (Redis, SQS, database, etc.) Built-in retry logic and configurable timeouts for reliable execution Dispatches events on completion and failure for easy integration ## Quick Start ### LLM Agent Queuing ```php theme={null} use App\Agents\MyAgent; // Queue an agent for background processing $result = MyAgent::ask('Analyze this large dataset...') ->onQueue('agents') ->go(); // Returns immediately with job info // [ // 'job_dispatched' => true, // 'job_id' => 'uuid-here', // 'queue' => 'agents', // 'agent' => 'my_agent' // ] ``` ### Media Agent Queuing ```php theme={null} use Vizra\VizraADK\Agents\ImageAgent; // Queue image generation $result = ImageAgent::run('A futuristic cityscape') ->onQueue('media') ->then(fn($image) => $image->storeAs('cityscape.png')) ->go(); // [ // 'job_dispatched' => true, // 'job_id' => 'uuid-here', // 'queue' => 'media', // 'agent' => 'image_agent', // 'prompt' => 'A futuristic cityscape' // ] ``` When you specify a queue with `onQueue()`, async mode is automatically enabled - no need to call `async()` separately. ## LLM Agent Queuing LLM agents can be queued using the fluent executor API. This is useful for tasks that may take a while to complete, such as complex analysis or multi-step reasoning. ### Enabling Async Execution ```php theme={null} // Method 1: Enable async mode explicitly MyAgent::ask('Process this...') ->async() ->go(); // Method 2: Specify a queue (auto-enables async) MyAgent::ask('Process this...') ->onQueue('agents') ->go(); ``` ### Queue Configuration Options | Method | Description | Default | | ----------------------------- | --------------------------------------- | ------------- | | `async(bool $enabled = true)` | Enable/disable async execution | `false` | | `onQueue(string $queue)` | Specify queue name (auto-enables async) | `'default'` | | `delay(int $seconds)` | Delay job execution | `null` | | `tries(int $tries)` | Number of retry attempts | `3` | | `timeout(int $seconds)` | Job timeout in seconds | `300` (5 min) | ### Complete LLM Agent Example ```php theme={null} use App\Agents\AnalysisAgent; $result = AnalysisAgent::ask('Analyze the quarterly sales data and generate insights') ->forUser($user) // Associate with user ->withSession('analysis-session') // Track with session ID ->withContext([ // Add extra context 'report_type' => 'quarterly', 'department' => 'sales', ]) ->onQueue('analysis') // Use specific queue ->delay(30) // Wait 30 seconds before processing ->tries(5) // Retry up to 5 times ->timeout(600) // 10-minute timeout ->go(); // Store the job ID for tracking $jobId = $result['job_id']; ``` ## Media Agent Queuing Media agents (ImageAgent, AudioAgent) support the same queuing options plus additional features like post-generation callbacks and auto-storage. ### Queue Configuration Options | Method | Description | Default | | ------------------------------------------ | --------------------------------------- | ------------- | | `async(bool $enabled = true)` | Enable/disable async execution | `false` | | `onQueue(string $queue)` | Specify queue name (auto-enables async) | `'default'` | | `delay(int $seconds)` | Delay job execution | `null` | | `tries(int $tries)` | Number of retry attempts | `3` | | `timeout(int $seconds)` | Job timeout in seconds | `120` (2 min) | | `then(Closure $callback)` | Post-generation callback | `null` | | `store(?string $disk)` | Auto-store with generated filename | - | | `storeAs(string $filename, ?string $disk)` | Auto-store with specific filename | - | ### Post-Generation Callbacks Use the `then()` method to execute code after generation completes: ```php theme={null} use Vizra\VizraADK\Agents\ImageAgent; ImageAgent::run('A product photo of a leather bag') ->onQueue('media') ->then(function ($image) { // Store the image $image->storeAs('products/leather-bag.png'); // Update database Product::find(123)->update([ 'image_path' => $image->path(), ]); // Send notification Notification::send($user, new ImageReady($image)); }) ->go(); ``` The `then()` callback must be serializable for queue processing. Avoid using `$this` or non-serializable objects inside the closure. ### Auto-Storage Options Configure automatic storage directly in the fluent chain: ```php theme={null} // Store with auto-generated filename (ULID) ImageAgent::run('A landscape photo') ->onQueue('media') ->store() ->go(); // Store with specific filename ImageAgent::run('A portrait photo') ->onQueue('media') ->storeAs('portraits/user-avatar.png') ->go(); // Store to a specific disk ImageAgent::run('A banner image') ->onQueue('media') ->storeAs('banners/hero.png', 's3') ->go(); ``` ### Complete Media Agent Example ```php theme={null} use Vizra\VizraADK\Agents\ImageAgent; $result = ImageAgent::run('Professional headshot of a business executive') ->forUser($user) ->withSession('avatar-generation') ->using('openai', 'dall-e-3') ->size('1024x1024') ->quality('hd') ->style('natural') ->onQueue('media') ->delay(5) ->tries(3) ->timeout(180) ->then(function ($image) use ($user) { $image->storeAs("avatars/{$user->id}.png"); $user->update(['avatar_path' => $image->path()]); }) ->go(); ``` ## Job Response & Tracking When you dispatch an agent to a queue, you receive an immediate response with job information: ### LLM Agent Response ```php theme={null} $result = MyAgent::ask('...')->onQueue('agents')->go(); // Returns: [ 'job_dispatched' => true, 'job_id' => '550e8400-e29b-41d4-a716-446655440000', 'queue' => 'agents', 'agent' => 'my_agent' ] ``` ### Media Agent Response ```php theme={null} $result = ImageAgent::run('...')->onQueue('media')->go(); // Returns: [ 'job_dispatched' => true, 'job_id' => '550e8400-e29b-41d4-a716-446655440000', 'queue' => 'media', 'agent' => 'image_agent', 'prompt' => 'The original prompt...' ] ``` ### Retrieving Results from Cache Job results are automatically cached for retrieval: ```php theme={null} // LLM Agent results (cached for 1 hour) $result = cache()->get("agent_job_result:{$jobId}"); $meta = cache()->get("agent_job_meta:{$jobId}"); // Media Agent results (cached for 1 hour) $mediaResult = cache()->get("media_job_result:{$jobId}"); // Check for failures (cached for 24 hours) $failure = cache()->get("agent_job_failure:{$jobId}"); $mediaFailure = cache()->get("media_job_failure:{$jobId}"); ``` ### Cache Key Reference | Cache Key | TTL | Description | | --------------------------- | -------- | --------------------------------------- | | `agent_job_result:{jobId}` | 1 hour | LLM agent execution result | | `agent_job_meta:{jobId}` | 1 hour | LLM agent job metadata | | `media_job_result:{jobId}` | 1 hour | Media generation result with URLs/paths | | `agent_job_failure:{jobId}` | 24 hours | LLM agent failure info | | `media_job_failure:{jobId}` | 24 hours | Media generation failure info | ## Events The queuing system dispatches events on job completion and failure, allowing you to react to agent execution status. ### LLM Agent Events | Event | Description | | ------------------------ | ------------------------------------------------------------------ | | `agent.job.completed` | Fired when any agent job completes successfully | | `agent.{name}.completed` | Agent-specific completion event (e.g., `agent.my_agent.completed`) | | `agent.job.failed` | Fired when any agent job permanently fails | ### Media Agent Events | Event | Description | | ------------------------ | --------------------------------------------------------------------- | | `media.job.completed` | Fired when any media job completes successfully | | `media.{name}.completed` | Agent-specific completion event (e.g., `media.image_agent.completed`) | | `media.job.failed` | Fired when any media job permanently fails | ### Event Listener Examples ```php theme={null} use Illuminate\Support\Facades\Event; // Listen for all agent job completions Event::listen('agent.job.completed', function (array $data) { Log::info('Agent job completed', [ 'job_id' => $data['job_id'], 'agent_class' => $data['agent_class'], 'result' => $data['result'], ]); }); // Listen for a specific agent Event::listen('agent.analysis_agent.completed', function (array $data) { $result = $data['result']; $sessionId = $data['session_id']; // Process the analysis result ProcessAnalysisResult::dispatch($result, $sessionId); }); // Listen for media generation completion Event::listen('media.image_agent.completed', function (array $data) { $response = $data['response']; $jobId = $data['job_id']; Log::info("Image generated: " . $response->url()); }); // Handle failures Event::listen('agent.job.failed', function (array $data) { Log::error('Agent job failed', [ 'job_id' => $data['job_id'], 'agent_class' => $data['agent_class'], 'error' => $data['error'], ]); // Notify admin or trigger alerting Notification::route('slack', config('services.slack.webhook')) ->notify(new AgentJobFailed($data)); }); ``` Event payloads include the `job_id`, `agent_class`, `session_id`, and the full `result` or `response` object. Use these to update your database, send notifications, or trigger follow-up actions. ## Job Tagging Jobs are automatically tagged for easy filtering and monitoring with tools like Laravel Horizon. ### LLM Agent Tags ```php theme={null} // Tags applied to AgentJob [ 'vizra:{agent_name}', // e.g., 'vizra:my_agent' 'session:{session_id}' // e.g., 'session:user_123_abc' ] ``` ### Media Agent Tags ```php theme={null} // Tags applied to MediaGenerationJob [ 'vizra:media', 'vizra:{agent_name}', // e.g., 'vizra:image_agent' 'session:{session_id}' ] ``` ### Viewing Tags in Horizon If you're using Laravel Horizon, you can filter jobs by these tags: ```php theme={null} // In your Horizon config, jobs will appear with tags like: // vizra:my_agent, session:user_123_abc // Search for all jobs from a specific agent // Filter: vizra:analysis_agent // Search for all media generation jobs // Filter: vizra:media ``` ## Configuration & Prerequisites ### Queue Driver Configuration Agent queuing uses Laravel's standard queue system. Configure your preferred driver in `.env`: ```env theme={null} QUEUE_CONNECTION=redis ``` Supported drivers include `redis`, `database`, `sqs`, `beanstalkd`, and others. ### Running Queue Workers Process queued jobs with Laravel's queue worker: ```bash theme={null} # Process all queues php artisan queue:work # Process specific queue php artisan queue:work --queue=agents # Process multiple queues with priority php artisan queue:work --queue=agents,media,default # With Horizon (recommended for production) php artisan horizon ``` ### Supervisor Configuration For production, use Supervisor to keep queue workers running: ```ini theme={null} [program:agent-worker] process_name=%(program_name)s_%(process_num)02d command=php /path/to/artisan queue:work --queue=agents,media --sleep=3 --tries=3 --max-time=3600 autostart=true autorestart=true stopasgroup=true killasgroup=true numprocs=4 redirect_stderr=true stdout_logfile=/path/to/logs/agent-worker.log ``` ### Horizon Configuration For advanced monitoring and management, use Laravel Horizon: ```php config/horizon.php theme={null} 'environments' => [ 'production' => [ 'agent-workers' => [ 'connection' => 'redis', 'queue' => ['agents', 'media'], 'balance' => 'auto', 'processes' => 4, 'tries' => 3, 'timeout' => 600, ], ], ], ``` ## Best Practices Separate agent and media jobs from other application jobs using dedicated queues LLM tasks may need longer timeouts (5-10 min), while media generation is typically faster (2 min) Use Laravel Horizon for real-time monitoring, metrics, and job management Always listen for failure events and implement appropriate error handling ### Recommended Queue Structure ```php theme={null} // Separate queues by workload type ->onQueue('agents') // For LLM agents ->onQueue('media') // For image/audio generation ->onQueue('analysis') // For long-running analysis tasks ``` ### Timeout Guidelines | Task Type | Recommended Timeout | | -------------------------- | ------------------- | | Simple LLM queries | 60-120 seconds | | Complex reasoning/analysis | 300-600 seconds | | Image generation | 120 seconds | | Audio generation | 180 seconds | | Multi-step workflows | 600+ seconds | ## Complete Examples ### Background Report Generation ```php theme={null} use App\Agents\ReportAgent; use App\Models\Report; class ReportController extends Controller { public function generate(Request $request) { $report = Report::create([ 'user_id' => $request->user()->id, 'status' => 'processing', 'type' => $request->input('type'), ]); $result = ReportAgent::ask("Generate a {$request->input('type')} report") ->forUser($request->user()) ->withSession("report-{$report->id}") ->withContext([ 'report_id' => $report->id, 'date_range' => $request->input('date_range'), ]) ->onQueue('reports') ->timeout(600) ->go(); $report->update(['job_id' => $result['job_id']]); return response()->json([ 'message' => 'Report generation started', 'report_id' => $report->id, 'job_id' => $result['job_id'], ]); } } // Event listener for completion Event::listen('agent.report_agent.completed', function ($data) { $sessionId = $data['session_id']; // 'report-123' $reportId = str_replace('report-', '', $sessionId); Report::find($reportId)->update([ 'status' => 'completed', 'content' => $data['result'], ]); }); ``` ### Batch Image Generation ```php theme={null} use Vizra\VizraADK\Agents\ImageAgent; class ProductImageController extends Controller { public function generateBatch(Request $request) { $products = Product::whereNull('image_path')->get(); $jobs = []; foreach ($products as $product) { $result = ImageAgent::run("Product photo of: {$product->name}") ->forUser($request->user()) ->withSession("product-{$product->id}") ->using('openai', 'dall-e-3') ->square() ->hd() ->style('natural') ->onQueue('media') ->then(function ($image) use ($product) { $path = "products/{$product->id}.png"; $image->storeAs($path); $product->update(['image_path' => $path]); }) ->go(); $jobs[] = [ 'product_id' => $product->id, 'job_id' => $result['job_id'], ]; } return response()->json([ 'message' => count($jobs) . ' image generation jobs queued', 'jobs' => $jobs, ]); } } ``` Learn about ImageAgent and its queuing features Learn about AudioAgent and speech generation # Macros & Mixins Source: https://docs.vizra.ai/advanced/macros Extend Vizra ADK with custom methods using Laravel's powerful macros and mixins pattern **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). ## What are Macros? Macros allow you to add custom methods to classes at runtime. This is a powerful extensibility pattern borrowed from Laravel that lets you extend Vizra ADK's functionality without modifying the core code. Add analytics and tracking to your agents Integrate with external services seamlessly Add business-specific functionality Create patterns you can use across your app ## Supported Classes The following Vizra ADK classes support macros: | Class | Access | Description | | ----------------- | ----------------- | ------------------------------------ | | `AgentManager` | `Agent` facade | Central agent management | | `AgentBuilder` | Fluent builder | Agent configuration and registration | | `WorkflowManager` | `Workflow` facade | Workflow orchestration | ## Basic Usage ### Registering a Macro Register macros in your `AppServiceProvider::boot()` method: ```php app/Providers/AppServiceProvider.php theme={null} trackedModel = $model; return $this; // Return $this for method chaining }); } } ``` ### Using Your Macro ```php theme={null} use App\Models\Unit; use Vizra\VizraADK\Facades\Agent; // Step 1: Use the macro when registering the agent Agent::build(CustomerSupportAgent::class) ->track(Unit::find(12)) ->register(); // Step 2: Run the agent using the executor API $response = CustomerSupportAgent::run('User input') ->forUser($user) ->go(); ``` **Always return `$this`** from your macros to maintain the fluent interface and enable method chaining. ## Real-World Examples ### Analytics Tracking Track which models are using AI features: ```php theme={null} use Vizra\VizraADK\Services\AgentBuilder; use Illuminate\Database\Eloquent\Model; AgentBuilder::macro('track', function (Model $model) { $this->trackedModel = $model; $this->trackedModelType = get_class($model); $this->trackedModelId = $model->getKey(); return $this; }); // Usage Agent::build(MyAgent::class) ->track(Unit::find(12)) ->register(); // Then run the agent MyAgent::run('User input') ->forUser($user) ->go(); ``` ### Conditional Execution Add conditional logic to your builder: ```php theme={null} AgentBuilder::macro('whenCondition', function ($condition, callable $callback) { if ($condition) { $callback($this); } return $this; }); // Usage $agentName = Agent::define('conditional-agent') ->whenCondition($user->isPremium(), function ($builder) { $builder->model('gpt-4o'); }) ->whenCondition(!$user->isPremium(), function ($builder) { $builder->model('gpt-4o-mini'); }) ->register(); // Run the defined agent Agent::named($agentName)->run('User input')->go(); ``` ### Metadata Tagging Add metadata for logging or debugging: ```php theme={null} AgentBuilder::macro('withTags', function (array $tags) { $this->tags = $tags; return $this; }); AgentBuilder::macro('withPriority', function (string $priority) { $this->priority = $priority; return $this; }); // Usage $agentName = Agent::define('support-agent') ->withTags(['customer-facing', 'urgent']) ->withPriority('high') ->register(); // Run the agent Agent::named($agentName)->run('User input')->go(); ``` ### Cost Tracking Track estimated costs for budget monitoring: ```php theme={null} AgentBuilder::macro('trackCost', function (string $costCenter) { $this->costCenter = $costCenter; $this->trackTokenUsage = true; return $this; }); // Usage Agent::build(AnalyticsAgent::class) ->trackCost('DEPT-001') ->register(); // Run the agent AnalyticsAgent::run('User input')->go(); ``` ## Using Mixins Mixins allow you to add multiple macros at once by grouping related functionality in a class: ```php app/Mixins/AnalyticsMixin.php theme={null} trackedModel = $model; return $this; }; } public function recordActivity() { return function ($activity) { $this->activityLog[] = $activity; return $this; }; } public function getActivityLog() { return function () { return $this->activityLog ?? []; }; } } ``` Register the mixin in your service provider: ```php app/Providers/AppServiceProvider.php theme={null} use Vizra\VizraADK\Services\AgentBuilder; use App\Mixins\AnalyticsMixin; public function boot(): void { AgentBuilder::mixin(new AnalyticsMixin()); } ``` Now use all the mixed-in methods: ```php theme={null} $agentName = Agent::define('analytics-agent') ->track($unit) ->recordActivity('agent_created') ->recordActivity('agent_configured') ->register(); // Run the agent Agent::named($agentName)->run('User input')->go(); ``` Mixins are ideal when you have a group of related macros that logically belong together, such as analytics, auditing, or domain-specific operations. ## Advanced Patterns ### Integration with Events Combine macros with Laravel events for powerful side effects: ```php theme={null} use App\Events\AgentTracked; AgentBuilder::macro('trackWithEvent', function (Model $model) { $this->trackedModel = $model; // Fire an event when the agent is used event(new AgentTracked($model, $this)); return $this; }); ``` ### Dynamic Configuration Load configuration dynamically from your models: ```php theme={null} AgentBuilder::macro('configureFor', function (Model $model) { // Load model-specific configuration $config = $model->agentConfiguration ?? []; if (isset($config['model'])) { $this->model($config['model']); } if (isset($config['instructions'])) { $this->instructions($config['instructions']); } return $this; }); // Usage Agent::build(MyAgent::class) ->configureFor($tenant) ->register(); // Run the agent MyAgent::run('User input')->go(); ``` ### Workflow Macros Add custom workflow types using the `WorkflowManager`: ```php theme={null} use Vizra\VizraADK\Facades\Workflow; use Vizra\VizraADK\Services\WorkflowManager; WorkflowManager::macro('retryable', function (string $agentClass, int $maxRetries = 3) { $retries = $maxRetries; return Workflow::loop($agentClass) ->until(function ($result) use (&$retries) { return $result->success || $retries-- <= 0; }); }); // Usage $result = Workflow::retryable(UnreliableAgent::class, 5)->go(); ``` ## Best Practices Always return `$this` to maintain the fluent interface Register macros in `AppServiceProvider::boot()` Use clear, descriptive names for your macros Document macros for other developers on your team Additional tips: * **Consider scope** - Macros are global, so namespace them if needed to avoid conflicts * **Test your macros** - Write tests to ensure macros work as expected * **Group related macros** - Use mixins to organize related functionality ## Testing Macros Test your macros using Pest or PHPUnit: ```php tests/Feature/MacroTest.php theme={null} use Vizra\VizraADK\Services\AgentBuilder; use Vizra\VizraADK\Facades\Agent; use App\Models\Unit; it('can track models with macro', function () { AgentBuilder::macro('track', function (Model $model) { $this->trackedModel = $model; return $this; }); $model = Unit::factory()->create(); $builder = Agent::define('test-agent')->track($model); expect($builder->trackedModel)->toBe($model); }); ``` The full test suite for macro support is available at `tests/Feature/MacroSupportTest.php` with 10 comprehensive tests covering all major use cases. ## Learn More Official Laravel documentation on macros Complete AgentBuilder reference # Audio Agent Source: https://docs.vizra.ai/agents/audio-agent Generate speech and audio with the AudioAgent **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). The `AudioAgent` provides text-to-speech (TTS) capabilities using AI models through Prism PHP. Convert any text into natural-sounding speech with multiple voice options and audio formats. ## Quick Start Generate speech from text with just a few lines of code: ```php theme={null} use Vizra\VizraADK\Agents\AudioAgent; // Simple TTS generation $audio = AudioAgent::run('Welcome to our application')->go(); // Save the audio $audio->storeAs('welcome.mp3'); // Get the URL $url = $audio->url(); ``` ## Voice Selection Choose from six distinct AI voices, each with their own character: ```php theme={null} // Using the voice() method $audio = AudioAgent::run('Hello world') ->voice('nova') ->go(); // Using voice preset methods $audio = AudioAgent::run('Hello world') ->shimmer() ->go(); ``` ### Available Voices | Voice | Description | | --------- | --------------------------------- | | `alloy` | Balanced, neutral voice (default) | | `echo` | Warm, conversational tone | | `fable` | Expressive, storytelling style | | `onyx` | Deep, authoritative voice | | `nova` | Friendly, energetic delivery | | `shimmer` | Clear, professional sound | Each voice has a shortcut method: `alloy()`, `echo()`, `fable()`, `onyx()`, `nova()`, `shimmer()`. ## Audio Formats Select your preferred output format based on your use case: ```php theme={null} $audio = AudioAgent::run('Your order has been confirmed') ->format('wav') ->go(); ``` ### Supported Formats | Format | Use Case | | ------ | ----------------------------------- | | `mp3` | Web playback, general use (default) | | `wav` | High quality, editing workflows | | `opus` | Streaming, low latency | | `aac` | iOS/Safari compatibility | | `flac` | Lossless archival | ## Speech Speed Adjust the speaking rate between 0.25x and 4.0x: ```php theme={null} // Slower for accessibility $audio = AudioAgent::run('Important instructions') ->speed(0.8) ->go(); // Faster for summaries $audio = AudioAgent::run('Quick recap of today') ->speed(1.5) ->go(); ``` ## Model Selection Override the default model for different quality/speed tradeoffs: ```php theme={null} // Use HD model for higher quality $audio = AudioAgent::run('Premium narration content') ->using('openai', 'tts-1-hd') ->go(); // Use mini model for faster generation $audio = AudioAgent::run('Quick notification') ->using('openai', 'gpt-4o-mini-tts') ->go(); ``` ### Available Models | Model | Description | | ----------------- | --------------------------------------------- | | `tts-1` | Standard quality, faster generation (default) | | `tts-1-hd` | Higher quality, slower generation | | `gpt-4o-mini-tts` | Latest model with enhanced capabilities | ## Storage ### Auto-store with Custom Filename ```php theme={null} $audio = AudioAgent::run('Welcome message') ->voice('nova') ->storeAs('welcome.mp3') ->go(); echo $audio->url(); // Returns the stored file URL echo $audio->path(); // Returns the storage path ``` ### Auto-store with Generated Filename ```php theme={null} $audio = AudioAgent::run('Dynamic content') ->store() ->go(); // File stored with ULID filename like: vizra-adk/generated/audio/01HXY...mp3 ``` ### Store to Specific Disk ```php theme={null} $audio = AudioAgent::run('Private audio') ->storeAs('private/message.mp3', 's3') ->go(); ``` ### Manual Storage ```php theme={null} $audio = AudioAgent::run('Hello world')->go(); // Store with custom filename $audio->storeAs('greetings/hello.mp3'); // Or auto-generate filename $audio->store(); ``` ## Working with AudioResponse The `AudioResponse` object provides methods to access and manipulate the generated audio: ```php theme={null} $audio = AudioAgent::run('Sample text') ->nova() ->format('mp3') ->go(); // Access audio data $audio->data(); // Raw binary audio data $audio->base64(); // Base64-encoded audio $audio->toDataUri(); // Data URI for embedding (data:audio/mpeg;base64,...) // Metadata $audio->text(); // Original input text $audio->voice(); // Voice used (e.g., 'nova') $audio->format(); // Format (e.g., 'mp3') $audio->mimeType(); // MIME type (e.g., 'audio/mpeg') $audio->metadata(); // Full metadata array // Storage status $audio->isStored(); // Check if audio was stored $audio->url(); // Get URL (stored or data URI) $audio->path(); // Get storage path $audio->disk(); // Get storage disk used ``` ### Embedding in HTML ```php theme={null} $audio = AudioAgent::run('Play this message')->go(); // Using data URI (no storage needed) $html = '

'; // Using stored URL $audio->store(); $html = '

'; ``` ## Async/Queued Generation For long-running generations or batch processing, use Laravel queues: ```php theme={null} // Basic async generation AudioAgent::run('Long article content here...') ->onQueue('media') ->go(); // With callback after completion AudioAgent::run('Order confirmation for order #12345') ->shimmer() ->format('mp3') ->onQueue('media') ->then(function ($audio) { $audio->storeAs('confirmations/order-12345.mp3'); // Send notification, update database, etc. Notification::send($user, new AudioReadyNotification($audio->url())); }) ->go(); ``` ### Queue Options ```php theme={null} AudioAgent::run('Background audio task') ->onQueue('media') // Specify queue name ->delay(60) // Delay in seconds ->tries(5) // Retry attempts ->timeout(180) // Timeout in seconds ->then(fn($audio) => $audio->store()) ->go(); ``` ### Async Return Value When using async mode, `go()` returns job information instead of the audio: ```php theme={null} $result = AudioAgent::run('Async generation') ->onQueue('media') ->go(); // $result contains: // [ // 'job_dispatched' => true, // 'job_id' => 'uuid-string', // 'queue' => 'media', // 'agent' => 'audio_agent', // 'prompt' => 'Async generation', // ] ``` ## User Context Associate audio generation with a user for tracking and personalization: ```php theme={null} AudioAgent::run('Personal greeting') ->forUser($user) ->withSession('session-123') ->go(); ``` ## Sub-agent Delegation Allow your LLM agents to generate audio by delegating to the AudioAgent: ```php theme={null} use Vizra\VizraADK\Agents\BaseLlmAgent; use Vizra\VizraADK\Tools\DelegateToMediaAgentTool; class AssistantAgent extends BaseLlmAgent { protected string $name = 'assistant'; protected string $instructions = <<<'PROMPT' You are a helpful assistant. When the user asks you to read something aloud or create audio, use the generate_audio tool to convert text to speech. PROMPT; protected array $tools = [ DelegateToMediaAgentTool::class, ]; protected array $mediaAgents = [ \Vizra\VizraADK\Agents\AudioAgent::class, ]; } ``` Or create the tool directly: ```php theme={null} use Vizra\VizraADK\Tools\DelegateToMediaAgentTool; class NarratorAgent extends BaseLlmAgent { protected function tools(): array { return [ DelegateToMediaAgentTool::forAudio(), ]; } } ``` When the LLM calls the `generate_audio` tool, it can specify: * `text` (required): The text to convert to speech * `voice` (optional): Voice selection * `format` (optional): Output format ## Configuration ### Environment Variables ```env theme={null} # Provider and model VIZRA_ADK_AUDIO_PROVIDER=openai VIZRA_ADK_AUDIO_MODEL=tts-1 # Defaults VIZRA_ADK_AUDIO_VOICE=alloy VIZRA_ADK_AUDIO_FORMAT=mp3 VIZRA_ADK_AUDIO_SPEED=1.0 ``` ### Config File In `config/vizra-adk.php`: ```php theme={null} 'media' => [ 'audio' => [ 'provider' => env('VIZRA_ADK_AUDIO_PROVIDER', 'openai'), 'model' => env('VIZRA_ADK_AUDIO_MODEL', 'tts-1'), 'default_voice' => env('VIZRA_ADK_AUDIO_VOICE', 'alloy'), 'default_format' => env('VIZRA_ADK_AUDIO_FORMAT', 'mp3'), 'default_speed' => env('VIZRA_ADK_AUDIO_SPEED', 1.0), ], 'storage' => [ 'disk' => env('VIZRA_ADK_MEDIA_DISK', 'public'), 'path' => env('VIZRA_ADK_MEDIA_PATH', 'vizra-adk/generated'), ], ], ``` ## Complete Example Here's a comprehensive example showing common patterns: ```php theme={null} use Vizra\VizraADK\Agents\AudioAgent; class PodcastService { public function generateIntro(string $episodeTitle): string { $text = "Welcome to today's episode: {$episodeTitle}"; $audio = AudioAgent::run($text) ->using('openai', 'tts-1-hd') ->nova() ->format('mp3') ->speed(1.0) ->storeAs("podcasts/intros/{$this->slug($episodeTitle)}.mp3") ->go(); return $audio->url(); } public function generateChapterAudio(array $chapters): void { foreach ($chapters as $index => $chapter) { AudioAgent::run($chapter['content']) ->onyx() ->format('mp3') ->onQueue('podcast-generation') ->then(function ($audio) use ($chapter) { $audio->storeAs("chapters/{$chapter['id']}.mp3"); Chapter::find($chapter['id'])->update([ 'audio_url' => $audio->url(), 'audio_generated_at' => now(), ]); }) ->go(); } } } ``` ## Error Handling ```php theme={null} try { $audio = AudioAgent::run('Generate this audio') ->nova() ->go(); $audio->store(); } catch (\Exception $e) { Log::error('Audio generation failed', [ 'error' => $e->getMessage(), ]); } ``` ## API Reference ### AudioAgent Methods | Method | Description | | -------------------------------------- | ----------------------------------- | | `run(string $text)` | Static method to start fluent chain | | `execute($input, $context)` | Direct execution with context | | `toToolDefinition()` | Get tool schema for sub-agent use | | `executeFromToolCall($args, $context)` | Execute from LLM tool call | ### MediaAgentExecutor Methods (Fluent Chain) | Method | Description | | ------------------------------------------ | --------------------------- | | `voice(string $voice)` | Set voice | | `format(string $format)` | Set output format | | `speed(float $speed)` | Set speech speed (0.25-4.0) | | `alloy()`, `echo()`, etc. | Voice presets | | `using(string $provider, string $model)` | Override provider/model | | `forUser(Model $user)` | Set user context | | `withSession(string $id)` | Set session ID | | `withContext(array $context)` | Add context data | | `store(?string $path, ?string $disk)` | Auto-store with path | | `storeAs(string $filename, ?string $disk)` | Auto-store with filename | | `onQueue(string $queue)` | Enable async on queue | | `delay(int $seconds)` | Delay queue execution | | `tries(int $count)` | Set retry attempts | | `timeout(int $seconds)` | Set timeout | | `then(Closure $callback)` | Callback after completion | | `go()` | Execute and return result | ### AudioResponse Methods | Method | Returns | Description | | ------------------------------------------ | --------- | -------------------------- | | `data()` | `string` | Raw binary audio | | `base64()` | `string` | Base64-encoded audio | | `toDataUri()` | `string` | Data URI for embedding | | `text()` | `string` | Original input text | | `voice()` | `string` | Voice used | | `format()` | `string` | Audio format | | `mimeType()` | `string` | MIME type | | `metadata()` | `array` | Full metadata | | `store(?string $disk)` | `static` | Store with auto filename | | `storeAs(string $filename, ?string $disk)` | `static` | Store with custom filename | | `url()` | `?string` | Get URL | | `path()` | `?string` | Get storage path | | `disk()` | `?string` | Get storage disk | | `isStored()` | `bool` | Check if stored | | `raw()` | `mixed` | Get raw Prism response | | `toArray()` | `array` | Convert to array | | `toJson()` | `string` | Convert to JSON | # Image Agent Source: https://docs.vizra.ai/agents/image-agent Generate images from text descriptions using AI models like DALL-E and Imagen **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). ## What is ImageAgent? ImageAgent is a specialized media agent that generates images from text descriptions. It integrates with providers like OpenAI (DALL-E) and Google (Imagen) through Prism PHP, offering a fluent API for configuration, built-in storage, and queue support. Works with OpenAI DALL-E, Google Imagen, and other image generation models Chain methods like `size()`, `quality()`, and `style()` for clean configuration Store generated images directly to Laravel's filesystem with `store()` and `storeAs()` Process image generation asynchronously with Laravel queues ## Quick Start Generate an image with just a few lines of code: ```php theme={null} use Vizra\VizraADK\Agents\ImageAgent; // Generate and store an image $image = ImageAgent::run('A sunset over the ocean') ->quality('hd') ->go(); $image->storeAs('sunset.png'); echo $image->url(); // URL to the stored image ``` The `run()` method returns a fluent executor - chain your configuration and call `go()` to execute! ## Basic Usage ### Simple Generation ```php theme={null} // Generate with default settings $image = ImageAgent::run('A mountain landscape at dawn')->go(); // Access the image $url = $image->url(); // Storage or provider URL $base64 = $image->base64(); // Base64-encoded data $data = $image->data(); // Raw binary data ``` ### Storing Images Images can be stored to Laravel's filesystem: ```php theme={null} // Store with auto-generated filename (ULID) $image = ImageAgent::run('An oil painting of flowers')->go(); $image->store(); // Stores as e.g., 01HWXYZ123.png $image->store('s3'); // Store to a specific disk // Store with custom filename $image->storeAs('flowers.png'); $image->storeAs('art/flowers.png', 's3'); // Check storage status if ($image->isStored()) { echo $image->path(); // vizra-adk/generated/images/flowers.png echo $image->disk(); // public } ``` Images are stored in the `vizra-adk/generated/images/` directory by default. Configure this in your `vizra-adk.php` config file. ### Auto-Store on Generation Chain storage directly in the fluent API: ```php theme={null} // Store with auto-generated name $image = ImageAgent::run('A futuristic city') ->store() ->go(); // Store with specific filename $image = ImageAgent::run('A vintage car') ->storeAs('vintage-car.png') ->go(); ``` ## Configuration Options ### Image Size Set dimensions using preset methods or custom sizes: ```php theme={null} // Preset sizes ImageAgent::run('...')->square()->go(); // 1024x1024 ImageAgent::run('...')->portrait()->go(); // 1024x1792 ImageAgent::run('...')->landscape()->go(); // 1792x1024 // Custom size ImageAgent::run('...')->size('512x512')->go(); ``` Available sizes depend on your provider: | Provider | Supported Sizes | | --------------- | ------------------------------- | | OpenAI DALL-E 3 | 1024x1024, 1024x1792, 1792x1024 | | OpenAI DALL-E 2 | 256x256, 512x512, 1024x1024 | | Google Imagen | Varies by model | ### Quality Control the quality level (OpenAI-specific): ```php theme={null} // Standard quality (default, faster) ImageAgent::run('...')->quality('standard')->go(); // HD quality (more detail, slower) ImageAgent::run('...')->quality('hd')->go(); ImageAgent::run('...')->hd()->go(); // Shorthand ``` ### Style Set the visual style (DALL-E 3 specific): ```php theme={null} // Vivid - hyper-real, dramatic (default) ImageAgent::run('...')->style('vivid')->go(); // Natural - more realistic, less exaggerated ImageAgent::run('...')->style('natural')->go(); ``` ### Provider & Model Override the default provider and model: ```php theme={null} // Use a specific provider and model ImageAgent::run('A serene lake') ->using('openai', 'dall-e-3') ->go(); // Switch to Google Imagen ImageAgent::run('A colorful abstract pattern') ->using('google', 'imagen-3') ->go(); ``` ### Complete Configuration Example ```php theme={null} $image = ImageAgent::run('An oil painting of flowers') ->using('openai', 'dall-e-3') ->landscape() ->hd() ->style('natural') ->storeAs('flowers.png') ->go(); ``` ## ImageResponse Class The `ImageResponse` object provides access to all image data and metadata: ### Accessing Image Data ```php theme={null} $image = ImageAgent::run('...')->go(); // URLs $url = $image->url(); // Best available URL (stored or provider) $providerUrl = $image->providerUrl(); // Original URL from provider // Binary data $data = $image->data(); // Raw binary content $base64 = $image->base64(); // Base64-encoded string $dataUri = $image->toDataUri(); // data:image/png;base64,... // Storage info (after storing) $path = $image->path(); // Storage path $disk = $image->disk(); // Storage disk name ``` ### Metadata ```php theme={null} $image = ImageAgent::run('A sunset')->go(); // Access metadata echo $image->prompt(); // "A sunset" echo $image->mimeType(); // "image/png" echo $image->hasImage(); // true echo $image->isStored(); // false (until stored) // Get all metadata as array $metadata = $image->metadata(); // [ // 'prompt' => 'A sunset', // 'provider' => 'openai', // 'model' => 'dall-e-3', // 'url' => '...', // 'provider_url' => '...', // 'path' => null, // 'disk' => null, // 'generated_at' => '2024-01-15T10:30:00.000Z' // ] ``` ### Serialization ```php theme={null} $image = ImageAgent::run('...')->go(); // Convert to array or JSON $array = $image->toArray(); $json = $image->toJson(); // String casting returns URL echo $image; // Outputs the URL ``` ## Async & Queue Processing For long-running generations, use Laravel queues: ### Basic Async ```php theme={null} // Dispatch to default queue $result = ImageAgent::run('A complex scene') ->async() ->go(); // Returns immediately with job info // [ // 'job_dispatched' => true, // 'job_id' => 'uuid-here', // 'queue' => 'default', // 'agent' => 'image_agent', // 'prompt' => 'A complex scene' // ] ``` ### Queue Configuration ```php theme={null} ImageAgent::run('A detailed illustration') ->onQueue('media') // Specific queue name ->delay(60) // Delay execution by 60 seconds ->tries(5) // Retry up to 5 times on failure ->timeout(120) // 2-minute timeout ->then(fn($image) => $image->storeAs('illustration.png')) ->go(); ``` The `then()` callback runs after generation completes - use it to store images or trigger notifications. The callback must be serializable for queue processing. ### Listening for Completion Listen for media generation events: ```php theme={null} // In your EventServiceProvider or Listener Event::listen('media.image_agent.completed', function ($event) { $jobId = $event['job_id']; $response = $event['response']; $sessionId = $event['session_id']; // Process the completed image Log::info("Image generated: " . $response->url()); }); // Generic event for all media types Event::listen('media.job.completed', function ($event) { // Handle any media job completion }); ``` ## Using with LLM Agents Allow your LLM agents to generate images using the `DelegateToMediaAgentTool`: ### Setup ```php theme={null} use Vizra\VizraADK\Agents\BaseLlmAgent; use Vizra\VizraADK\Agents\ImageAgent; use Vizra\VizraADK\Tools\DelegateToMediaAgentTool; class CreativeAssistantAgent extends BaseLlmAgent { protected string $name = 'creative_assistant'; protected string $description = 'A creative assistant that can generate images'; protected string $instructions = <<tools[] = DelegateToMediaAgentTool::forImage(); } ``` ### How It Works When the LLM decides to generate an image, it calls the `generate_image` tool with these parameters: ```json theme={null} { "name": "generate_image", "parameters": { "prompt": "Detailed description of the image", "size": "1024x1024", "quality": "hd", "style": "natural" } } ``` The tool automatically: 1. Executes the ImageAgent with the provided parameters 2. Stores the generated image 3. Returns the URL and path to the LLM ## Context & User Tracking Associate generations with users and sessions: ```php theme={null} ImageAgent::run('A personalized avatar') ->forUser($user) // Associate with a user model ->withSession('session-123') // Custom session ID ->withContext([ // Additional context data 'source' => 'profile_editor', 'request_id' => $requestId, ]) ->go(); ``` ## Configuration Reference ### Environment Variables | Variable | Description | Default | | -------------------------- | -------------------------------- | --------------------- | | `VIZRA_ADK_MEDIA_ENABLED` | Enable media generation | `true` | | `VIZRA_ADK_MEDIA_DISK` | Storage disk for generated media | `public` | | `VIZRA_ADK_MEDIA_PATH` | Base path for stored media | `vizra-adk/generated` | | `VIZRA_ADK_IMAGE_PROVIDER` | Default image provider | `openai` | | `VIZRA_ADK_IMAGE_MODEL` | Default image model | `dall-e-3` | | `VIZRA_ADK_IMAGE_SIZE` | Default image size | `1024x1024` | | `VIZRA_ADK_IMAGE_QUALITY` | Default quality | `standard` | | `VIZRA_ADK_IMAGE_STYLE` | Default style | `vivid` | | `VIZRA_ADK_IMAGE_FORMAT` | Response format | `url` | ### Config File ```php config/vizra-adk.php theme={null} 'media' => [ 'enabled' => env('VIZRA_ADK_MEDIA_ENABLED', true), 'storage' => [ 'disk' => env('VIZRA_ADK_MEDIA_DISK', 'public'), 'path' => env('VIZRA_ADK_MEDIA_PATH', 'vizra-adk/generated'), ], 'image' => [ 'provider' => env('VIZRA_ADK_IMAGE_PROVIDER', 'openai'), 'model' => env('VIZRA_ADK_IMAGE_MODEL', 'dall-e-3'), 'default_size' => env('VIZRA_ADK_IMAGE_SIZE', '1024x1024'), 'default_quality' => env('VIZRA_ADK_IMAGE_QUALITY', 'standard'), 'default_style' => env('VIZRA_ADK_IMAGE_STYLE', 'vivid'), 'response_format' => env('VIZRA_ADK_IMAGE_FORMAT', 'url'), ], ], ``` ### Supported Providers & Models * **dall-e-3** - Latest, highest quality * **dall-e-2** - Faster, more sizes * **gpt-image-1** - Newest model * **imagen-3** - High quality generation * **imagen-4** - Latest Imagen model * **gemini-2.0-flash-preview-image-generation** ## API Reference ### ImageAgent Static Methods | Method | Description | | --------------------- | ------------------------------- | | `run(string $prompt)` | Start a fluent generation chain | ### MediaAgentExecutor Methods (Fluent API) | Method | Description | | ------------------------------------------ | ------------------------------ | | `using(string $provider, string $model)` | Override provider and model | | `forUser(Model $user)` | Associate with a user | | `withSession(string $sessionId)` | Set session ID | | `withContext(array $context)` | Add context data | | `size(string $size)` | Set custom dimensions | | `square()` | 1024x1024 | | `portrait()` | 1024x1792 | | `landscape()` | 1792x1024 | | `quality(string $quality)` | Set quality level | | `hd()` | Shorthand for HD quality | | `style(string $style)` | Set visual style | | `store(?string $disk)` | Auto-store with generated name | | `storeAs(string $filename, ?string $disk)` | Auto-store with specific name | | `async(bool $enabled)` | Enable async processing | | `onQueue(string $queue)` | Specify queue name | | `delay(int $seconds)` | Delay execution | | `tries(int $tries)` | Set retry attempts | | `timeout(int $seconds)` | Set timeout | | `then(Closure $callback)` | Post-generation callback | | `go()` | Execute the generation | ### ImageResponse Methods | Method | Returns | Description | | ------------------------------------------ | --------- | ---------------------- | | `url()` | `?string` | Best available URL | | `providerUrl()` | `?string` | Original provider URL | | `path()` | `?string` | Storage path | | `disk()` | `?string` | Storage disk | | `base64()` | `string` | Base64-encoded data | | `data()` | `string` | Raw binary data | | `toDataUri()` | `string` | Data URI for embedding | | `prompt()` | `string` | Original prompt | | `mimeType()` | `string` | MIME type | | `metadata()` | `array` | All metadata | | `hasImage()` | `bool` | Has image data | | `isStored()` | `bool` | Has been stored | | `store(?string $disk)` | `static` | Store with auto name | | `storeAs(string $filename, ?string $disk)` | `static` | Store with name | | `toArray()` | `array` | Convert to array | | `toJson()` | `string` | Convert to JSON | Generate speech and audio with the AudioAgent Learn more about async agent processing # Specialized Agents Source: https://docs.vizra.ai/agents/overview Discover specialized agent types for audio, images, and more **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). Coming soon... # Agent Class Reference Source: https://docs.vizra.ai/api-reference/agent-class Master the heart of Vizra ADK. The BaseLlmAgent class is your gateway to building incredible AI agents. Every property, method, and secret revealed. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). Think of `BaseLlmAgent` as the DNA of your AI agents. Every intelligent behavior, every smart response, every amazing capability starts here. ## The Agent Family Tree Understanding the class hierarchy is like knowing your agent's family tree. Here's how everything connects: ```php Agent Class Hierarchy theme={null} namespace Vizra\VizraADK\Agents; abstract class BaseAgent { // Base contract for all agents } abstract class BaseLlmAgent extends BaseAgent { // LLM-powered agent implementation } ``` ## Agent Properties - Your Agent's DNA These properties are like your agent's personality traits. Set them right, and your agent will be amazing. ### Required Properties **Type:** `string` (required) Your agent's unique identifier. Like a superhero name, it should be memorable and descriptive. No two agents can share the same name. ```php theme={null} protected string $name = 'customer_support'; ``` **Type:** `string` (required) Tell everyone what your agent does. This human-readable description helps you and your team understand the agent's purpose at a glance. ```php theme={null} protected string $description = 'Helps customers with support inquiries'; ``` ### Optional Properties | Property | Type | Default | Description | | ---------------- | --------- | ------- | -------------------------------------------------- | | `$instructions` | string | `''` | System prompt for the agent | | `$promptVersion` | ?string | `null` | Default prompt version to use | | `$model` | string | `''` | LLM model to use (e.g., 'gpt-4o', 'claude-3-opus') | | `$provider` | ?Provider | `null` | LLM provider (OpenAI, Anthropic, Gemini) | | `$temperature` | ?float | `null` | Response creativity (0-1) | | `$maxTokens` | ?int | `null` | Maximum response tokens | | `$topP` | ?float | `null` | Nucleus sampling parameter | | `$streaming` | bool | `false` | Enable streaming responses | | `$tools` | array | `[]` | Array of tool class names | ## Static Method - The Universal Entry Point All agents use a single, powerful entry point - the `run()` method. This unified approach keeps things simple and consistent across all your agents. ### run() ```php Method Signature theme={null} public static function run(mixed $input): AgentExecutor ``` The `run()` method creates a fluent agent executor that you can configure with various options before executing. It's your Swiss Army knife for all agent interactions. Whether you're asking questions, processing data, analyzing events, or monitoring systems - it all starts with `run()`. The input you provide and the context you set determine what your agent does. ### Common Usage Patterns #### Conversational Interactions Perfect for chat-like interactions where you need natural language responses. ```php theme={null} // Ask a question $response = CustomerSupportAgent::run('Where is my order?') ->forUser($user) ->go(); // Have a conversation with context $response = CustomerSupportAgent::run('I need help with shipping') ->withSession($sessionId) ->withContext(['order_id' => $orderId]) ->go(); ``` #### Event Processing Handle events, webhooks, or triggers from your application. ```php theme={null} // Process an event $response = NotificationAgent::run($orderCreatedEvent) ->forUser($user) ->go(); // Handle webhook data $response = WebhookAgent::run($webhookPayload) ->withContext(['source' => 'stripe']) ->go(); ``` #### Data Analysis Analyze data, detect patterns, or generate insights. ```php theme={null} // Analyze payment data $response = FraudDetectionAgent::run($paymentData) ->withContext(['risk_threshold' => 0.7]) ->go(); // Generate insights $response = AnalyticsAgent::run($salesData) ->withContext(['period' => 'last_quarter']) ->go(); ``` #### Batch Processing Process large datasets or perform batch operations asynchronously. ```php theme={null} // Process batch data $response = DataProcessorAgent::run($largeDataset) ->async() ->onQueue('processing') ->go(); // Import data with progress tracking $response = ImportAgent::run($csvFile) ->withContext(['notify_on_complete' => true]) ->async() ->go(); ``` #### Monitoring & Alerts Monitor systems, metrics, or conditions continuously. ```php theme={null} // Monitor system metrics $response = SystemMonitorAgent::run($metrics) ->onQueue('monitoring') ->withContext(['alert_threshold' => 90]) ->go(); // Health checks $response = HealthCheckAgent::run($serviceEndpoints) ->delay(60) // Check every minute ->go(); ``` #### Report Generation Generate reports, summaries, or formatted outputs. ```php theme={null} // Generate daily report $response = ReportAgent::run('daily_sales') ->withContext(['date' => today()]) ->go(); // Create summary with specific format $response = SummaryAgent::run($quarterlyData) ->withContext(['format' => 'executive_summary']) ->go(); ``` ## Instance Methods (BaseLlmAgent) ### Core Methods ```php theme={null} // Execute the agent's primary logic (must be implemented) public function run(mixed $input, AgentContext $context): mixed // Get agent properties public function getName(): string public function getDescription(): string ``` ### Configuration Methods ```php theme={null} // Getters public function getModel(): string public function getProvider(): Provider public function getInstructions(): string public function getInstructionsWithMemory(AgentContext $context): string public function getTemperature(): ?float public function getMaxTokens(): ?int public function getTopP(): ?float public function getStreaming(): bool // Setters public function setInstructions(string $instructions): static public function setModel(string $model): static public function setProvider(Provider $provider): static public function setTemperature(?float $temperature): static public function setMaxTokens(?int $maxTokens): static public function setTopP(?float $topP): static public function setStreaming(bool $streaming): static ``` ### Dynamic Prompts (via VersionablePrompts trait) ```php theme={null} // Prompt version management public function setPromptVersion(?string $version): static public function getPromptVersion(): ?string public function setPromptOverride(string $prompt): static public function getAvailablePromptVersions(): array ``` ### Tool Management ```php theme={null} public function loadTools(): void public function getLoadedTools(): array ``` ### Sub-Agent Management ```php theme={null} public function getSubAgent(string $name): ?BaseLlmAgent public function getLoadedSubAgents(): array ``` ## Protected Methods (Override in Your Agent) ### Sub-Agent Registration ```php theme={null} /** * Sub-agents available for delegation. * The framework automatically generates names from class names: * - TechnicalSupportAgent -> technical_support * - BillingSupportAgent -> billing_support * - SalesAgent -> sales */ protected array $subAgents = [ TechnicalSupportAgent::class, BillingSupportAgent::class, SalesAgent::class, ]; ``` ### Lifecycle Hooks ```php theme={null} // Called before sending messages to LLM public function beforeLlmCall(array $inputMessages, AgentContext $context): array { // Modify messages, start tracing, etc. return $inputMessages; } // Called after receiving LLM response public function afterLlmResponse(Response|Generator $response, AgentContext $context): mixed { // Process response, access token usage, etc. return $response; } // Called before tool execution public function beforeToolCall(string $toolName, array $arguments, AgentContext $context): array { // Modify tool arguments before execution return $arguments; } // Called after tool execution public function afterToolResult(string $toolName, string $result, AgentContext $context): string { // Process tool results return $result; } ``` ### Sub-Agent Delegation Hooks ```php theme={null} // Called before delegating to sub-agent public function beforeSubAgentDelegation( string $subAgentName, string $taskInput, string $contextSummary, AgentContext $parentContext ): array { // Returns [subAgentName, taskInput, contextSummary] return [$subAgentName, $taskInput, $contextSummary]; } // Called after sub-agent completes public function afterSubAgentDelegation( string $subAgentName, string $taskInput, string $subAgentResult, AgentContext $parentContext, AgentContext $subAgentContext ): string { // Process and return the result return $subAgentResult; } ``` ## AgentExecutor Methods The `AgentExecutor` provides a fluent API for configuring agent execution: ```php theme={null} // User context public function forUser(?Model $user): self // Session management public function withSession(string $sessionId): self // Context data public function withContext(array $context): self // Prompt versioning public function withPromptVersion(string $version): self // Streaming public function streaming(bool $enabled = true): self // Parameters public function withParameters(array $parameters): self public function temperature(float $temperature): self public function maxTokens(int $maxTokens): self // Async execution public function async(bool $enabled = true): self public function onQueue(string $queue): self public function delay(int $seconds): self public function tries(int $tries): self public function timeout(int $seconds): self // Multimodal support public function withImage(string $path, ?string $mimeType = null): self public function withImageFromBase64(string $base64Data, string $mimeType): self public function withImageFromUrl(string $url): self public function withDocument(string $path, ?string $mimeType = null): self public function withDocumentFromBase64(string $base64Data, string $mimeType): self public function withDocumentFromUrl(string $url): self // Execute the agent public function go(): mixed ``` ## Complete Example ```php CustomerSupportAgent.php theme={null} > */ protected array $tools = [ OrderLookupTool::class, RefundProcessorTool::class, EmailSenderTool::class, ]; // Sub-agents available for delegation protected array $subAgents = [ TechnicalSupportAgent::class, // Becomes: technical_support BillingAgent::class, // Becomes: billing ]; // Customize LLM call behavior public function beforeLlmCall(array $inputMessages, AgentContext $context): array { // Add customer context to messages if available if ($userId = $context->getState('user_id')) { logger()->info('Processing request for user', ['user_id' => $userId]); } return parent::beforeLlmCall($inputMessages, $context); } // Process response after LLM call public function afterLlmResponse(Response|Generator $response, AgentContext $context): mixed { if ($response instanceof Response) { // Log token usage for monitoring if ($response->usage) { logger()->info('Token usage', [ 'input_tokens' => $response->usage->inputTokens ?? 0, 'output_tokens' => $response->usage->outputTokens ?? 0, ]); } } return parent::afterLlmResponse($response, $context); } } ``` ## Complete Usage Examples Let's see how the unified `run()` method handles all your agent needs. From simple queries to complex multimodal interactions, it's all here. ```php Real-World Examples theme={null} // Basic conversation $response = CustomerSupportAgent::run('How do I reset my password?') ->go(); // With user context and session $response = CustomerSupportAgent::run('Show me my recent orders') ->forUser($user) ->withSession($sessionId) ->go(); // Streaming response for real-time output $stream = CustomerSupportAgent::run('Explain our refund policy') ->streaming() ->go(); foreach ($stream as $chunk) { echo $chunk; // Display as it streams } // Async processing for heavy workloads $job = DataProcessorAgent::run($supportTickets) ->async() ->onQueue('support') ->timeout(3600) // 1 hour timeout ->go(); // Multimodal with images $response = ProductAnalysisAgent::run('What\'s wrong with this product?') ->withImage('/path/to/product-photo.jpg') ->go(); // Complex analysis with multiple inputs $response = FinancialAdvisorAgent::run('Analyze my portfolio') ->withDocument('/path/to/portfolio.pdf') ->withContext([ 'risk_tolerance' => 'moderate', 'investment_horizon' => '10_years' ]) ->temperature(0.3) // Lower temperature for financial advice ->go(); // Event-driven processing $response = OrderProcessingAgent::run($orderEvent) ->withContext(['priority' => 'high']) ->forUser($customer) ->go(); // Scheduled monitoring $response = SystemHealthAgent::run($systemMetrics) ->onQueue('monitoring') ->delay(300) // Check every 5 minutes ->tries(3) // Retry up to 3 times ->go(); ``` **Agent Best Practices:** * Keep agent instructions focused and clear * Override lifecycle hooks for custom behavior * Use appropriate temperature settings for your use case * Implement error handling for robustness * Test agents thoroughly with evaluations * Monitor token usage and costs ## Next Steps Tool class reference Learn more about agents # Artisan Commands Source: https://docs.vizra.ai/api-reference/artisan-commands Your CLI toolkit for building amazing AI agents. Master these commands to create, debug, and deploy intelligent agents with ease. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). Vizra's Artisan commands are your Swiss Army knife for AI agent development. From creating new agents to debugging complex interactions, everything is just a command away. ## Agent Development Commands These commands help you scaffold new agents, tools, and testing components in seconds. Create a new AI agent with a single command. This is where the magic begins. ```bash Terminal theme={null} # Create a new agent interactively php artisan vizra:make:agent # The command will prompt for the agent name # Creates: app/Agents/YourAgent.php ``` **What you get:** * Name, description, and instructions properties ready to customize * Model configuration (defaults to the blazing-fast gemini-2.0-flash) * Temperature and max tokens settings pre-configured * Empty tools array ready for your custom capabilities Give your agents superpowers. Tools let your agents interact with databases, APIs, and more. ```bash Terminal theme={null} # Create a new tool interactively php artisan vizra:make:tool # The command will prompt for the tool name # Creates: app/Tools/YourTool.php ``` **Smart features:** * Tool names are automatically snake\_cased for consistency * "\_tool" suffix is intelligently removed from the name * Implements ToolInterface with definition() and execute() methods * Includes JSON schema parameter definition for type safety Group related tools together with shared configuration. Toolboxes provide a clean way to organize tools with common gates, policies, or middleware. ```bash Terminal theme={null} # Create a new toolbox interactively php artisan vizra:make:toolbox # Create with specific options php artisan vizra:make:toolbox --gate=admin --policy=App\\Policies\\ToolPolicy # Create with pre-configured tools php artisan vizra:make:toolbox --tools=SearchTool,DatabaseTool,ApiTool # Creates: app/Toolboxes/YourToolbox.php ``` **Options:** | Option | Description | | ---------- | ------------------------------------------------ | | `--gate` | Apply a Laravel gate to all tools in the toolbox | | `--policy` | Apply a policy class for authorization | | `--tools` | Comma-separated list of tools to include | **What you get:** * A toolbox class extending BaseToolbox * Centralized configuration for related tools * Built-in support for gates and policies * Easy tool registration via the `$tools` property Quality assurance for your AI. Create evaluations to ensure your agents perform consistently. ```bash Terminal theme={null} # Create a new evaluation interactively php artisan vizra:make:eval # The command will prompt for the evaluation name # Creates: app/Evaluations/YourEvaluation.php ``` Create custom assertions for your evaluations. Assertions define the criteria for pass/fail outcomes. ```bash Terminal theme={null} # Create a new assertion interactively php artisan vizra:make:assertion # The command will prompt for the assertion name # Creates: app/Assertions/YourAssertion.php ``` **What you get:** * An assertion class extending BaseAssertion * The `evaluate()` method ready for your custom logic * Access to the agent response and expected outcome * Integration with the evaluation framework ## Agent Discovery Commands Explore your AI workforce. These commands help you discover and manage your agents. See all your available agents. This command shows you every agent in your application, whether registered or not. ```bash Terminal theme={null} # Discover all agents php artisan vizra:discover-agents # Clear discovery cache and rediscover php artisan vizra:discover-agents --clear-cache ``` **What you'll see:** * Complete list of all agents in your `app/Agents` directory * Agent names, class names, and registration status * Automatic discovery of new agents without manual registration * Clear indication of which agents are already registered vs available **Options:** | Option | Description | | --------------- | --------------------------------------- | | `--clear-cache` | Force rediscovery by clearing the cache | Vizra ADK automatically discovers and registers agents when you use them. You don't need to manually register agents in your `AppServiceProvider` anymore. Just create an agent and start using it. ## Agent Interaction Commands Time to talk. These commands let you interact with your agents directly from the terminal. Have a conversation with your AI agent right in your terminal. Perfect for testing and development. ```bash Terminal theme={null} php artisan vizra:chat {agent_name} # Example: Chat with the weather reporter agent php artisan vizra:chat weather_reporter # Type "exit" or "quit" to end the chat ``` **Features:** * Interactive terminal-based chat interface for instant feedback * Unique session ID for each chat to track conversations * Handles string, array, and object responses gracefully * Comprehensive error handling so nothing breaks ## Evaluation Commands Test your agents like a pro. Run comprehensive evaluations to ensure quality at scale. Put your agents through their paces. Run evaluations to measure performance and catch regressions. ```bash Terminal theme={null} php artisan vizra:run:eval {name} [--output=results.csv] # Example: Run CustomerServiceEvaluation php artisan vizra:run:eval CustomerServiceEvaluation # Save results to CSV file php artisan vizra:run:eval CustomerServiceEvaluation --output=results.csv ``` **Options:** | Option | Description | | ---------- | -------------------------------------------------- | | `name` | The evaluation class name (e.g., MyTestEvaluation) | | `--output` | Path to save CSV results for analysis | **What you'll see:** * Real-time progress bar during evaluation * Summary statistics with pass/fail counts * Detailed results with assertion breakdowns * CSV export with all test data for deeper analysis ## Prompt Management Commands Master your AI's voice. Manage prompt versions without touching code. Your command center for prompt versioning. Create, list, activate, and manage different prompt versions for each agent. ```bash Terminal theme={null} # Create a new prompt version php artisan vizra:prompt create {agent} {version} [--content="..."] # List all prompt versions php artisan vizra:prompt list [agent] # Activate a version (database only) php artisan vizra:prompt activate {agent} {version} # Export prompt to file php artisan vizra:prompt export {agent} {version} [--file=output.txt] # Import prompt from file php artisan vizra:prompt import {agent} {version} --file=input.txt ``` **Examples in action:** ```bash Terminal theme={null} # Create friendly customer support prompt php artisan vizra:prompt create customer_support friendly \ --content="You are a warm and friendly support agent. Use casual language." # List all weather reporter prompts php artisan vizra:prompt list weather_reporter # Switch to professional version in production php artisan vizra:prompt activate customer_support professional # Share prompts between projects php artisan vizra:prompt export my_agent v2 --file=awesome_prompt.txt php artisan vizra:prompt import other_agent v1 --file=awesome_prompt.txt ``` **Prompt versioning features:** * File-based storage by default at `resources/prompts/{agent}/{version}.txt` * Optional database storage for production environments * Runtime version switching without code changes * Perfect for A/B testing and experimentation * Integration with evaluations for systematic testing ## Debugging Commands Become a debugging detective. These commands help you understand exactly what your agents are doing. X-ray vision for your agents. See every step of execution in beautiful detail. ```bash Terminal theme={null} php artisan vizra:trace {session_id} [options] # View trace in tree format (default) php artisan vizra:trace abc123 # View specific trace ID php artisan vizra:trace abc123 --trace-id=xyz789 # Show detailed span data php artisan vizra:trace abc123 --show-input --show-output --show-metadata # Show only errors php artisan vizra:trace abc123 --errors-only # Different output formats php artisan vizra:trace abc123 --format=table php artisan vizra:trace abc123 --format=json ``` **Options:** | Option | Description | | ----------------- | -------------------------------------- | | `--trace-id` | Zero in on a specific trace | | `--show-input` | See what went in | | `--show-output` | See what came out | | `--show-metadata` | View all the extra details | | `--errors-only` | Focus on problems | | `--format` | Choose your view: tree, table, or json | Keep things tidy. Clean up old trace data to save space and maintain performance. ```bash Terminal theme={null} # Clean up traces older than config value (default: 30 days) php artisan vizra:trace:cleanup # Clean up traces older than 7 days php artisan vizra:trace:cleanup --days=7 # Preview what would be deleted php artisan vizra:trace:cleanup --dry-run # Skip confirmation prompt php artisan vizra:trace:cleanup --force ``` ## Vector Memory Commands Give your agents perfect memory. Store and search through knowledge using advanced vector embeddings. Feed your agent's brain. Store documents, manuals, and knowledge for instant recall. ```bash Terminal theme={null} php artisan vector:store {agent} [options] # Store content from a file php artisan vector:store customer_support --file=docs.txt # Store direct content php artisan vector:store customer_support --content="Important product information" # With metadata php artisan vector:store customer_support \ --file=manual.pdf \ --namespace=products \ --source=manual \ --source-id=v2.0 \ --metadata='{"category":"electronics","version":"2.0"}' ``` **Storage options:** | Option | Description | | ------------- | ------------------------ | | `--file` | Path to document or file | | `--content` | Direct text to store | | `--namespace` | Organize your memories | | `--source` | Track where it came from | | `--source-id` | Version or ID tracking | | `--metadata` | Extra context as JSON | Find anything instantly. Search through your agent's knowledge with semantic understanding. ```bash Terminal theme={null} php artisan vector:search {agent} {query} [options] # Basic search php artisan vector:search customer_support "refund policy" # Search with custom parameters php artisan vector:search customer_support "shipping rates" \ --limit=10 \ --threshold=0.8 \ --namespace=policies # Generate RAG context php artisan vector:search customer_support "product warranty" --rag # JSON output php artisan vector:search customer_support "user manual" --json ``` **Search options:** | Option | Description | | ------------- | -------------------------------- | | `--namespace` | Search specific memory spaces | | `--limit` | How many results to return | | `--threshold` | Similarity score (0.0-1.0) | | `--rag` | Get formatted context for agents | | `--json` | Machine-readable output | Monitor your memory usage. Get insights into how your vector storage is performing. ```bash Terminal theme={null} # Global statistics php artisan vector:stats # Agent-specific statistics php artisan vector:stats customer_support # Detailed statistics php artisan vector:stats --detailed # Specific namespace php artisan vector:stats customer_support --namespace=products # JSON output php artisan vector:stats --json ``` ## MCP Commands Manage Model Context Protocol servers. Connect your agents to external tool servers for extended capabilities. List and test your configured MCP servers. See which external tool servers are available and their connection status. ```bash Terminal theme={null} # List all configured MCP servers php artisan vizra:mcp:servers # Test connectivity to all servers php artisan vizra:mcp:servers --test ``` **Options:** | Option | Description | | -------- | ------------------------------------------- | | `--test` | Test connectivity to each configured server | **What you'll see:** * All configured MCP servers from your config * Server names, URLs, and transport types * Connection status when using `--test` * Available tools from each connected server ## Setup Commands Get started quickly. These commands help you set up and configure Vizra ADK. One command to rule them all. Install everything you need to start building agents. ```bash Terminal theme={null} php artisan vizra:install # This command: # - Publishes configuration files # - Publishes database migrations # - Shows post-installation information ``` Install Vizra ADK guidelines for Laravel Boost integration. This adds AI agent development context to your Boost configuration. ```bash Terminal theme={null} # Install Boost integration php artisan vizra:boost:install # Force overwrite existing configuration php artisan vizra:boost:install --force ``` **Options:** | Option | Description | | --------- | ----------------------------------- | | `--force` | Overwrite existing Boost guidelines | **What it does:** * Adds Vizra ADK context to your Laravel Boost setup * Provides AI coding assistants with agent development patterns * Includes tool and evaluation scaffolding guidance Access your command center. Launch the web dashboard to test and monitor your agents. ```bash Terminal theme={null} # Display dashboard URL php artisan vizra:dashboard # Open dashboard in browser php artisan vizra:dashboard --open ``` ## Pro Tips * Use `vizra:chat` for rapid testing * Create agents and tools with make commands * Debug with traces when things get complex * Run evaluations before deploying * Clean up traces to save storage * Monitor vector memory performance ## Next Steps Jump back to the basics and get started Explore all the APIs and capabilities # Error Handling Source: https://docs.vizra.ai/api-reference/error-handling Handle errors like a pro. The Vizra ADK provides a robust error handling system with specialized exceptions to help you build resilient AI agents that gracefully handle the unexpected. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). ## Exception Types The framework includes three specialized exception classes that extend PHP's base `\Exception` class: ### AgentNotFoundException Thrown when attempting to access or execute an agent that doesn't exist in the registry. ```php AgentRegistry.php theme={null} use Vizra\VizraADK\Exceptions\AgentNotFoundException; // Example from AgentRegistry::getAgent() if (!isset($this->registeredAgents[$name])) { throw new AgentNotFoundException("Agent '{$name}' is not registered."); } ``` ### AgentConfigurationException Thrown when there are issues with agent configuration, such as invalid settings or missing required parameters. ```php Configuration Validation theme={null} use Vizra\VizraADK\Exceptions\AgentConfigurationException; // Example usage if (!class_exists($config)) { throw new AgentConfigurationException( "Agent class '{$config}' does not exist." ); } ``` ### ToolExecutionException Thrown when errors occur during tool execution, including invalid parameters or runtime failures. ```php Tool Execution theme={null} use Vizra\VizraADK\Exceptions\ToolExecutionException; // Example in tool execution try { $result = $tool->execute($arguments, $context); } catch (\Exception $e) { throw new ToolExecutionException( "Tool '{$toolName}' failed: " . $e->getMessage() ); } ``` ## Handling Exceptions in Controllers The `AgentApiController` demonstrates comprehensive exception handling: ```php AgentApiController.php theme={null} use Vizra\VizraADK\Facades\Agent; use Vizra\VizraADK\Exceptions\AgentNotFoundException; use Vizra\VizraADK\Exceptions\ToolExecutionException; use Vizra\VizraADK\Exceptions\AgentConfigurationException; class AgentApiController extends Controller { public function handleAgentInteraction(Request $request): JsonResponse { try { // Check if agent exists if (!Agent::hasAgent($agentName)) { return response()->json([ 'error' => "Agent '{$agentName}' is not registered or found.", 'message' => "Please ensure the agent is registered..." ], 404); } // Execute agent $response = Agent::run($agentName, $input, $sessionId); return response()->json([ 'agent_name' => $agentName, 'session_id' => $sessionId, 'response' => $response, ]); } catch (AgentNotFoundException $e) { logger()->error("Agent not found: " . $e->getMessage()); return response()->json([ 'error' => "Agent '{$agentName}' could not be found or loaded.", 'detail' => $e->getMessage() ], 404); } catch (ToolExecutionException $e) { logger()->error("Tool execution error for agent {$agentName}: " . $e->getMessage(), [ 'exception' => $e ]); return response()->json([ 'error' => 'A tool required by the agent failed to execute.', 'detail' => $e->getMessage() ], 500); } catch (AgentConfigurationException $e) { logger()->error("Agent configuration error for agent {$agentName}: " . $e->getMessage(), [ 'exception' => $e ]); return response()->json([ 'error' => 'Agent configuration error.', 'detail' => $e->getMessage() ], 500); } catch (\Throwable $e) { // Catch-all for unexpected errors logger()->error("Error during agent '{$agentName}' execution: " . $e->getMessage(), [ 'exception' => $e ]); return response()->json([ 'error' => 'An unexpected error occurred while processing your request.', 'detail' => $e->getMessage() ], 500); } } } ``` ## Error Handling in Commands Artisan commands handle errors gracefully to provide helpful feedback: ```php RunEvalCommand.php theme={null} // Example from RunEvalCommand try { $evaluation = $this->loadEvaluation($evaluationName); $results = $evaluation->run(); } catch (\Exception $e) { $this->error("Evaluation failed: " . $e->getMessage()); return 1; // Return non-zero exit code } ``` ## Error Handling in Services Services like `AgentRegistry` validate configuration and throw appropriate exceptions: ```php AgentRegistry.php theme={null} public function getAgent(string $name): BaseAgent { if (!isset($this->registeredAgents[$name])) { throw new AgentNotFoundException("Agent '{$name}' is not registered."); } $config = $this->registeredAgents[$name]; if (is_string($config)) { if (!class_exists($config)) { throw new AgentConfigurationException( "Agent class '{$config}' does not exist." ); } $agent = new $config(); if (!$agent instanceof BaseAgent) { throw new AgentConfigurationException( "Agent class '{$config}' must extend BaseAgent." ); } return $agent; } // Handle array configuration... } ``` ## Best Practices ### 1. Use Specific Exception Types Always throw the most specific exception type for the error scenario: ```php Exception Best Practices theme={null} // Good - specific exception if (!$agent) { throw new AgentNotFoundException("Agent 'chatbot' not found"); } // Avoid - generic exception if (!$agent) { throw new \Exception("Agent not found"); } ``` ### 2. Provide Meaningful Error Messages Include context in error messages to help with debugging: ```php Descriptive Error Messages theme={null} throw new ToolExecutionException( "Weather tool failed for location '{$location}': API key not configured" ); ``` ### 3. Log Errors with Context Always log errors with relevant context for debugging: ```php Error Logging theme={null} catch (ToolExecutionException $e) { logger()->error("Tool execution failed", [ 'agent' => $agentName, 'tool' => $e->getToolName(), 'session_id' => $sessionId, 'exception' => $e ]); } ``` ### 4. Handle Errors at the Right Level Let exceptions bubble up to where they can be handled appropriately: ```php Error Handling Layers theme={null} // In a tool - let exception bubble up public function execute(array $arguments, AgentContext $context): string { if (!isset($arguments['location'])) { throw new ToolExecutionException("Location parameter is required"); } // Tool logic... } // In the controller - handle and return appropriate response try { $result = $agent->run($input); } catch (ToolExecutionException $e) { return response()->json(['error' => $e->getMessage()], 400); } ``` ## Creating Custom Exceptions You can create custom exceptions for your specific use cases: ```php app/Exceptions/ApiRateLimitException.php theme={null} service = $service; $this->retryAfter = $retryAfter; parent::__construct( "Rate limit exceeded for {$service}. Retry after {$retryAfter} seconds." ); } public function getRetryAfter(): int { return $this->retryAfter; } } ``` ## Error Recovery Strategies ### Graceful Degradation Provide fallback behavior when non-critical operations fail: ```php Fallback Pattern theme={null} public function execute(array $arguments, AgentContext $context): string { try { // Try to get weather from primary API $weather = $this->primaryApi->getWeather($location); } catch (ToolExecutionException $e) { logger()->warning("Primary weather API failed, using fallback", [ 'error' => $e->getMessage() ]); // Fallback to secondary API try { $weather = $this->fallbackApi->getWeather($location); } catch (ToolExecutionException $e) { // Return cached or default response return json_encode([ 'error' => 'Weather service temporarily unavailable', 'cached' => true, 'data' => $this->getCachedWeather($location) ]); } } return json_encode($weather); } ``` ### Retry Logic Implement retry logic for transient failures: ```php Retry with Exponential Backoff theme={null} use Illuminate\Support\Facades\Http; public function executeWithRetry(callable $operation, int $maxAttempts = 3) { $attempt = 1; while ($attempt <= $maxAttempts) { try { return $operation(); } catch (ToolExecutionException $e) { if ($attempt === $maxAttempts) { throw $e; } logger()->warning("Operation failed, retrying", [ 'attempt' => $attempt, 'max_attempts' => $maxAttempts, 'error' => $e->getMessage() ]); sleep(pow(2, $attempt)); // Exponential backoff $attempt++; } } } ``` ## Testing Error Handling Always test your error handling paths: ```php tests/Feature/ErrorHandlingTest.php theme={null} use Vizra\VizraADK\Exceptions\AgentNotFoundException; test('throws exception for non-existent agent', function () { $registry = app(AgentRegistry::class); expect(fn() => $registry->getAgent('non-existent')) ->toThrow(AgentNotFoundException::class, "Agent 'non-existent' is not registered."); }); test('handles tool execution failure gracefully', function () { $response = $this->postJson('/api/vizra/interact', [ 'agent_name' => 'test_agent', 'input' => 'trigger tool failure' ]); $response->assertStatus(500) ->assertJson([ 'error' => 'A tool required by the agent failed to execute.' ]); }); ``` ## Next Steps Agent class documentation Build custom tools Test your agents # Evaluation Class Reference Source: https://docs.vizra.ai/api-reference/evaluation-class Turn your agents into perfection machines. The BaseEvaluation class is your quality assurance superhero - test, validate, and polish your agents until they shine. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). **Looking for agent evals?** This page documents ADK's legacy CSV-based evaluation system. Its successor, [Vizra Evals](/), runs evals as Pest tests, records every run to a database, auto-baselines, and fails CI on regressions. Start with the [Quickstart](/evals/quickstart) or the [ADK migration guide](/evals/migrate-from-vizra-adk). ## Class Overview ```php theme={null} namespace Vizra\VizraADK\Evaluations; abstract class BaseEvaluation { // Your evaluation extends this class } ``` ## Properties | Property | Type | Required | Description | | ------------------ | ------ | -------- | --------------------------------------------------- | | `$agentName` | string | Yes | Agent alias to evaluate (e.g., 'customer\_support') | | `$name` | string | Yes | Human-readable evaluation name | | `$description` | string | Yes | Brief description of what this evaluation tests | | `$csvPath` | string | Yes | Path to CSV file relative to base\_path() | | `$promptCsvColumn` | string | No | CSV column containing prompts (default: 'prompt') | ## Abstract Methods ### preparePrompt() ```php theme={null} abstract public function preparePrompt(array $csvRowData): string ``` Prepares the prompt to be sent to the agent based on CSV row data. ```php theme={null} public function preparePrompt(array $csvRowData): string { // Basic implementation return $csvRowData[$this->getPromptCsvColumn()] ?? ''; // Or with context $prompt = $csvRowData['prompt']; if (isset($csvRowData['context'])) { $prompt = "Context: " . $csvRowData['context'] . "\n\n" . $prompt; } return $prompt; } ``` ### evaluateRow() ```php theme={null} abstract public function evaluateRow(array $csvRowData, string $llmResponse): array ``` Evaluates a single row of CSV data against the LLM's response using assertion methods. ```php theme={null} public function evaluateRow(array $csvRowData, string $llmResponse): array { // Reset assertions for this row $this->resetAssertionResults(); // Run assertions $this->assertResponseContains($llmResponse, 'expected'); $this->assertResponseHasPositiveSentiment($llmResponse); // Return structured results return [ 'row_data' => $csvRowData, 'llm_response' => $llmResponse, 'assertions' => $this->assertionResults, 'final_status' => 'pass' // or 'fail' ]; } ``` ## Content Assertion Methods ### Text Content Assertions ```php theme={null} // Check if response contains substring $this->assertResponseContains($response, 'expected text'); // Check if response does NOT contain substring $this->assertResponseDoesNotContain($response, 'unwanted'); // Regex pattern matching $this->assertResponseMatchesRegex($response, '/\d{3}-\d{4}/'); // Check start and end of response $this->assertResponseStartsWith($response, 'Hello'); $this->assertResponseEndsWith($response, '.'); // Check for multiple substrings $this->assertContainsAnyOf($response, ['yes', 'sure', 'okay']); $this->assertContainsAllOf($response, ['thank', 'you']); // Check if response is not empty $this->assertResponseIsNotEmpty($response); ``` ### Length and Size Assertions ```php theme={null} // Character length range $this->assertResponseLengthBetween($response, 100, 500); // Word count range $this->assertWordCountBetween($response, 20, 100); ``` ### Quality and Safety Assertions ```php theme={null} // Sentiment analysis $this->assertResponseHasPositiveSentiment($response); // Grammar and readability $this->assertGrammarCorrect($response); $this->assertReadabilityLevel($response, 12); // Max grade level $this->assertNoRepetition($response, 0.3); // Max repetition ratio // Content safety $this->assertNotToxic($response); $this->assertNotToxic($response, ['custom', 'bad', 'words']); $this->assertNoPII($response); // Spelling conventions $this->assertIsBritishSpelling($response); $this->assertIsAmericanSpelling($response); ``` ## Format and Structure Assertions ### JSON Validation ```php theme={null} // Check if response is valid JSON $this->assertResponseIsValidJson($response); // Check if JSON contains specific key $this->assertJsonHasKey($response, 'result'); ``` ### XML Validation ```php theme={null} // Check if response is valid XML $this->assertResponseIsValidXml($response); // Check if XML contains specific tag $this->assertXmlHasValidTag($response, 'result'); ``` ## Comparison Assertions ```php theme={null} // Equality checks $this->assertEquals('expected', $actual); $this->assertTrue($condition); $this->assertFalse($condition); // Numeric comparisons $this->assertGreaterThan(10, $value); $this->assertLessThan(100, $value); ``` ## LLM as Judge ### Pass/Fail Judge ```php theme={null} // Use LLM to evaluate pass/fail $this->judge($response) ->using(PassFailJudgeAgent::class) ->expectPass(); ``` ### Quality Scoring ```php theme={null} // Get quality score from LLM (0-10) $this->judge($response) ->using(QualityJudgeAgent::class) ->expectMinimumScore(7.5); ``` ### Multi-dimensional Evaluation ```php theme={null} // Evaluate multiple dimensions $this->judge($response) ->using(ComprehensiveJudgeAgent::class) ->expectMinimumScore([ 'accuracy' => 8, 'helpfulness' => 7, 'clarity' => 7 ]); ``` ## Helper Methods ### Protected Methods ```php theme={null} // Reset assertion results (called automatically) $this->resetAssertionResults(); // Get the CSV column name for prompts $columnName = $this->getPromptCsvColumn(); // Returns 'prompt' by default // Record custom assertion result $this->recordAssertion( 'customCheck', true, // status 'Custom check passed', 'expected', 'actual' ); ``` ### Assertion Results Structure ```php theme={null} // Each assertion returns an array with: [ 'assertion_method' => 'assertResponseContains', 'status' => 'pass' // or 'fail', 'message' => 'Response should contain substring.', 'expected' => 'expected text', 'actual' => 'actual response...' ] ``` ## Result Structure ### evaluateRow() Return Format ```php theme={null} // Your evaluateRow() method should return: [ 'row_data' => $csvRowData, // Original CSV row 'llm_response' => $llmResponse, // Agent's response 'assertions' => $this->assertionResults, // Array of assertion results 'final_status' => 'pass', // 'pass', 'fail', or 'error' 'error' => null // Optional error message ] ``` ## Complete Example ```php CustomerSupportEvaluation.php theme={null} getPromptCsvColumn()] ?? ''; // Add customer context if available if (isset($csvRowData['customer_type'])) { $prompt = "Customer Type: " . $csvRowData['customer_type'] . "\n\n" . $prompt; } return $prompt; } public function evaluateRow(array $csvRowData, string $llmResponse): array { $this->resetAssertionResults(); // Basic quality checks $this->assertResponseIsNotEmpty($llmResponse); $this->assertNotToxic($llmResponse); $this->assertNoPII($llmResponse); // Test-specific assertions based on scenario $scenario = $csvRowData['scenario'] ?? ''; switch ($scenario) { case 'greeting': $this->assertResponseHasPositiveSentiment($llmResponse); $this->assertContainsAnyOf($llmResponse, ['hello', 'hi', 'welcome']); break; case 'complaint': $this->assertResponseContains($llmResponse, 'sorry'); $this->judge($llmResponse) ->using(PassFailJudgeAgent::class) ->expectPass('Response should be empathetic and helpful'); break; case 'technical_support': $this->assertReadabilityLevel($llmResponse, 10); $this->assertGrammarCorrect($llmResponse); break; } // Check for expected content if specified if (isset($csvRowData['must_contain'])) { $requiredTerms = explode(',', $csvRowData['must_contain']); $this->assertContainsAllOf($llmResponse, $requiredTerms); } // Determine overall pass/fail $allPassed = collect($this->assertionResults) ->every(fn($result) => $result['status'] === 'pass'); return [ 'row_data' => $csvRowData, 'llm_response' => $llmResponse, 'assertions' => $this->assertionResults, 'final_status' => $allPassed ? 'pass' : 'fail', ]; } } ``` **CSV File Example:** ```csv theme={null} user_message,scenario,customer_type,must_contain "Hello, I need help",greeting,new,help "My order hasn't arrived",complaint,vip,"sorry,assist" "How do I reset my password?",technical_support,regular,"reset,password" ``` Structure your CSV files with clear columns for prompts, test scenarios, and expected outcomes. ## Next Steps Event reference documentation Learn more about evaluations # Events API Reference Source: https://docs.vizra.ai/api-reference/events Become an agent lifecycle ninja. Master every moment of agent execution with comprehensive event hooks. Monitor, debug, extend - turn your agents into observable, intelligent systems. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). ## Event Overview All class-based events are in the `Vizra\VizraADK\Events` namespace and use Laravel's event system. ```php theme={null} use Vizra\VizraADK\Events\AgentExecutionStarting; use Illuminate\Support\Facades\Event; // Listen to an event Event::listen(AgentExecutionStarting::class, function (AgentExecutionStarting $event) { Log::info('Agent starting', [ 'agent' => $event->agentName, 'input' => $event->input ]); }); ``` ## Agent Execution Events Fired when an agent begins execution. ```php theme={null} class AgentExecutionStarting { public function __construct( public AgentContext $context, public string $agentName, public mixed $input ) {} } ``` **Properties:** | Property | Description | | ------------ | ----------------------------------- | | `$context` | The agent execution context | | `$agentName` | Name of the agent being executed | | `$input` | Initial input provided to the agent | Fired when an agent completes execution. ```php theme={null} class AgentExecutionFinished { public function __construct( public AgentContext $context, public string $agentName ) {} } ``` **Properties:** | Property | Description | | ------------ | ------------------------------- | | `$context` | The agent execution context | | `$agentName` | Name of the agent that finished | Fired when an agent generates its final response. ```php theme={null} class AgentResponseGenerated { public function __construct( public AgentContext $context, public string $agentName, public mixed $finalResponse ) {} } ``` **Properties:** | Property | Description | | ---------------- | ----------------------------------------- | | `$context` | The agent execution context | | `$agentName` | Name of the agent | | `$finalResponse` | The final response generated by the agent | ## LLM Interaction Events Fired before making a call to the LLM. ```php theme={null} class LlmCallInitiating { public function __construct( public AgentContext $context, public string $agentName, public array $promptMessages ) {} } ``` **Properties:** | Property | Description | | ----------------- | --------------------------------------- | | `$context` | The agent execution context | | `$agentName` | Name of the agent making the call | | `$promptMessages` | Array of messages being sent to the LLM | Fired after receiving a response from the LLM. ```php theme={null} class LlmResponseReceived { public function __construct( public AgentContext $context, public string $agentName, public mixed $llmResponse, public ?PendingRequest $request = null ) {} } ``` **Properties:** | Property | Description | | -------------- | ----------------------------------- | | `$context` | The agent execution context | | `$agentName` | Name of the agent | | `$llmResponse` | The response from the LLM | | `$request` | The Prism request object (optional) | Fired when an LLM API call fails. ```php theme={null} class LlmCallFailed { public function __construct( public AgentContext $context, public string $agentName, public Throwable $exception, public ?PendingRequest $request = null ) {} } ``` **Properties:** | Property | Description | | ------------ | ------------------------------------- | | `$context` | The agent execution context | | `$agentName` | Name of the agent | | `$exception` | The exception that caused the failure | | `$request` | The failed request details (optional) | ## Tool Execution Events Fired before executing a tool. ```php theme={null} class ToolCallInitiating { public function __construct( public AgentContext $context, public string $agentName, public string $toolName, public array $arguments ) {} } ``` **Properties:** | Property | Description | | ------------ | ------------------------------------ | | `$context` | The agent execution context | | `$agentName` | Name of the agent executing the tool | | `$toolName` | Name of the tool being called | | `$arguments` | Arguments passed to the tool | Fired after a tool completes execution. ```php theme={null} class ToolCallCompleted { public function __construct( public AgentContext $context, public string $agentName, public string $toolName, public string $result // JSON string ) {} } ``` **Properties:** | Property | Description | | ------------ | --------------------------------- | | `$context` | The agent execution context | | `$agentName` | Name of the agent | | `$toolName` | Name of the tool that completed | | `$result` | JSON-encoded result from the tool | Fired when a tool call encounters an error. ```php theme={null} class ToolCallFailed { public function __construct( public AgentContext $context, public string $agentName, public string $toolName, public Throwable $exception ) {} } ``` **Properties:** | Property | Description | | ------------ | ------------------------------------- | | `$context` | The agent execution context | | `$agentName` | Name of the agent | | `$toolName` | Name of the tool that failed | | `$exception` | The exception that caused the failure | ## State Management Events Fired when agent state is updated. ```php theme={null} class StateUpdated { public function __construct( public AgentContext $context, public string $key, public mixed $value ) {} } ``` **Properties:** | Property | Description | | ---------- | ------------------------------ | | `$context` | The agent execution context | | `$key` | The state key that was updated | | `$value` | The new value | Fired when agent memory is updated. ```php theme={null} class MemoryUpdated { public function __construct( public AgentMemory $memory, public ?AgentSession $session, public string $updateType ) {} } ``` **Properties:** | Property | Description | | ------------- | --------------------------------- | | `$memory` | The agent memory instance | | `$session` | The associated session (optional) | | `$updateType` | Type of update performed | **Update Types:** | Type | Description | | --------------------- | ------------------------- | | `'session_completed'` | Session summary extracted | | `'learning_added'` | New learning stored | | `'fact_added'` | New fact stored | ## Multi-Agent Events Fired when an agent delegates a task to a sub-agent. ```php theme={null} class TaskDelegated { public function __construct( public AgentContext $parentContext, public AgentContext $subAgentContext, public string $parentAgentName, public string $subAgentName, public string $taskInput, public string $contextSummary, public int $delegationDepth ) {} } ``` **Properties:** | Property | Description | | ------------------ | ---------------------------------------- | | `$parentContext` | Context of the delegating agent | | `$subAgentContext` | Context for the sub-agent | | `$parentAgentName` | Name of the parent agent | | `$subAgentName` | Name of the sub-agent receiving the task | | `$taskInput` | The task being delegated | | `$contextSummary` | Summary of context passed to sub-agent | | `$delegationDepth` | Current depth of delegation chain | ## Media Generation Events These are string-based events fired by `MediaGenerationJob` when processing queued media generation tasks (images, audio, etc.). ```php theme={null} use Illuminate\Support\Facades\Event; // Listen to media events Event::listen('media.job.completed', function (array $payload) { Log::info('Media job completed', [ 'job_id' => $payload['job_id'], 'agent' => $payload['agent_class'], ]); }); ``` Fired when any media generation job completes successfully. ```php theme={null} event('media.job.completed', [ 'job_id' => $jobId, 'agent_class' => $agentClass, 'response' => $response, // ImageResponse or AudioResponse ]); ``` **Payload:** | Key | Type | Description | | ------------- | ------------------------------ | --------------------------------------------- | | `job_id` | `string` | Unique identifier for the job | | `agent_class` | `string` | Fully qualified class name of the media agent | | `response` | `ImageResponse\|AudioResponse` | The generated media response object | **Example:** ```php theme={null} Event::listen('media.job.completed', function (array $payload) { $url = $payload['response']->url(); // Notify user that their image is ready Notification::send($user, new MediaReadyNotification($url)); }); ``` Fired when a specific agent's media job completes. The event name includes the agent's name (e.g., `media.image_agent.completed`). ```php theme={null} event("media.{$agentName}.completed", [ 'job_id' => $jobId, 'response' => $response, 'session_id' => $sessionId, ]); ``` **Payload:** | Key | Type | Description | | ------------ | ------------------------------ | ----------------------------------- | | `job_id` | `string` | Unique identifier for the job | | `response` | `ImageResponse\|AudioResponse` | The generated media response object | | `session_id` | `string` | Session ID associated with the job | **Example:** ```php theme={null} // Listen specifically for image agent completions Event::listen('media.image_agent.completed', function (array $payload) { $sessionId = $payload['session_id']; $imageUrl = $payload['response']->url(); // Update the session with the generated image broadcast(new ImageGeneratedEvent($sessionId, $imageUrl)); }); // Listen for audio agent completions Event::listen('media.audio_agent.completed', function (array $payload) { // Handle audio-specific logic }); ``` Fired when a media generation job fails after all retry attempts. ```php theme={null} event('media.job.failed', [ 'job_id' => $jobId, 'agent_class' => $agentClass, 'error' => $exception->getMessage(), ]); ``` **Payload:** | Key | Type | Description | | ------------- | -------- | --------------------------------------------- | | `job_id` | `string` | Unique identifier for the failed job | | `agent_class` | `string` | Fully qualified class name of the media agent | | `error` | `string` | Error message describing the failure | **Example:** ```php theme={null} Event::listen('media.job.failed', function (array $payload) { Log::error('Media generation failed', [ 'job_id' => $payload['job_id'], 'agent' => $payload['agent_class'], 'error' => $payload['error'], ]); // Notify user of failure Notification::send($user, new MediaFailedNotification($payload['error'])); }); ``` ## Usage Examples ### Monitoring Agent Performance ```php theme={null} use Vizra\VizraADK\Events\{AgentExecutionStarting, AgentExecutionFinished}; use Illuminate\Support\Facades\Event; class AgentPerformanceMonitor { private array $startTimes = []; public function register(): void { Event::listen(AgentExecutionStarting::class, [$this, 'handleStart']); Event::listen(AgentExecutionFinished::class, [$this, 'handleFinish']); } public function handleStart(AgentExecutionStarting $event): void { $this->startTimes[$event->context->getSessionId()] = microtime(true); } public function handleFinish(AgentExecutionFinished $event): void { $sessionId = $event->context->getSessionId(); $duration = microtime(true) - $this->startTimes[$sessionId]; Log::info('Agent execution completed', [ 'agent' => $event->agentName, 'duration' => round($duration, 2) . 's' ]); } } ``` ### Debugging Tool Calls ```php theme={null} Event::listen(ToolCallInitiating::class, function (ToolCallInitiating $event) { Log::debug('Tool call starting', [ 'tool' => $event->toolName, 'arguments' => $event->arguments, 'session' => $event->context->getSessionId() ]); }); Event::listen(ToolCallCompleted::class, function (ToolCallCompleted $event) { $result = json_decode($event->result, true); Log::debug('Tool call completed', [ 'tool' => $event->toolName, 'success' => $result['success'] ?? false ]); }); ``` ### Memory Tracking ```php theme={null} Event::listen(MemoryUpdated::class, function (MemoryUpdated $event) { switch ($event->updateType) { case 'learning_added': Log::info('New learning stored', [ 'agent' => $event->memory->agent_name, 'learning' => $event->memory->learnings->last() ]); break; case 'fact_added': Log::info('New fact stored', [ 'agent' => $event->memory->agent_name, 'fact' => $event->memory->facts->last() ]); break; } }); ``` **Event Best Practices:** * Use event listeners for cross-cutting concerns like logging and monitoring * Keep event handlers lightweight to avoid impacting agent performance * Use queued listeners for heavy processing tasks * Events are synchronous by default - be mindful of execution time * All class-based events include the AgentContext for accessing session state * Tool results are JSON strings - decode them for processing * Media events are string-based and receive array payloads ## Next Steps Learn about the agent execution flow View all API references # Tool Class Reference Source: https://docs.vizra.ai/api-reference/tool-class Unleash your agent's superpowers. The ToolInterface is your gateway to giving agents incredible abilities. From database wizardry to API mastery - build tools that make agents unstoppable. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). Think of the `ToolInterface` as your agent's superhero training academy. Every tool you build expands what your agents can do - from simple calculations to complex integrations. ## The Simple Yet Powerful Interface The beauty of Vizra tools lies in their simplicity. Just two methods to implement, and your agent gains a whole new superpower. Here's the elegant contract: ```php The ToolInterface Contract theme={null} namespace Vizra\VizraADK\Contracts; use Vizra\VizraADK\System\AgentContext; use Vizra\VizraADK\Memory\AgentMemory; interface ToolInterface { public function definition(): array; public function execute(array $arguments, AgentContext $context, AgentMemory $memory): string; } ``` ## The Two Magical Methods Just two methods stand between you and tool mastery. Each serves a crucial purpose in making your agent interactions seamless and powerful. **The blueprint method.** Tell the AI exactly how to use your tool with a JSON Schema definition. It's like writing a user manual that AI can understand perfectly. ```php theme={null} public function definition(): array ``` **The action hero.** This is where the magic happens - take the AI's parameters and make something amazing occur. Database queries, API calls, calculations - you name it. ```php theme={null} public function execute(array $arguments, AgentContext $context, AgentMemory $memory): string ``` ### definition() - Teaching AI How to Use Your Tool This method returns a JSON Schema that's like a detailed instruction manual for AI. Here's how to write one that makes your tool shine: ```php Perfect Tool Definition theme={null} public function definition(): array { return [ 'name' => 'order_lookup', 'description' => 'Look up order information by ID', 'parameters' => [ 'type' => 'object', 'properties' => [ 'order_id' => [ 'type' => 'string', 'description' => 'The order ID to look up', ], 'include_items' => [ 'type' => 'boolean', 'description' => 'Include order items in response', ], ], 'required' => ['order_id'], ], ]; } ``` ### execute() ```php theme={null} public function execute(array $arguments, AgentContext $context, AgentMemory $memory): string ``` Execute the tool with provided arguments, context, and agent memory. Must return a JSON-encoded string. ```php theme={null} public function execute(array $arguments, AgentContext $context): string { $order = Order::find($arguments['order_id']); if (!$order) { return json_encode([ 'status' => 'error', 'message' => 'Order not found', ]); } return json_encode([ 'status' => 'success', 'order' => $order->toArray(), ]); } ``` ## Parameter Schema Examples ### Basic Parameter Types ```php theme={null} 'parameters' => [ 'type' => 'object', 'properties' => [ 'name' => [ 'type' => 'string', 'description' => 'User name', ], 'age' => [ 'type' => 'integer', 'description' => 'User age', 'minimum' => 0, 'maximum' => 150, ], 'active' => [ 'type' => 'boolean', 'description' => 'Whether user is active', ], 'price' => [ 'type' => 'number', 'description' => 'Product price', 'minimum' => 0, ], ], 'required' => ['name'], ] ``` ### Complex Parameter Types ```php theme={null} 'parameters' => [ 'type' => 'object', 'properties' => [ 'status' => [ 'type' => 'string', 'enum' => ['pending', 'processing', 'completed'], 'description' => 'Order status', ], 'tags' => [ 'type' => 'array', 'description' => 'List of tags', 'items' => [ 'type' => 'string', ], ], 'address' => [ 'type' => 'object', 'description' => 'User address', 'properties' => [ 'street' => ['type' => 'string'], 'city' => ['type' => 'string'], 'zip' => ['type' => 'string'], ], ], ], 'required' => [], ] ``` ## Return Value Format Tools must return a JSON-encoded string. Common patterns include: ```php theme={null} // Success response return json_encode([ 'status' => 'success', 'data' => $resultData, 'message' => 'Operation completed successfully', ]); // Error response return json_encode([ 'status' => 'error', 'message' => 'Detailed error message', 'error_code' => 'SPECIFIC_ERROR', ]); ``` ## Working with AgentContext The `AgentContext` parameter provides access to session state and conversation history: ```php theme={null} public function execute(array $arguments, AgentContext $context): string { // Get session ID $sessionId = $context->getSessionId(); // Access state $previousValue = $context->getState('last_order_id'); // Store state for future use $context->setState('last_order_id', $orderId); // Access user information if available $userId = $context->getState('user_id'); return json_encode(['status' => 'success']); } ``` ## Advanced Features ### Tool with User Context ```php theme={null} class UserOrdersTool implements ToolInterface { public function execute(array $arguments, AgentContext $context): string { // Get user ID from context $userId = $context->getState('user_id'); if (!$userId) { return json_encode([ 'status' => 'error', 'message' => 'User authentication required', ]); } $user = User::find($userId); $orders = $user->orders() ->latest() ->limit($arguments['limit'] ?? 10) ->get(); return json_encode([ 'status' => 'success', 'orders' => $orders->toArray(), ]); } } ``` ### Tool with Session Context ```php theme={null} public function execute(array $arguments, AgentContext $context): string { // Access session context $previousOrder = $context->getState('last_order_discussed'); // Store data in session $context->setState('current_order', $order->id); return json_encode([ 'status' => 'success', 'order' => $order->toArray(), ]); } ``` ### Tool with File Handling ```php theme={null} class FileProcessorTool implements ToolInterface { public function definition(): array { return [ 'name' => 'file_processor', 'description' => 'Process uploaded files', 'parameters' => [ 'type' => 'object', 'properties' => [ 'file_content' => [ 'type' => 'string', 'description' => 'Base64 encoded content', ], 'filename' => [ 'type' => 'string', ], ], 'required' => ['file_content', 'filename'], ], ]; } public function execute(array $arguments, AgentContext $context): string { $content = base64_decode($arguments['file_content']); $path = Storage::put('uploads/' . $arguments['filename'], $content); return json_encode([ 'status' => 'success', 'path' => $path, 'size' => strlen($content), ]); } } ``` ## Complete Example ```php RefundProcessorTool.php theme={null} 'process_refund', 'description' => 'Process a refund for an order', 'parameters' => [ 'type' => 'object', 'properties' => [ 'order_id' => [ 'type' => 'string', 'description' => 'The order ID to refund', ], 'amount' => [ 'type' => 'number', 'description' => 'Refund amount (optional for partial refunds)', 'minimum' => 0, ], 'reason' => [ 'type' => 'string', 'description' => 'Reason for refund', ], ], 'required' => ['order_id', 'reason'], ], ]; } public function execute(array $arguments, AgentContext $context): string { // Validate inputs $validator = Validator::make($arguments, [ 'order_id' => 'required|string', 'amount' => 'nullable|numeric|min:0', 'reason' => 'required|string', ]); if ($validator->fails()) { return json_encode([ 'status' => 'error', 'message' => 'Validation failed: ' . $validator->errors()->first(), ]); } // Check user authorization $userId = $context->getState('user_id'); if (!$userId) { return json_encode([ 'status' => 'error', 'message' => 'User authentication required', ]); } try { $order = Order::findOrFail($arguments['order_id']); // Verify order belongs to user if ($order->user_id !== $userId) { return json_encode([ 'status' => 'error', 'message' => 'Order not found', ]); } $refund = $this->refunds->process( $order, $arguments['amount'] ?? $order->total, $arguments['reason'] ); return json_encode([ 'status' => 'success', 'refund_id' => $refund->id, 'amount' => $refund->amount, 'refund_status' => $refund->status, 'message' => 'Refund processed successfully', ]); } catch (\Exception $e) { return json_encode([ 'status' => 'error', 'message' => 'Failed to process refund: ' . $e->getMessage(), ]); } } } ``` **Tool Best Practices:** * Keep tools focused on a single action * Implement the ToolInterface, not extend a base class * Use descriptive names and descriptions * Return JSON-encoded strings from execute() * Use AgentContext for session state management * Validate parameters thoroughly * Handle errors gracefully * Return clear success/error messages ## Next Steps Workflow class reference Learn more about tools # Tracing Classes API Reference Source: https://docs.vizra.ai/api-reference/tracing-class Complete API reference for the TraceSpan model and Tracer service that power ADK's tracing system. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). This page documents the TraceSpan model and Tracer service that power the observability features in Vizra ADK. ## TraceSpan Model The `TraceSpan` model represents an individual span within an agent execution trace. ```php theme={null} namespace Vizra\VizraADK\Models; class TraceSpan extends Model { use HasUlids; } ``` ### Properties | Property | Type | Description | | ---------------- | ------------------- | ------------------------------------------------------------------------- | | `id` | string (ULID) | Primary key for this span | | `trace_id` | string (ULID) | Identifies the entire agent run trace | | `parent_span_id` | string\|null | Parent span ID for hierarchy | | `span_id` | string (ULID) | Unique ID for this specific span | | `session_id` | string | Session ID from AgentContext | | `agent_name` | string | Name of the agent executing this span | | `type` | string | Operation type: agent\_run, llm\_call, tool\_call, sub\_agent\_delegation | | `name` | string | Specific name (model name, tool name, etc.) | | `input` | array\|null | Input data for the operation | | `output` | array\|null | Result/output of the operation | | `metadata` | array\|null | Additional contextual information | | `status` | string | Execution status: running, success, error | | `error_message` | string\|null | Error message if status is error | | `start_time` | decimal(16,6) | Start timestamp with microseconds | | `end_time` | decimal(16,6)\|null | End timestamp with microseconds | | `duration_ms` | integer\|null | Duration in milliseconds | ### Relationships ```php theme={null} // Get the parent span $parentSpan = $span->parent(); // Get child spans $childSpans = $span->children(); // Get all spans in the same trace $traceSpans = $span->traceSpans(); ``` ### Query Scopes ```php theme={null} // Filter by trace ID $spans = TraceSpan::forTrace($traceId)->get(); // Filter by session ID $spans = TraceSpan::forSession($sessionId)->get(); // Filter by type $llmCalls = TraceSpan::ofType('llm_call')->get(); // Filter by status $errors = TraceSpan::withStatus('error')->get(); // Get root spans only $rootSpans = TraceSpan::rootSpans()->get(); // Order chronologically $orderedSpans = TraceSpan::chronological()->get(); ``` ### Helper Methods ```php theme={null} // Check if span is completed if ($span->isCompleted()) { echo "Span finished in " . $span->duration_ms . "ms"; } // Check if span has error if ($span->hasError()) { echo "Error: " . $span->error_message; } // Check if root span if ($span->isRoot()) { echo "This is the root span"; } // Get formatted duration echo $span->getFormattedDuration(); // "125ms" or "1.5s" // Get status icon echo $span->getStatusIcon(); // Check, X, or spinner icon // Get type icon echo $span->getTypeIcon(); // Robot, brain, wrench, or users icon // Get summary echo $span->getSummary(); // "llm_call: gpt-4 - 1.2s (success)" ``` ## Tracer Service The `Tracer` service manages span creation and lifecycle. ```php theme={null} namespace Vizra\VizraADK\Services; class Tracer { // Service is typically injected via dependency injection } ``` ### Creating Spans ```php theme={null} // Start a new span $spanId = $tracer->startSpan( type: 'tool_call', name: 'database_query', input: ['query' => 'SELECT * FROM users'], metadata: ['database' => 'mysql'] ); // End the span successfully $tracer->endSpan($spanId, [ 'rows_returned' => 42, 'cache_hit' => false ]); // Mark span as failed $tracer->failSpan($spanId, $exception); ``` ### Span Types | Type | Description | | ---------------------- | ----------------------------- | | `agent_run` | Root span for agent execution | | `llm_call` | LLM API calls | | `tool_call` | Tool executions | | `sub_agent_delegation` | Sub-agent invocations | ### Trace Management ```php theme={null} // Create a new trace $traceId = $tracer->createTrace( sessionId: $context->getSessionId(), agentName: 'customer_support' ); // Set current trace context $tracer->setCurrentTrace($traceId); // Get current trace ID $currentTraceId = $tracer->getCurrentTraceId(); ``` ### Cleanup Operations ```php theme={null} // Get count of old traces $count = $tracer->getOldTracesCount(30); // 30 days // Clean up old traces with progress callback $deleted = $tracer->cleanupOldTraces(30, function($batchCount) { echo "Deleted {$batchCount} traces...\n"; }); ``` ## Usage Examples ### Analyzing Trace Performance ```php theme={null} use Vizra\VizraADK\Models\TraceSpan; // Find slowest operations in a trace $slowestSpans = TraceSpan::forTrace($traceId) ->where('duration_ms', '>', 1000) ->orderBy('duration_ms', 'desc') ->get(); foreach ($slowestSpans as $span) { echo $span->getSummary() . "\n"; } ``` ### Error Analysis ```php theme={null} // Get all errors for a session $errors = TraceSpan::forSession($sessionId) ->withStatus('error') ->with('parent') ->get(); foreach ($errors as $error) { echo "Error in {$error->name}: {$error->error_message}\n"; // Show parent context if ($error->parent) { echo " Parent: {$error->parent->name}\n"; } } ``` ### Building Trace Trees ```php theme={null} // Build hierarchical trace tree function buildTraceTree(string $traceId): array { $spans = TraceSpan::forTrace($traceId) ->with('children') ->chronological() ->get(); $tree = []; $spanMap = []; // First pass: create span map foreach ($spans as $span) { $spanMap[$span->span_id] = $span; } // Second pass: build tree foreach ($spans as $span) { if ($span->isRoot()) { $tree[] = $span; } } return $tree; } ``` **Best Practices:** * Always end spans properly to ensure accurate duration tracking * Use descriptive span names for better trace readability * Include relevant metadata for debugging purposes * Fail spans with exceptions to capture error context * Use appropriate span types for automatic categorization * Implement regular cleanup to manage storage ## Next Steps Learn about tracing concepts View all API references # Workflow Class Reference Source: https://docs.vizra.ai/api-reference/workflow-class Orchestrate agents like a conductor. The BaseWorkflowAgent turns you into a master of AI coordination - chain agents together, create complex flows, and build intelligent automation pipelines. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). ## Class Overview ```php theme={null} namespace Vizra\VizraADK\Agents; abstract class BaseWorkflowAgent extends BaseAgent { // Workflow agents extend this class } ``` ## BaseWorkflowAgent Properties | Property | Type | Default | Description | | ---------------- | ----- | ------- | -------------------------- | | `$steps` | array | `[]` | Array of workflow steps | | `$results` | array | `[]` | Step execution results | | `$timeout` | int | `300` | Workflow timeout (seconds) | | `$retryAttempts` | int | `0` | Default retry attempts | | `$retryDelay` | int | `1000` | Retry delay (milliseconds) | ## Key Methods ### executeWorkflow() ```php theme={null} abstract protected function executeWorkflow(mixed $input, AgentContext $context): mixed ``` Each workflow type must implement this method to define its execution logic. ### addAgent() ```php theme={null} public function addAgent( string $agentClass, mixed $params = null, array $options = [] ): static ``` Add an agent step to the workflow using the agent class name. ## Workflow Methods ### Configuration Methods ```php theme={null} // Set timeout for the entire workflow $workflow->timeout(300); // 5 minutes // Set retry policy $workflow->retryOnFailure(3, 1000); // 3 attempts, 1s delay // Set callbacks $workflow->onSuccess(function($result, $stepResults) { Log::info('Success!', ['result' => $result]); }); $workflow->onFailure(function(\Throwable $e, $stepResults) { Log::error('Failed: ' . $e->getMessage()); }); $workflow->onComplete(function($result, $success, $stepResults) { // Always runs }); ``` ### Result Access Methods ```php theme={null} // Get all results $allResults = $workflow->getResults(); // Get specific step result $stepResult = $workflow->getStepResult(DataProcessor::class); // Reset workflow for reuse $workflow->reset(); ``` ## Workflow Types ### Sequential Workflow ```php theme={null} use Vizra\VizraADK\Agents\SequentialWorkflow; use App\Agents\AnalyzerAgent; use App\Agents\ProcessorAgent; use App\Agents\ValidatorAgent; use App\Agents\CleanupAgent; $workflow = new SequentialWorkflow(); // Add steps in sequence $workflow ->start(AnalyzerAgent::class, 'Analyze input') ->then(ProcessorAgent::class, fn($prev) => "Process: " . $prev) ->when(ValidatorAgent::class, fn($input, $results) => $results[ProcessorAgent::class]['status'] === 'needs_validation' ) ->finally(CleanupAgent::class); // Always runs // Create context and execute use Vizra\VizraADK\System\AgentContext; $context = new AgentContext('session-123'); $result = $workflow->execute('Input data', $context); ``` ### Parallel Workflow ```php theme={null} use Vizra\VizraADK\Agents\ParallelWorkflow; use App\Agents\EmailSender; use App\Agents\SmsSender; use App\Agents\WebhookCaller; $workflow = new ParallelWorkflow(); $workflow ->agents([ EmailSender::class => 'Send email notification', SmsSender::class => 'Send SMS', WebhookCaller::class => 'Call webhook' ]) ->waitForAll(); // or waitForAny() ``` ### Conditional Workflow ```php theme={null} use Vizra\VizraADK\Agents\ConditionalWorkflow; use App\Agents\UrgentHandler; use App\Agents\SupportAgent; use App\Agents\DefaultHandler; $workflow = new ConditionalWorkflow(); $workflow ->when( fn($input) => $input['priority'] === 'high', UrgentHandler::class ) ->when( fn($input) => $input['type'] === 'support', SupportAgent::class ) ->otherwise(DefaultHandler::class); ``` ### Loop Workflow ```php theme={null} use Vizra\VizraADK\Agents\LoopWorkflow; use App\Agents\ProcessorAgent; use App\Agents\ItemProcessor; // While loop $workflow = new LoopWorkflow(); $workflow->while( ProcessorAgent::class, fn($result) => $result['continue'] === true, 10 // max iterations ); // For each loop $items = ['item1', 'item2', 'item3']; $workflow = new LoopWorkflow(); $workflow->forEach(ItemProcessor::class, $items); ``` ## Step Parameters ### Dynamic Parameters ```php theme={null} // Static parameters $workflow->addAgent('agent_name', 'Static input'); // Dynamic parameters with closure $workflow->addAgent( 'processor', fn($input, $results, $context) => [ 'data' => $results['previous_agent'], 'mode' => $context->getState('processing_mode') ] ); // Step options $workflow->addAgent('risky_agent', 'Input', [ 'retries' => 5, 'timeout' => 60, 'condition' => fn() => config('app.env') === 'production' ]); ``` ## Creating Custom Workflows ```php theme={null} use App\Agents\FirstAgent; class CustomWorkflow extends BaseWorkflowAgent { protected string $name = 'custom_workflow'; protected string $description = 'Custom workflow implementation'; protected function executeWorkflow(mixed $input, AgentContext $context): mixed { // Custom workflow logic $result1 = $this->executeStep([ 'agent' => 'first_agent', 'params' => $input, 'retries' => 2 ], $input, $context); // Access previous results $allResults = $this->getResults(); return [ 'workflow_type' => 'custom', 'results' => $allResults ]; } } ``` ## Running Workflows ### Basic Execution ```php theme={null} use Vizra\VizraADK\System\AgentContext; // Workflows are agents, so they run like any agent $workflow = new SequentialWorkflow(); $workflow ->then('agent1') ->then('agent2'); // Create context (required) $context = new AgentContext('session-123'); // Execute with input and context $result = $workflow->execute('Process this data', $context); ``` ### Using the Workflow Facade ```php theme={null} use Vizra\VizraADK\Facades\Workflow; use Vizra\VizraADK\System\AgentContext; // Create context $context = new AgentContext('session-123'); // Create and execute in one go $result = Workflow::sequential('agent1', 'agent2', 'agent3') ->execute('Input data', $context); // With callbacks $workflow = Workflow::sequential('agent1', 'agent2') ->onSuccess(function($result) { Log::info('Workflow completed!', ['result' => $result]); }); $result = $workflow->execute('Start', $context); ``` ## Workflow Results ### Sequential Workflow Results ```json theme={null} // Sequential workflow returns { "final_result": "Last agent output", "step_results": { "agent1": "First result", "agent2": "Second result" }, "workflow_type": "sequential" } ``` ### Parallel Workflow Results ```json theme={null} // Parallel workflow returns { "results": { "agent1": "Result 1", "agent2": "Result 2" }, "completed": ["agent1", "agent2"], "workflow_type": "parallel" } ``` ### Error Handling ```php theme={null} // Set retry policy for all steps $workflow->retryOnFailure(3, 1000); // Handle errors with callbacks $workflow->onFailure(function(\Throwable $e, $stepResults) { // Log the error Log::error('Workflow failed', [ 'error' => $e->getMessage(), 'completed_steps' => array_keys($stepResults) ]); // Perform cleanup CleanupService::rollback($stepResults); }); ``` ## Complete Example ```php OrderProcessingWorkflow.php theme={null} timeout(300) // 5 minutes ->retryOnFailure(2, 2000); // 2 retries, 2s delay // Define the workflow steps $this ->start(OrderValidator::class, fn($input) => [ 'order_id' => $input['order_id'] ]) ->then(InventoryChecker::class, fn($prevResult) => $prevResult['items'] ) ->when( PaymentProcessor::class, fn($input, $results) => $results[InventoryChecker::class]['available'] === true, fn($input, $results) => [ 'amount' => $results[OrderValidator::class]['total'], 'payment_method' => $input['payment_method'] ] ) ->then(ShippingCreator::class) ->then(NotificationSender::class, 'Send order confirmation') ->finally(OrderLogger::class); // Always runs // Set callbacks $this->onSuccess(function($result, $stepResults) { Log::info('Order processed successfully', [ 'order_id' => $stepResults[OrderValidator::class]['order_id'] ]); }); $this->onFailure(function(\Throwable $e, $stepResults) { // Rollback any completed steps if (isset($stepResults[PaymentProcessor::class])) { PaymentService::refund( $stepResults[PaymentProcessor::class]['transaction_id'] ); } }); } } ``` **Workflow Best Practices:** * Workflows are agents - they extend BaseWorkflowAgent * Use the appropriate workflow type for your use case * Set reasonable timeouts and retry limits * Leverage callbacks for monitoring and debugging * Pass results between steps using closures * Handle errors at both step and workflow levels * Test workflows with various input scenarios ## Next Steps Evaluation class reference Learn more about workflows # Connect a Project Source: https://docs.vizra.ai/cloud/connect Create a project, mint a key, add two lines to .env — and your next run appears. A **project** groups the eval suites of one codebase. An **API key** belongs to exactly one project, and runs reported with it are filed there. ## 1. Create the project From the sidebar's `+`, or **Projects → New project**. Name it after the application, not the agent — one Laravel app with six agents is one project. ## 2. Create a key New projects show their connection steps immediately, including a key created for you: A new project's connect panel: composer require, the two env lines, and the run command

A new project's connect panel: composer require, the two env lines, and the run command

The full key is shown **once**, when it is created. Only a sha256 hash is stored, so we genuinely cannot show it to you again — not "won't". If you lose it, create another. ## 3. Add it to your environment ```bash .env theme={null} VIZRA_CLOUD_KEY=vz_your_key_here ``` That is the whole configuration on the hosted service. `VIZRA_CLOUD_ENDPOINT` defaults to `https://vizra.ai/api/v1/runs` and only needs setting if you are pointing at a self-hosted instance. ## 4. Run your evals ```bash Terminal theme={null} ./vendor/bin/pest --evals # or php artisan evals:run ``` The run finishes, reports, and appears in the project. Nothing else changes — the same run is still recorded locally, and the gate still decides pass or fail exactly as before. See [what gets reported](/cloud/reporting). ## Managing keys The API keys tab, showing masked keys with their project, last use, expiry and rotate/revoke actions

The API keys tab, showing masked keys with their project, last use, expiry and rotate/revoke actions

Each key shows its project, when it was created, whether it has **ever been used**, and its expiry. "Never used" is usually the answer to "why aren't my runs showing up". ### Expiry Choose **Never**, **30 days**, **90 days** or **a year** when creating a key. An expired key stops being accepted at ingest; runs still record locally. ### Rotate, don't revoke **Rotate** creates a replacement and keeps the old key working for **7 days**, so you can deploy the new one everywhere before the old one stops. Rotation never *extends* an expiry that was already sooner than the grace period. **Revoke** stops the key immediately. Worth knowing what that looks like from the other side: a rejected upload never fails a build, so anything still using the key goes quiet rather than erroring — you notice by the missing runs, not by a red pipeline. Give CI its own key, named for the pipeline. When you rotate it you know exactly what has to be redeployed, and revoking it doesn't stop anyone's laptop reporting. # Editing Datasets Source: https://docs.vizra.ai/cloud/datasets Fix a test question in the browser, run the variant, and commit it back to your repository. The people who know whether an answer is good are often not the people who can edit a JSONL file. Cloud lets them propose a change, validate it against a real run, and hand the developer something to commit. The dataset editor: input and expected per row, with disable and add-a-row controls

The dataset editor: input and expected per row, with disable and add-a-row controls

## Where the dataset comes from It is derived from the samples your runs already reported — no import step, no new payload. The first version anyone edits is the one your eval actually ran. This needs sample detail, so a project running [`VIZRA_CLOUD_SAMPLES=false`](/cloud/reporting#sending-less) has nothing to derive from and editing is unavailable. ## Editing creates a version, not a change Editing opens a **draft**. The current version stays exactly as it was until you publish, so the runs that used it still mean what they said. This is not caution for its own sake. A row's identity is a hash of its content — input, prior messages and expected value. Change the expected answer and it becomes a *different row*, because every past score was graded against a different expectation. Keeping the same identity would splice two incompatible gradings into one trend line. So a run of an edited dataset is not compared to a baseline that used the old one. The run page says so plainly rather than showing a delta that means nothing: > Different dataset to the baseline — no overall comparison. This run used dataset v3. > 1 row is unchanged and still compared below. Unchanged rows still line up, and are still compared. That is the only place a comparison means anything. ## Rows are disabled, not deleted Deleting is the one operation where a well-meaning edit can quietly destroy coverage, and unlike an edit it leaves nothing behind to show what used to be tested. Disabled rows are excluded from runs, stay visible, and are restored with one click. ## Getting it back into your repository **Your repository stays canonical.** An edit that only ever lives in Cloud is a fork of your test data, so the workflow ends with a file you commit: 1. **Download JSONL** — the exact format `Dataset::fromJsonl()` reads. 2. Commit it. Your next ordinary run — CI, terminal, anywhere — uses it without knowing any of this happened. 3. **Mark as committed**, which records that the version is now in the repo. Disabled rows are excluded from the export, so committing it removes them from your repository. That is a real deletion arriving by the back door, and the page says so before you download. The export refuses outright when it cannot write the dataset faithfully — a multi-turn row's earlier turns never reach Cloud, so the file would be missing part of it. Handing over a plausible file that quietly loses data is worse than refusing; edit those in the repository instead. ## Why not just let Cloud own the dataset? Because then Cloud becomes a runtime dependency of your eval suite, and our outage becomes your broken build. Storing variants and handing you a file to merge costs one step and buys you a review — which is also what stops someone deleting half the test set. # Vizra Cloud Source: https://docs.vizra.ai/cloud/index Your team's eval history in one place — laptop runs, CI runs, and the ability to run a suite from the browser. Vizra Cloud is the hosted counterpart to `vizra/evals`. Same engine, same tables, same scores — it just keeps them somewhere your whole team can see, across every environment your evals run in. The Vizra Cloud overview: suites needing attention, then recent runs across every project

The Vizra Cloud overview: suites needing attention, then recent runs across every project

## Local recording happens either way This is the part worth being precise about, because it decides whether you need Cloud at all: | | When it happens | Where it lands | | ------------------- | ----------------------------------- | -------------------------------------------------------------- | | **Local recording** | Always. Every run, no configuration | Your own database → the `/evals` [dashboard](/evals/dashboard) | | **Cloud recording** | Only when an API key is configured | vizra.ai | Nothing about installing `vizra/evals` sends us anything. Runs persist to your tables and the local dashboard reads them, key or no key. Cloud is additive: set [`VIZRA_CLOUD_KEY`](/cloud/connect) and finished runs are *also* pushed to your project. Nothing executes on our side. Your evals run in your app, against your models, with your keys. Cloud stores what your runs produced — which is why it never becomes a runtime dependency of your test suite, and why an outage on our end cannot break your build. ## What the key actually buys * **One history across every environment.** A run from your laptop, a run from CI and a run someone triggered from the browser all land in the same suite, tagged with the [environment](/cloud/reporting#environments) they came from — so "does this pass on main" and "does this pass on my branch" stop being the same number. * **Your team's runs, not just yours.** The local dashboard shows what your machine did. Cloud shows what everyone's did. * **Triage first.** The overview leads with suites that fell below their baseline or broke outright, worst first, across every project — rather than leaving you to click into each one and read the score. * **CI context.** Runs carry the build URL, branch and pull-request number, detected automatically from GitHub, GitLab, CircleCI or Buildkite. * **[Dataset editing](/cloud/datasets).** Change a test question in the browser, run the variant, and export it back to your repository — without booking a developer's afternoon. * **[Running a suite from the dashboard](/cloud/triggering)**, on your own infrastructure. ## Every tier records Runs are never metered and reporting is never rate-limited — on any plan, including the free one. The [plans](/cloud/plans) differ on projects, seats and how long per-sample detail is kept, not on whether your evals are allowed to report. Project, key, `.env` — and your next run appears. Which surfaces push, what's in the payload, and how to send less. Fix a test question in the browser, commit it back to the repo. Limits, and exactly what ages out. # Plans & Retention Source: https://docs.vizra.ai/cloud/plans What each tier includes, and exactly what ages out of your history. Three tiers. The levers are **projects** and **retention** — not seats, and not runs. | | Free | Pro | Agency | | ---------------- | --------- | --------- | --------- | | **Price** | \$0 | \$29/mo | \$99/mo | | Eval runs | Unlimited | Unlimited | Unlimited | | Projects | 1 | 5 | Unlimited | | Seats | 2 | Unlimited | Unlimited | | Sample retention | 14 days | 90 days | 12 months | | Support | | | Priority | Prices and limits are read from one config on our side and rendered on the pricing page, the product page and your billing tab — so they cannot disagree with what you are charged. ## What is not metered **Runs.** Evals are bursty and CI-shaped, and metering them would show you a frightening number on the exact day you are evaluating us. Reporting is never rate-limited and no tier restricts it — see [Reporting](/cloud/reporting). **Seats.** Unlimited from the first paid tier. Per-seat pricing punishes the agency with six developers and one client project, which is precisely the shape of team this is for. ## Retention deletes less than you think This is the part most worth reading carefully, because the natural assumption is wrong. **Only per-sample detail ages out.** Runs, their per-row scores, and baselines are kept **forever**, on every tier including free. | Kept forever | Aged out at the retention window | | -------------------------------- | ----------------------------------- | | The run, its score and pass rate | Verbatim model input | | Per-row scores and comparisons | Response text and structured output | | Baselines and trend history | Tool calls, usage, judge reasoning | "Is this worse than it was three months ago" is the entire value of the product — deleting the run would delete the answer. What ages out is the verbatim model input and output hanging off it, which is both the bulk of the storage and the whole of the compliance surface. In practice: on the free tier your trend line still goes back to your first run a year later. What you lose after 14 days is the ability to open a row from back then and read what the model actually said. ## Projects The project limit is enforced when you create one — you will be told which plan you are on and what it includes, rather than the form silently failing. Existing projects are never removed if you downgrade. ## Changing plan **Settings → Billing.** Only the team owner can change a plan; members see why they cannot rather than a button that fails. Billing runs through Stripe, and the customer portal handles cards, invoices and cancellation. # Reporting Runs Source: https://docs.vizra.ai/cloud/reporting Which surfaces push to Cloud, what's in the payload, and why a flaky uplink can never fail your build. With [a key configured](/cloud/connect), a finished run is pushed to Cloud automatically. There is no flag to remember and no separate cloud package — reporting is built into `vizra/evals`. ## Which surfaces report | Surface | Reports | | ---------------------------------------------------------------- | --------- | | `./vendor/bin/pest --evals` | Yes | | `php artisan evals:run` | Yes | | `php artisan evals:runner` (the [Run button](/cloud/triggering)) | Yes | | `php artisan evals:run --dry-run` | **Never** | | Any eval marked `->report(false)` | **Never** | Dry runs are excluded deliberately: their numbers come from faked agents and would sit in the history looking exactly like real ones. ## Reporting cannot change a run's outcome A run that passed its gate passed it whether or not the network was up. So every failure in the reporter is returned as a message to print, never thrown: * One retry, because the common failure is a dropped connection. * A `4xx` is never retried — sending the same rejected document again cannot start working. * A failed push writes a line to stderr and nothing else. A flaky uplink cannot turn a green build red, and reporting successfully cannot turn a failed gate green. **Failing runs are reported too**, before the test is failed. A run that got worse is the one you most need in the dashboard — a history of only your successes would answer no useful question. ## Excluding a wiring test Evals that [fake the agent](/evals/testing-without-tokens) produce real-looking scores from canned responses. Keep them out of the history: ```php theme={null} expect(SupportBot::class)->toPassEval(fn ($eval) => $eval ->dataset(base_path('evals/support.jsonl')) ->report(false) // wiring test, not measurement ->assert(fn ($a) => $a->notEmpty()) ); ``` From the CLI, `--no-report` does the same for one run. `--report` forces a warning when no key is configured, which is useful in CI to catch a missing secret. ## What is sent The run's metadata (suite, status, git sha, branch, timestamps, config snapshot), the aggregate summary, every row's scores — and, unless you turn them off, the samples: response text, structured output, tool calls, token usage, finish reason, cost, and every assertion's expected/actual plus the judge's reasoning. A run in Cloud: score, dataset-comparison notice, per-row results and Copy for AI

A run in Cloud: score, dataset-comparison notice, per-row results and Copy for AI

### Sending less ```bash .env theme={null} VIZRA_CLOUD_SAMPLES=false ``` You keep scores, pass rates, trends and baselines. You lose the drill-down, the judge's reasoning, **Copy for AI**, and [dataset editing](/cloud/datasets) — which derives from reported samples and has nothing to work from without them. Worth weighing honestly: the sample detail is both the bulk of the storage and the whole of the compliance surface. If your eval inputs contain customer data, this is the switch. ## Environments Runs are filed under an environment so a laptop run and a CI run are not the same number. * Anything running in CI is `ci`, regardless of `APP_ENV` — CI boxes almost always set `APP_ENV=testing`, and filing every CI run under "testing" would bucket them with someone's local test run. * Otherwise the app's environment is used. * Override with `VIZRA_CLOUD_ENVIRONMENT`. GitHub Actions, GitLab CI, CircleCI and Buildkite are detected automatically from the variables they set, so each run carries its build URL, branch and pull-request number with nothing to configure. ## Configuration ```php config/evals.php theme={null} 'cloud' => [ 'endpoint' => env('VIZRA_CLOUD_ENDPOINT', 'https://vizra.ai/api/v1/runs'), 'key' => env('VIZRA_CLOUD_KEY'), 'samples' => env('VIZRA_CLOUD_SAMPLES', true), 'timeout' => env('VIZRA_CLOUD_TIMEOUT', 15), 'environment' => env('VIZRA_CLOUD_ENVIRONMENT'), 'auto_schedule' => env('VIZRA_CLOUD_AUTO_SCHEDULE', true), ], ``` `auto_schedule` is covered in [Running from the dashboard](/cloud/triggering). # Running from the Dashboard Source: https://docs.vizra.ai/cloud/triggering A Run button in the browser that executes on your infrastructure — via your scheduler, or your CI. Suite pages have a **Run now** button. Pressing it does not run anything on our servers — we have no access to your models, your keys or your data. It records a request, and something you control picks it up. A suite page in Cloud with the Run now button and the runner's last check-in

A suite page in Cloud with the Run now button and the runner's last check-in

The line under the button — *runner last seen 13 hours ago* — is how you know a request will actually be collected. If it says the runner has never checked in, pressing Run will queue something nothing is listening for. ## Path 1: your scheduler The default, and the one to use if your app is deployed anywhere with a cron. With a key configured, the package registers `evals:runner` on your scheduler for you — every minute, `withoutOverlapping()`, riding the cron that already runs `schedule:run`. It polls for requested runs, executes them locally and reports the results back. Nothing to add. If you would rather register it yourself: ```bash .env theme={null} VIZRA_CLOUD_AUTO_SCHEDULE=false ``` ```php routes/console.php theme={null} Schedule::command('evals:runner')->everyMinute()->withoutOverlapping(); ``` This needs `vizra/evals` installed as a regular dependency rather than `--dev`, because the runner has to exist in the deployed application. ## Path 2: CI dispatch For teams whose evals only ever run in CI — no long-running app, nothing to schedule. Connect a GitHub repository in the project's settings, and Run now triggers a `workflow_dispatch` on a workflow you nominate. Your existing pipeline runs the suite with its own secrets and reports through the same endpoint. You supply four things: repository, workflow file, ref (branch or tag), and a token. The token needs **Actions: write** on that one repository, and nothing else. Do *not* use a classic token with the `repo` scope — that is write access to every repository the grantor can see, handed to a third party, to press one button. The workflow must accept `workflow_dispatch`: ```yaml .github/workflows/evals.yml theme={null} on: workflow_dispatch: pull_request: paths: ['app/Agents/**', 'evals/**'] ``` ### When dispatch fails GitHub returns **404** for a missing repository, a missing workflow *and* a token without permission — the same response for all three, deliberately, so a token cannot be used to probe for private repositories. Unhelpful when it is your own repo, so Cloud spells out the possibilities rather than passing the bare status through. Check, in order: the workflow file exists on the ref you named, it declares `workflow_dispatch`, and the token has Actions: write. ## Choosing | | Use the scheduler | Use CI dispatch | | ------------------------- | ----------------- | ------------------ | | App deployed with a cron | ✓ | | | Evals only ever run in CI | | ✓ | | Wants results in seconds | ✓ (next minute) | (a pipeline start) | | Nothing to configure | ✓ | needs a token | Both are available on every plan. The run executes on your infrastructure either way. # AI Agents Source: https://docs.vizra.ai/concepts/agents Meet your new AI workforce. Agents are the heart and soul of Vizra ADK - intelligent, conversational, and incredibly powerful. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). ## What's an Agent? Think of agents as your **smart AI assistants** that can handle complex tasks, remember conversations, and even work with other agents. They're not just chatbots - they're intelligent systems that can analyze data, make decisions, and take actions. Remembers conversations, understands context, and learns from interactions Uses custom tools to access databases, APIs, and perform real actions Works with OpenAI, Anthropic Claude, and Google Gemini models Agents can delegate tasks to other specialized agents Built-in semantic search and RAG capabilities with public API access Public methods make testing and experimentation a breeze ## Creating Your First Agent Ready to build your first AI agent? It's easier than you think! Let's create a helpful customer support agent that can answer questions and solve problems. ### Quick Start with Artisan Fire up your terminal and run this magical command: ```bash Terminal theme={null} php artisan vizra:make:agent CustomerSupportAgent ``` **Boom!** You've just created your first agent! Let's peek inside and see what makes it tick: ### The Agent Blueprint ```php app/Agents/CustomerSupportAgent.php expandable theme={null} **One Method to Rule Them All** - The `run()` method is your gateway to agent intelligence. Simple, powerful, and flexible - it handles everything from conversations to complex data processing. ### Let's Chat! Using the Fluent API Working with agents feels natural with our fluent API. Check out these examples: ```php theme={null} // Basic conversational usage $response = CustomerSupportAgent::run('How do I reset my password?') ->go(); // With user context $response = CustomerSupportAgent::run('Show me my recent orders') ->forUser($user) ->go(); // With session for conversation continuity $response = CustomerSupportAgent::run('What was my previous question?') ->withSession($sessionId) ->go(); // Event-driven execution OrderProcessingAgent::run($orderCreatedEvent) ->async() ->go(); // Data analysis with context $insights = AnalyticsAgent::run($salesData) ->withContext(['period' => 'last_quarter']) ->go(); // Process large datasets asynchronously DataProcessorAgent::run($largeDataset) ->async() ->onQueue('processing') ->go(); // Monitor system health with thresholds SystemMonitorAgent::run($metrics) ->withContext(['threshold' => 0.95]) ->go(); // Generate reports with specific formats $report = ReportAgent::run('weekly_summary') ->withContext(['format' => 'pdf']) ->go(); ``` ### Real-time Magic with Streaming Want to see your agent think in real-time? Enable streaming for that ChatGPT-like experience: ```php theme={null} // Enable streaming for real-time responses $stream = StorytellerAgent::run('Tell me a story') ->streaming() ->go(); foreach ($stream as $chunk) { echo $chunk; flush(); } ``` ### Managing Conversation History **Important:** All messages are always saved to the database for session continuity. These settings only control what previous messages are sent to the LLM for context. By default, agents don't send conversation history to the LLM (for better performance and lower costs). But for chat agents, you'll want context! Here's how to control what history gets sent to the LLM: ```php theme={null} // Enable history for conversational agents class ChatAgent extends BaseLlmAgent { protected bool $includeConversationHistory = true; // Send history to LLM protected string $contextStrategy = 'recent'; // 'none', 'recent', 'full' protected int $historyLimit = 10; // last 10 messages (for 'recent' strategy) } // Override at runtime $context->setState('include_history', true); $context->setState('history_depth', 5); ``` ## Customizing Agent Behavior Want to add your own special sauce? Agents come with powerful lifecycle hooks that let you customize exactly how they work. It's like having backstage passes to your agent's brain! ### Available Hooks * **beforeLlmCall** - Tweak messages before they hit the AI * **afterLlmResponse** - Process AI responses your way * **beforeToolCall** - Modify tool inputs on the fly * **afterToolResult** - Transform tool outputs * **onToolException** - Handle tool execution errors Here's how to use these superpowers: ```php theme={null} class CustomAgent extends BaseLlmAgent { public function beforeLlmCall(array $inputMessages, AgentContext $context): array { // Modify messages before sending to LLM // This is also where tracing starts return $inputMessages; } public function afterLlmResponse(Response|Generator $response, AgentContext $context, ?PendingRequest $request = null): mixed { // Process the LLM response // Access token usage: $response->usage // Access original request: $request return $response; } public function beforeToolCall(string $toolName, array $arguments, AgentContext $context): array { // Modify tool arguments before execution return $arguments; } public function afterToolResult(string $toolName, string $result, AgentContext $context): string { // Process tool results return $result; } public function onToolException(string $toolName, Throwable $e, AgentContext $context): void { // Handle tool execution errors // Log errors, send alerts, or implement recovery strategies } } ``` ## Dynamic Prompts Want to test different personalities without changing code? Prompt versioning lets you A/B test, switch between tones, and evolve your agent's voice on the fly! **Switch Prompts at Runtime** - No more hardcoding prompts! Store different versions and switch between them instantly. ```php theme={null} // Use a specific prompt version $response = CustomerSupportAgent::run('Help me with my order') ->withPromptVersion('friendly') ->go(); // A/B test different tones $version = rand(0, 1) ? 'professional' : 'casual'; $response = CustomerSupportAgent::run($query) ->withPromptVersion($version) ->go(); ``` Store prompts as `.md` files in `resources/prompts/{agent_name}/` for easy version control! [Learn more about Dynamic Prompts](/concepts/dynamic-prompts) ## Dynamic Prompts with Blade Templates Create **dynamic, context-aware prompts** using Laravel's Blade templating engine. Your prompts can adapt based on user data, session state, and custom variables. ### Quick Example Save your prompt as `.blade.php` to enable dynamic content: ```blade resources/prompts/agent_name/default.blade.php theme={null} You are {{ $agent['name'] }}, a helpful assistant. @if(isset($user_name)) Hello {{ $user_name }}! How can I help you today? @else Hello! How can I assist you? @endif @if($context && $context->getState('premium_user')) Premium support mode activated @endif @if($tools->isNotEmpty()) I can help you with: {{ $tools->pluck('description')->join(', ') }} @endif ``` ### Adding Custom Variables Inject your own data by implementing `getPromptData()` in your agent: ```php theme={null} class CustomerSupportAgent extends BaseLlmAgent { protected function getPromptData(AgentContext $context): array { return [ 'company_name' => config('app.company_name'), 'support_hours' => '9 AM - 5 PM EST', 'ticket_count' => $this->getUserTicketCount($context), ]; } } ``` Then use them in your template: ```blade theme={null} Welcome to {{ $company_name }} support! Our hours: {{ $support_hours }} @if($ticket_count > 0) You have {{ $ticket_count }} open tickets. @endif ``` [Learn more about Blade Templates & Advanced Features](/concepts/dynamic-prompts) ## SubAgents: Building Agent Teams Why have one agent when you can have a whole team? **SubAgents** let you create specialized agents that work together, with a manager agent delegating tasks to the right specialist. Think of it as building your own AI company! ### What Are SubAgents? SubAgents are specialized agents that a parent agent can delegate tasks to. When you define subAgents on an agent, Vizra ADK automatically adds the `DelegateToSubAgentTool` - allowing your manager agent to intelligently route requests to the right specialist. ```php theme={null} class CustomerServiceManager extends BaseLlmAgent { protected string $name = 'customer_service_manager'; protected string $instructions = "You are a customer service manager. Route technical questions to the technical support agent. Route billing questions to the billing agent. Handle general inquiries yourself."; protected array $subAgents = [ TechnicalSupportAgent::class, BillingSupportAgent::class, SalesAgent::class, ]; } ``` ### How SubAgent Delegation Works When a user asks a question, the manager agent decides whether to handle it directly or delegate: ``` User: "My payment failed" ↓ Manager Agent analyzes the request ↓ Decides: "This is a billing issue" ↓ Delegates to BillingSupportAgent ↓ BillingSupportAgent handles the request ↓ Response returned to user ``` ### Creating Specialized SubAgents Each subAgent should be focused on a specific domain: ```php theme={null} class TechnicalSupportAgent extends BaseLlmAgent { protected string $name = 'technical_support'; protected string $description = 'Handles technical issues, bugs, and integration problems'; protected string $instructions = "You are a technical support specialist. Help users with technical issues, debugging, and integration problems. Be patient and thorough in your explanations."; protected array $tools = [ LogLookupTool::class, SystemStatusTool::class, ]; } class BillingSupportAgent extends BaseLlmAgent { protected string $name = 'billing_support'; protected string $description = 'Handles billing, payments, and subscription questions'; protected string $instructions = "You are a billing specialist. Help users with payment issues, invoices, and subscription management."; protected array $tools = [ InvoiceLookupTool::class, RefundProcessorTool::class, ]; } ``` **Pro Tip:** Write clear `description` properties on your subAgents - the manager agent uses these to decide which specialist to delegate to! ### SubAgent Benefits Each agent focuses on what it does best SubAgents can have their own unique tools Use GPT-4 for complex tasks, GPT-3.5 for simple ones Test each specialist agent independently ## Advanced Agent Techniques Ready to level up? Here are some pro tips and advanced features that'll make your agents work harder and smarter! ### Vector Memory & RAG Give your agents superpowers with built-in semantic search and knowledge retrieval! Perfect for building documentation assistants, knowledge bases, and intelligent Q\&A systems. ```php Agent with Vector Memory theme={null} class DocumentationAgent extends BaseLlmAgent { protected string $name = 'docs_agent'; public function loadKnowledge(string $content): void { // Simple: just add content $this->vector()->addDocument($content); // Advanced: with full metadata and organization $this->vector()->addDocument([ 'content' => $content, 'metadata' => ['type' => 'docs', 'version' => '2.0'], 'namespace' => 'knowledge_base', 'source' => 'user_upload' ]); } public function answerQuestion(string $question, AgentContext $context): mixed { // Generate contextual answer using RAG $ragContext = $this->rag()->generateRagContext($question, [ 'namespace' => 'knowledge_base', 'limit' => 5, 'threshold' => 0.7 ]); if ($ragContext['total_results'] > 0) { $contextualInput = "Based on this knowledge:\n" . $ragContext['context'] . "\n\nAnswer: " . $question; } else { $contextualInput = $question; } return parent::run($contextualInput, $context); } } // Public access means you can test directly: $agent = Agent::named('docs_agent'); $agent->vector()->addDocument('Laravel is awesome!'); $results = $agent->vector()->search('framework'); ``` **Vector Memory Pro Tips** * Use progressive API: simple strings for prototypes, arrays for production * Public methods make testing in Tinkerwell super easy * Organize with namespaces: 'docs', 'faqs', 'policies', etc. * Perfect for building chatbots that remember context ### Background Processing with Async Got heavy lifting to do? Send your agents to work in the background: ```php theme={null} // Execute agent asynchronously via queue $job = DataProcessorAgent::run($largeDataset) ->async() ->onQueue('processing') ->go(); // With delay and retries $job = ReportAgent::run('quarterly_report') ->delay(300) // 5 minutes ->tries(3) ->timeout(600) // 10 minutes ->go(); ``` ### Vision & Multimodal Magic Your agents aren't just text wizards - they have **eyes** too! Send images, documents, and watch the magic happen: ```php theme={null} // Send images with your request $response = VisionAgent::run('What\'s in this image?') ->withImage('/path/to/image.jpg') ->go(); // Multiple images and documents $response = DocumentAnalyzer::run('Summarize these documents') ->withDocument('/path/to/report.pdf') ->withImage('/path/to/chart.png') ->withImageFromUrl('https://example.com/diagram.jpg') ->go(); ``` ### Fine-Tune on the Fly Need more creativity? Want faster responses? **Override any parameter** at runtime: ```php theme={null} // Override agent parameters at runtime $response = CreativeWriterAgent::run('Write a poem') ->temperature(0.9) // More creative ->maxTokens(500) ->go(); // Set multiple parameters $response = AnalyticalAgent::run($data) ->withParameters([ 'temperature' => 0.2, 'max_tokens' => 2000, 'top_p' => 0.95 ]) ->go(); ``` ## Memory That Actually Remembers Unlike that friend who forgets your birthday every year, your agents have **perfect memory**! They remember everything important about your conversations and context. Every message in the session What tools did and returned Who they're talking to Any context you provide ```php theme={null} // Add custom context $response = ShoppingAssistant::run('Find me a laptop') ->withContext([ 'budget' => 1500, 'preferences' => ['brand' => 'Apple'], 'location' => 'New York' ]) ->go(); ``` ## Understanding the Agent Lifecycle When you ask an agent to do something, it goes through a **carefully orchestrated lifecycle**. Understanding this flow helps you build more powerful agents and debug issues faster! ### The Request Journey ``` 1. Your Code → AgentExecutor (Fluent API) 2. AgentExecutor → AgentManager (Orchestration) 3. AgentManager → Agent Instance (Your Logic) 4. Agent → LLM Provider (AI Processing) 5. LLM → Tools (If Needed) 6. Response → Back Through the Chain ``` ### Lifecycle Hooks - Your Power Points Agents provide strategic hooks where you can intercept and modify behavior. These are your **control points** for customization: #### beforeLlmCall() - Pre-Processing Called before sending messages to the AI. Perfect for adding context, filtering messages, or injecting system prompts. ```php theme={null} public function beforeLlmCall(array $messages, AgentContext $context): array { // Add user preferences to system message $messages[0]['content'] .= "\nUser prefers formal tone."; return $messages; } ``` #### afterLlmResponse() - Post-Processing Called after receiving the AI's response. Transform responses, extract insights, or trigger side effects. Now includes the original request for complete logging. ```php theme={null} public function afterLlmResponse(Response $response, AgentContext $context, ?PendingRequest $request): mixed { // Log token usage and request details for monitoring Log::info('LLM call completed', [ 'model' => $request?->model(), 'usage' => $response->usage ]); return $response; } ``` #### Tool Execution Hooks Control tool execution with beforeToolCall() and afterToolResult() hooks. ```php theme={null} public function beforeToolCall(string $toolName, array $arguments, AgentContext $context): array { // Add API keys or validate permissions if ($toolName === 'weather_api') { $arguments['api_key'] = config('services.weather.key'); } return $arguments; } ``` ### Lifecycle Events As your agent works, it broadcasts events you can listen to for monitoring, logging, or triggering other actions: | Event | Description | | ------------------------ | --------------------- | | `AgentExecutionStarting` | When execution begins | | `LlmCallInitiating` | Before calling the AI | | `ToolCallInitiating` | Before using a tool | | `ToolCallCompleted` | After tool finishes | | `LlmResponseReceived` | AI response received | | `AgentExecutionFinished` | All done! | ## Pro Tips from the Trenches Want to build agents that users **actually love**? Here's the wisdom we've gathered from building hundreds of agents: Write instructions like you're explaining to a smart friend. Be specific about what you want! GPT-4 for complex reasoning, GPT-3.5 for quick tasks. Don't use a Ferrari to go to the corner store! Use lifecycle hooks for debugging and monitoring. You'll thank yourself later! Complex workflows? Use sub-agents! Let specialists handle what they do best. Enable streaming for long responses. Users love seeing agents "think" in real-time! ## Ready to Build Something Amazing? You've got the knowledge, now let's put it to work! Give your agents superpowers with custom tools Every method, property, and secret revealed # Architecture Source: https://docs.vizra.ai/concepts/architecture Understand the technical architecture of Vizra ADK - how agents, tools, context, and services work together **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). This page explains the internal architecture of Vizra ADK, helping you understand how the various components interact to power your AI agents. ## Package Structure ``` vizra-adk/src/ ├── Agents/ # Agent base classes and workflows ├── Contracts/ # Interfaces (ToolInterface, etc.) ├── Console/Commands/ # Artisan commands ├── Events/ # Laravel events for agent lifecycle ├── Exceptions/ # Custom exception classes ├── Execution/ # AgentExecutor for fluent execution ├── Facades/ # Agent and Workflow facades ├── Memory/ # Memory management ├── Models/ # Eloquent models (sessions, messages, traces) ├── Providers/ # Service providers and embedding providers ├── Services/ # Core services (Manager, Registry, Tracer) ├── System/ # AgentContext └── Tools/ # Built-in tools and MCP wrapper ``` ## Core Components ### Agent Hierarchy ``` BaseAgent (abstract) ├── BaseLlmAgent (abstract) │ └── GenericLlmAgent (for ad-hoc definitions) │ └── BaseWorkflowAgent (abstract) ├── SequentialWorkflow ├── ParallelWorkflow ├── ConditionalWorkflow └── LoopWorkflow ``` **BaseAgent** defines the core contract - every agent must have a `name`, `description`, and implement `execute()`. **BaseLlmAgent** extends this with LLM-specific features: system prompts, tool registration, model selection, and conversation handling. **BaseWorkflowAgent** provides orchestration capabilities without LLM overhead - pure PHP logic for complex multi-agent coordination. ### Service Container Bindings Vizra ADK registers these singletons in Laravel's service container: | Service | Purpose | | --------------- | ----------------------------------------- | | `AgentRegistry` | Stores and retrieves agent class mappings | | `AgentBuilder` | Fluent builder for configuring agents | | `AgentManager` | Central facade for running agents | | `StateManager` | Loads and persists agent context | | `MemoryManager` | Manages conversation history | | `Tracer` | Records execution spans for debugging | Access via dependency injection or the `Agent` facade: ```php theme={null} use Vizra\VizraADK\Facades\Agent; // Via facade $agent = Agent::named('customer_support'); // Via container $manager = app(AgentManager::class); ``` ## Request Lifecycle When you run an agent, here's what happens: ``` ┌─────────────────────────────────────────────────────────────────┐ │ 1. AgentExecutor::run($input) │ │ └─> Builds execution configuration (user, session, params) │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ 2. AgentExecutor::go() │ │ └─> Resolves session ID, loads/creates AgentContext │ │ └─> Dispatches AgentExecutionStarting event │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ 3. AgentManager::run() │ │ └─> Gets agent from AgentRegistry │ │ └─> Calls agent->execute($input, $context) │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ 4. BaseLlmAgent::execute() │ │ └─> Builds system prompt │ │ └─> Sends to LLM via Prism PHP │ │ └─> Handles tool calls in a loop │ │ └─> Returns final response │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ 5. StateManager::saveContext() │ │ └─> Persists conversation history │ │ └─> Saves state to database │ │ └─> Dispatches AgentExecutionFinished event │ └─────────────────────────────────────────────────────────────────┘ ``` ## AgentContext `AgentContext` is the carrier object that flows through the entire execution: ```php theme={null} class AgentContext { protected ?string $sessionId; protected mixed $userInput; protected array $state = []; protected Collection $conversationHistory; } ``` It provides: * **Session tracking** - Links conversations across requests * **State storage** - Key-value store for execution data * **Conversation history** - Full message history for context Tools receive the context and can read/write state: ```php theme={null} public function execute(array $arguments, AgentContext $context, AgentMemory $memory): string { // Read state $userId = $context->getState('user_id'); // Write state $context->setState('last_order_id', $orderId); return json_encode(['success' => true]); } ``` ## Tool Execution Flow When an LLM decides to call a tool: ``` ┌──────────────────────────────────────┐ │ LLM Response with tool_calls │ │ [{ name: "lookup_order", args: {}}] │ └──────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────┐ │ ToolCallInitiating Event │ │ (for logging/tracing) │ └──────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────┐ │ Tool::execute($args, $context) │ │ Returns JSON string result │ └──────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────┐ │ ToolCallCompleted Event │ │ Result sent back to LLM │ └──────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────┐ │ Loop continues until LLM │ │ returns final text response │ └──────────────────────────────────────┘ ``` ## Event System Vizra ADK fires Laravel events at key points: | Event | When Fired | | ------------------------ | ----------------------------- | | `AgentExecutionStarting` | Before agent execution begins | | `AgentExecutionFinished` | After execution completes | | `AgentResponseGenerated` | When final response is ready | | `LlmCallInitiating` | Before each LLM API call | | `LlmResponseReceived` | After LLM responds | | `ToolCallInitiating` | Before tool execution | | `ToolCallCompleted` | After tool returns | | `ToolCallFailed` | When tool throws exception | | `MemoryUpdated` | When memory is modified | | `StateUpdated` | When context state changes | Listen to these events for logging, analytics, or custom behavior: ```php theme={null} // In EventServiceProvider protected $listen = [ AgentExecutionFinished::class => [ LogAgentMetrics::class, ], ]; ``` ## Database Schema Vizra ADK creates these tables: | Table | Purpose | | ----------------------- | ------------------------------------- | | `agent_sessions` | Tracks conversation sessions | | `agent_messages` | Stores message history | | `agent_memories` | Persistent memory storage | | `agent_trace_spans` | Execution traces for debugging | | `agent_vector_memories` | Vector embeddings for RAG | | `agent_prompt_versions` | Prompt versioning | | `agent_prompt_usage` | Tracks which prompt versions are used | ## LLM Integration Vizra ADK uses [Prism PHP](https://prismphp.org) for LLM communication: ``` Your Agent │ ▼ Vizra ADK (BaseLlmAgent) │ ▼ Prism PHP │ ├──▶ OpenAI ├──▶ Anthropic ├──▶ Google Gemini └──▶ Ollama ``` Configure providers in `config/vizra-adk.php`: ```php theme={null} 'providers' => [ 'openai' => [ 'api_key' => env('OPENAI_API_KEY'), ], 'anthropic' => [ 'api_key' => env('ANTHROPIC_API_KEY'), ], ], ``` ## Extending the Architecture ### Custom Service Providers Register custom services that integrate with Vizra ADK: ```php theme={null} class MyAgentServiceProvider extends ServiceProvider { public function boot() { // Register custom tools globally Agent::macro('withAnalytics', function ($tracker) { // Custom behavior return $this; }); } } ``` ### Custom Base Agents Create your own base agent with shared behavior: ```php theme={null} abstract class MyBaseAgent extends BaseLlmAgent { protected function getSystemPrompt(): string { return "Company-wide instructions...\n\n" . $this->getAgentSpecificPrompt(); } abstract protected function getAgentSpecificPrompt(): string; } ``` ## Next Steps * [Agents](/concepts/agents) - Learn how to build agents * [Tools](/concepts/tools) - Give agents capabilities * [Sessions & Memory](/concepts/sessions-memory) - Understand persistence # Dynamic Prompts Source: https://docs.vizra.ai/concepts/dynamic-prompts Master your AI's voice with dynamic prompts. Track improvements over time with versions, test different personalities with variants, and evolve your agents without touching code. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). ## Why Dynamic Prompts Change Everything Remember changing code just to tweak your AI's personality? Those days are OVER! With dynamic prompts, you can experiment, test, and perfect your agent's responses without deploying a single line of code. Test different prompts side-by-side Revert to any previous version See which prompts work best ## Versions vs Variants: What's the Difference? There's an important distinction between **versions** and **variants**. Vizra ADK supports **BOTH** seamlessly! Track **improvements over time**. Each version builds on the last, getting better and smarter. ``` v1: "You are a helpful assistant" v2: "You are a helpful assistant. Be concise." v3: "You are a helpful assistant. Be concise and cite sources." ``` Use for: A/B testing improvements, safe rollbacks, tracking prompt evolution Different **styles for different contexts**. Same capability, different personality. ``` formal: "I shall assist you with utmost professionalism" casual: "Hey! Happy to help out" technical: "Initiating support protocol. State your query." ``` Use for: User preferences, context-specific responses, brand voices ### The Magic: Use Both Together! **File Structure:** ``` resources/prompts/support_agent/ ├── v1/ │ ├── default.md │ ├── formal.md │ └── casual.md └── v2/ ├── default.md ├── formal.md └── casual.md ``` **Usage Examples:** ```php theme={null} // Use latest version, default variant SupportAgent::run('Help')->go(); // Use specific variant SupportAgent::run('Help') ->withPromptVersion('formal') ->go(); // Use specific version & variant SupportAgent::run('Help') ->withPromptVersion('v1/casual') ->go(); ``` ## Getting Started with Dynamic Prompts When you create an agent using the `vizra:make:agent` command, it automatically creates a prompt blade file for you in `/resources/prompts/YourAgent`. This gives you instant access to dynamic prompts without any additional setup! **Quick Example** ```php theme={null} // Use a variant MyAgent::run('Hello') ->withPromptVersion('brief') ->go(); // Use a specific version/variant MyAgent::run('Hello') ->withPromptVersion('v2/formal') ->go(); ``` ## Real-World Example: Customer Support Agent Let's see how versions and variants work together in practice. Imagine evolving a customer support agent over time while maintaining different tones for different customers. ```php Usage in Production theme={null} // A/B test new version with specific users $version = $user->isInTestGroup() ? 'v3' : 'v2'; $variant = $user->preferredTone ?? 'default'; $response = CustomerSupportAgent::run($query) ->withPromptVersion("$version/$variant") ->forUser($user) ->go(); // Or use smart routing based on context $promptVersion = match(true) { $isAngryCustomer => 'v3/empathetic', $isTechnicalIssue => 'v3/technical', $isVipCustomer => 'v3/formal', default => 'v3/default' }; $response = CustomerSupportAgent::run($query) ->withPromptVersion($promptVersion) ->go(); ``` ## Blade Templates for Dynamic Prompts Use Laravel's Blade templating engine to create dynamic prompts that adapt based on user data, context, and session state. ### How It Works Save your prompts with a `.blade.php` extension instead of `.md`: ```blade resources/prompts/customer_support/default.blade.php theme={null} You are {{ $agent['name'] }}, a helpful assistant. @if(isset($user_name)) Hello {{ $user_name }}! How can I help you today? @else Hello! How can I assist you? @endif @if($tools->isNotEmpty()) Available capabilities: @foreach($tools as $tool) - {{ $tool['description'] }} @endforeach @endif ``` ### Available Variables * `$agent` - Agent info (name, model, etc.) * `$context` - Full AgentContext object * `$user_input` - Current user input * `$session_id` - Session identifier * `$user`, `$user_name` - User data * `$memory_context` - Previous interactions * `$tools` - Available tools collection * `$sub_agents` - Sub-agents collection ### Custom Variables Add your own variables to inject into the blade prompt with `getPromptData()`: ```php theme={null} class SalesAgent extends BaseLlmAgent { protected function getPromptData(?AgentContext $context): array { return [ 'company_name' => config('app.company_name'), 'promotions' => $this->getActivePromotions(), 'business_hours' => '9 AM - 5 PM EST', ]; } } ``` ### When to Use What * Prompts are mostly static * Simple version control needed * No dynamic content required * Need user personalization * Conditional logic required * Dynamic content injection Blade templates work seamlessly with the existing versioning system - just use `withPromptVersion('name')` as usual! ## Organizing Your Prompts Keep your prompts organized with these flexible structures: ``` resources/prompts/ ├── weather_reporter/ │ ├── default.md # Standard tone │ ├── detailed.md # Comprehensive │ ├── concise.md # Brief responses │ └── professional.md # Business tone └── customer_support/ ├── default.md ├── friendly.md └── technical.md ``` ``` resources/prompts/ ├── weather_reporter/ │ ├── v1/ │ │ ├── default.md │ │ └── concise.md │ └── v2/ │ ├── default.md │ ├── concise.md │ └── detailed.md # New in v2 └── customer_support/ ├── v1/ │ └── default.md └── v2/ ├── default.md ├── friendly.md # New variant └── technical.md # New variant ``` Start simple with just variants. Add versioning when you need to track improvements over time! ## CLI Command Mastery The `vizra:prompt` command is your Swiss Army knife for prompt management! ### Creating Prompts ```bash Terminal theme={null} # Interactive mode - enter multi-line content php artisan vizra:prompt create weather_reporter friendly # Quick mode - inline content php artisan vizra:prompt create weather_reporter v2 --content="You are a helpful weather assistant." ``` ### Listing Prompts ```bash Terminal theme={null} # List all prompts php artisan vizra:prompt list # List for specific agent php artisan vizra:prompt list weather_reporter # Example output: # +------------------+----------+--------+--------+---------------------+ # | Agent | Version | Source | Active | Created | # +------------------+----------+--------+--------+---------------------+ # | weather_reporter | default | File | ✓ | 2024-01-15 10:30:00 | # | weather_reporter | detailed | File | | 2024-01-15 11:00:00 | # | weather_reporter | concise | File | | 2024-01-15 11:30:00 | # +------------------+----------+--------+--------+---------------------+ ``` ### Import & Export ```bash Terminal theme={null} # Export a prompt to file php artisan vizra:prompt export weather_reporter detailed --file=weather_detailed.md # Import from file php artisan vizra:prompt import weather_reporter v3 --file=weather_detailed.md # View prompt content directly php artisan vizra:prompt export weather_reporter detailed ``` ## Runtime Magic Switch prompt versions on the fly - no redeploys needed! ### Setting Default Version in Agent Class You can define a default prompt version directly in your agent class: ```php Agent with Default Prompt Version theme={null} class WeatherReporterAgent extends BaseLlmAgent { protected string $name = 'weather_reporter'; // Set default prompt version for this agent protected ?string $promptVersion = 'professional'; // This will be overridden by the professional.md file protected string $instructions = 'You are a weather reporter.'; } // Now all calls use 'professional' by default $response = WeatherReporterAgent::run('What\'s the weather?')->go(); // Override the default when needed $response = WeatherReporterAgent::run('Current temperature?') ->withPromptVersion('concise') ->go(); ``` ### Using Blade Templates at Runtime Blade templates work seamlessly with the existing prompt versioning system: ```php Using Blade Templates with Runtime API theme={null} // Blade templates work just like markdown versions $response = WeatherReporter::run('What\'s the weather?') ->withPromptVersion('detailed') // Can be detailed.blade.php ->go(); // Mix and match - the system automatically detects the file type // Version 'friendly' could be friendly.md or friendly.blade.php $response = CustomerSupport::run($query) ->withPromptVersion('friendly') ->go(); // Pass additional context for Blade templates $response = SalesAgent::run('Show me pricing') ->withContext([ 'premium_user' => true, 'account_age_days' => 365, 'preferred_language' => 'es', ]) ->withPromptVersion('personalized') // Uses personalized.blade.php ->go(); // Works with async operations too $job = ReportGenerator::run($data) ->withPromptVersion('executive') // executive.blade.php with dynamic formatting ->async() ->go(); ``` ### Runtime Version Selection ```php Using Prompt Versions in Code theme={null} // Using specific version $response = WeatherReporterAgent::run('What\'s the weather?') ->withPromptVersion('detailed') ->go(); // Using Agent Facade $response = Agent::build('weather_reporter') ->withPromptVersion('concise') ->run('Current temperature?') ->go(); // Version from variable (great for A/B testing!) $version = $user->prefers_detailed ? 'detailed' : 'concise'; $response = WeatherReporter::run($query) ->withPromptVersion($version) ->go(); ``` ### Advanced Patterns ```php Advanced Prompt Version Usage theme={null} // A/B Testing Pattern $versions = ['default', 'friendly', 'professional']; $results = []; foreach ($versions as $version) { $startTime = microtime(true); $response = CustomerSupportAgent::run($testQuery) ->withPromptVersion($version) ->go(); $results[$version] = [ 'response' => $response, 'time' => microtime(true) - $startTime, 'length' => strlen($response) ]; } // Find the best performer $bestVersion = collect($results)->sortBy('time')->keys()->first(); ``` ## Evaluation Integration Test different prompt versions systematically with evaluations! ### CSV-Driven Testing ```csv evaluation_data.csv theme={null} location,expected_format,prompt_version "New York",detailed,detailed "London",concise,concise "Tokyo",professional,professional "Paris",friendly,friendly ``` ```php Evaluation Class with Prompt Versions theme={null} class PromptComparisonEval extends BaseEvaluation { public string $agentName = 'weather_reporter'; // CSV column that specifies prompt version public ?string $promptVersionColumn = 'prompt_version'; // Or set a default for all tests public array $agentConfig = [ 'prompt_version' => 'detailed', 'temperature' => 0.7, ]; public function evaluateRow(array $csvRow, string $response): array { // Your evaluation logic here // The correct prompt version is automatically used! } } ``` ## Configuration Options Customize prompt versioning behavior in `config/vizra-adk.php`: ```php Prompt Configuration theme={null} 'prompts' => [ // Enable database storage (optional) 'use_database' => env('VIZRA_ADK_PROMPTS_USE_DATABASE', false), // Custom storage path 'storage_path' => env('VIZRA_ADK_PROMPTS_PATH', resource_path('prompts')), // Track usage analytics 'track_usage' => env('VIZRA_ADK_PROMPTS_TRACK_USAGE', false), ], ``` ### Database Storage (Advanced) For production environments, enable database storage: ```bash Terminal theme={null} # Run migrations php artisan migrate # Update .env VIZRA_ADK_PROMPTS_USE_DATABASE=true # Activate a version php artisan vizra:prompt activate weather_reporter v2 ``` ## Best Practices * `default` - Always have one! * `friendly` - Descriptive names * `2024_01_15_improved` - Date versions * ~~`v2`~~ - Too vague! * Start with file-based prompts * Test with evaluations * A/B test in production * Move winners to database ## Troubleshooting ### Prompt Not Loading? 1. Check file exists: `resources/prompts/{agent_name}/{version}.md` 2. Verify file permissions are readable 3. Clear cache: `php artisan cache:clear` 4. Check agent name matches directory name ### Database Prompts Not Working? 1. Ensure `use_database` is `true` in config 2. Run migrations: `php artisan migrate` 3. Check prompt exists: `php artisan vizra:prompt list` 4. Verify database connection is working *** ## You're Now a Dynamic Prompts Pro! With dynamic prompts, your AI agents can evolve and improve without touching code. Test freely, roll back instantly, and find the perfect voice for every agent. The future of AI development is here! # Sessions & Memory Source: https://docs.vizra.ai/concepts/sessions-memory Give your agents a memory that lasts. Learn how to build agents that remember conversations, learn from interactions, and provide personalized experiences over time. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). **Sessions** provide conversation context while **memory** enables long-term knowledge retention. Together, they create intelligent, personalized agent interactions that improve over time! ## Understanding Sessions Think of sessions as your agent's short-term memory during a conversation! Every message between your agent and user is tracked State persists across multiple interactions Keep track of important conversation data Connect to long-term agent memory ## Creating and Managing Sessions ### Using Sessions with Agents Let's see how easy it is to maintain conversations across multiple interactions! ```php app/Http/Controllers/ChatController.php theme={null} use App\Agents\CustomerSupportAgent; // Start a conversation with a session $response = CustomerSupportAgent::run('I need help with my order') ->forUser($user) ->withSession($sessionId) ->go(); // Continue in the same session - agent remembers! $response2 = CustomerSupportAgent::run('The order number is #12345') ->forUser($user) ->withSession($sessionId) ->go(); ``` ### Session Models Sessions are first-class citizens in Vizra, stored safely in your database! ```php theme={null} use Vizra\VizraADK\Models\AgentSession; use Vizra\VizraADK\Models\AgentMessage; // Sessions are stored in the database $session = AgentSession::where('session_id', $sessionId)->first(); // Access session messages $messages = $session->messages()->get(); // Access session state $stateData = $session->state_data; ``` ### AgentContext Magic The `AgentContext` is your Swiss Army knife for session management! ```php app/Agents/MyAgent.php theme={null} // AgentContext provides session state management public function run(string $input, AgentContext $context): mixed { // Get session ID $sessionId = $context->getSessionId(); // Store state in session $context->setState('order_id', '#12345'); // Retrieve state $orderId = $context->getState('order_id'); // Get conversation history $history = $context->getConversationHistory(); } ``` ## Memory System Give your agents the gift of memory! Vizra provides persistent memory across sessions through the powerful `AgentMemory` model and `MemoryManager` service: Short-term context stored in AgentContext during a conversation - like working memory! Long-term memory with learnings, facts, and session summaries - like permanent storage! ## Using Persistent Memory ### Agent Memory API Every agent now has a built-in memory system that's super easy to use! ```php app/Agents/SupportAgent.php theme={null} class SupportAgent extends BaseLlmAgent { protected string $name = 'support_agent'; public function afterLlmResponse(Response|Generator $response, AgentContext $context): mixed { // Extract insights from the conversation if (str_contains($response, 'prefer email')) { $this->memory()->addLearning('User prefers email communication'); } // Store facts if ($this->detectAccountType($response)) { $this->memory()->addFact('Customer has premium account', 0.9); } // Add preferences $this->memory()->addPreference('Email notifications', 'communication'); // Update summary periodically if ($this->shouldUpdateSummary()) { $this->memory()->updateSummary( 'Premium customer who prefers email, interested in API features' ); } return $response; } protected function buildSystemPrompt(): string { // Automatically include memory context in prompts! $memoryContext = $this->memory()->getContext(1000); return $this->instructions . "\n\nUser Context:\n" . $memoryContext; } } ``` ### Memory Methods The agent memory API provides specialized methods for different types of information: ```php theme={null} // Inside any agent method, access memory via $this->memory() // Learnings - Insights gained from interactions $this->memory()->addLearning('User works in healthcare', [ 'confidence' => 'high', 'source' => 'direct_mention' ]); // Facts - Immutable truths about the user $this->memory()->addFact('Located in New York', 1.0); // confidence score // Preferences - User likes and dislikes $this->memory()->addPreference('Detailed explanations', 'communication_style'); // Summary - Overall user profile (replaces previous) $this->memory()->updateSummary('Tech-savvy healthcare professional in NYC'); // Recall - Search through memories $learnings = $this->memory()->getLearnings(); $facts = $this->memory()->getFacts(); $preferences = $this->memory()->getPreferences('communication_style'); $summary = $this->memory()->getSummary(); // Context - Get formatted memory for prompts $context = $this->memory()->getContext(1500); // max tokens ``` ### Memory Manager Service For advanced use cases, you can still access the Memory Manager directly: ```php app/Services/AgentService.php theme={null} use Vizra\VizraADK\Services\MemoryManager; // Get or create memory for an agent $memory = $memoryManager->getOrCreateMemory('support_agent', $userId); // Add a learning $memoryManager->addLearning( 'support_agent', 'User prefers email communication', $userId ); // Add a fact $memoryManager->addFact( 'support_agent', 'preferred_contact', 'email', $userId ); // Update memory summary $memoryManager->updateSummary( 'support_agent', 'Customer is a premium member who prefers email contact', $userId ); ``` ### Using Memory Tool Agents can manage their own memory - how cool is that? ```php app/Agents/SupportAgent.php theme={null} // Agents can use the MemoryTool to manage their own memory class SupportAgent extends BaseLlmAgent { protected array $tools = [ MemoryTool::class, ]; protected string $instructions = '... Use the manage_memory tool to: - Add important learnings about the user - Store facts for future reference - Retrieve context from previous conversations '; } ``` ### Memory Context in Instructions Inject past memories into your agent's instructions for truly personalized interactions! ```php theme={null} // Get memory context to include in agent instructions $memoryContext = $memoryManager->getMemoryContext( 'support_agent', $userId, 1000 // max length ); // Memory context includes: // - Previous Knowledge (summary) // - Key Learnings (last 5) // - Important Facts (up to 10) ``` ## Memory Model Structure Let's peek under the hood! The `AgentMemory` model is beautifully organized: ### Memory Properties ```json database/agent_memories table theme={null} // AgentMemory model structure { "agent_name": "support_agent", "user_id": 123, "memory_summary": "Previous conversation summaries", "memory_data": { // Facts stored as key-value pairs "preferred_contact": "email", "account_type": "premium" }, "key_learnings": [ // Array of learnings with timestamps { "learning": "User prefers detailed explanations", "learned_at": "2024-01-15T10:30:00Z" } ], "total_sessions": 42, "last_session_at": "2024-01-15T10:30:00Z" } ``` ### Accessing Memory Data Working with memory data is a breeze! Check out these handy methods: ```php theme={null} use Vizra\VizraADK\Models\AgentMemory; // Find memory for agent and user $memory = AgentMemory::where('agent_name', 'support_agent') ->where('user_id', $userId) ->first(); // Get context summary for agent instructions $contextSummary = $memory->getContextSummary(1000); // Add a learning $memory->addLearning('User works in healthcare industry'); // Update memory data $memory->updateMemoryData([ 'industry' => 'healthcare', 'timezone' => 'EST' ]); // Record a new session $memory->recordSession(); ``` ## Session and Memory Integration ### Automatic Memory Updates The magic happens automatically! Sessions and memory work together seamlessly: ```php theme={null} // Sessions automatically link to memory $session = AgentSession::find($id); $memory = $session->getOrCreateMemory(); // Update memory from session $session->updateMemory([ 'learnings' => [ 'User is interested in API integration', 'Prefers Python code examples' ], 'facts' => [ 'programming_language' => 'Python', 'api_experience' => 'intermediate' ] ]); ``` ### Memory Extraction Watch as Vizra intelligently extracts insights from conversations! ```php theme={null} // MemoryManager automatically extracts insights from sessions $memoryManager->updateMemoryFromSession($session); // This automatically: // - Records the session in memory // - Extracts preferences ("I like/prefer...") // - Extracts personal info ("My name is...") // - Updates session count and timestamps ``` ## Memory Lifecycle ### Session Summarization Keep memories fresh and relevant with intelligent summarization! ```php theme={null} // Summarize recent sessions into memory $memoryManager->summarizeMemory( $memory, 10 // Number of recent sessions to summarize ); // Get recent conversations $recentConversations = $memoryManager->getRecentConversations( 'support_agent', $userId, 5 // Limit ); ``` ### Memory Cleanup Keep your database tidy with smart cleanup strategies! ```php theme={null} // Clean up old memories $deletedCount = $memoryManager->cleanupOldMemories( 90, // Days old 1000 // Max sessions ); // Clean up old sessions while preserving memory $deletedSessions = $memoryManager->cleanupOldSessions( 'support_agent', 30 // Days old ); ``` ## Implementation Examples ### Customer Support Agent A complete example of a memory-aware support agent that learns and adapts! ```php app/Agents/CustomerSupportAgent.php theme={null} class CustomerSupportAgent extends BaseLlmAgent { protected string $name = 'customer_support'; protected string $instructions = 'You are a helpful customer support agent...'; public function afterLlmResponse(Response|Generator $response, AgentContext $context): mixed { $responseText = $response instanceof Response ? $response->text : ''; // Extract and store contact preferences if (preg_match('/prefer.*(email|phone|chat)/i', $responseText, $matches)) { $this->memory()->addPreference( "Prefers {$matches[1]} communication", 'contact' ); } // Detect product interests if (str_contains($responseText, 'API') || str_contains($responseText, 'integration')) { $this->memory()->addLearning('Interested in API/integration features'); } // Store important facts if (preg_match('/order\s*#?(\d+)/i', $responseText, $matches)) { $this->memory()->addFact("Recent order: #{$matches[1]}", 1.0); } return $response; } protected function buildSystemPrompt(): string { // Include all relevant memory context $memory = $this->memory(); $summary = $memory->getSummary(); $preferences = $memory->getPreferences('contact'); $recentFacts = $memory->getFacts()->take(5); $prompt = $this->instructions; if ($summary) { $prompt .= "\n\nCustomer Profile: " . $summary; } if ($preferences->isNotEmpty()) { $prompt .= "\n\nCommunication Preferences:"; foreach ($preferences as $pref) { $prompt .= "\n- " . $pref->content; } } if ($recentFacts->isNotEmpty()) { $prompt .= "\n\nRecent Information:"; foreach ($recentFacts as $fact) { $prompt .= "\n- " . $fact->content; } } return $prompt; } } ``` ### Educational Tutor Agent An agent that tracks learning progress and adapts teaching style! ```php app/Agents/TutorAgent.php theme={null} class TutorAgent extends BaseLlmAgent { protected string $name = 'tutor_agent'; public function afterLlmResponse(Response|Generator $response, AgentContext $context): mixed { // Track topics covered if ($topic = $this->extractTopic($response)) { $this->memory()->addLearning("Studied topic: {$topic}", [ 'timestamp' => now()->toIso8601String(), 'session_id' => $context->getSessionId() ]); } // Detect learning style preferences $this->detectLearningStyle($response, $context); // Update progress summary $learnings = $this->memory()->getLearnings(); if ($learnings->count() % 5 === 0) { // Every 5 learnings $topics = $learnings->pluck('content')->take(10)->implode(', '); $this->memory()->updateSummary( "Student has covered: {$topics}. Shows strong interest in practical examples." ); } return $response; } private function detectLearningStyle($response, $context): void { $userInput = $context->getLastUserMessage(); if (str_contains($userInput, 'example') || str_contains($userInput, 'show me')) { $this->memory()->addPreference('Prefers examples', 'learning_style'); } if (str_contains($userInput, 'why') || str_contains($userInput, 'explain')) { $this->memory()->addPreference('Likes detailed explanations', 'learning_style'); } if (str_contains($userInput, 'quick') || str_contains($userInput, 'summary')) { $this->memory()->addPreference('Prefers concise answers', 'learning_style'); } } } ``` ### Memory-Aware Agent Base Class Create a reusable base class for all memory-aware agents: ```php app/Agents/MemoryAwareAgent.php theme={null} abstract class MemoryAwareAgent extends BaseLlmAgent { protected bool $autoExtractPreferences = true; protected bool $includeMemoryInPrompt = true; protected function buildSystemPrompt(): string { $prompt = $this->instructions; if ($this->includeMemoryInPrompt) { $memoryContext = $this->memory()->getContext(800); if ($memoryContext) { $prompt .= "\n\nUser Context:\n" . $memoryContext; } } return $prompt; } public function afterLlmResponse(Response|Generator $response, AgentContext $context): mixed { if ($this->autoExtractPreferences) { $this->extractPreferences($response, $context); } // Let child classes add their own memory logic $this->updateMemory($response, $context); return $response; } /** * Override this in child classes to add custom memory updates */ protected function updateMemory(Response|Generator $response, AgentContext $context): void { // Child classes implement their specific memory logic } private function extractPreferences($response, $context): void { $text = $response instanceof Response ? $response->text : ''; $userInput = $context->getLastUserMessage(); // Common preference patterns $patterns = [ '/I (?:prefer|like|want|need) (.+?)(?:\.|,|$)/i' => 'general', '/(?:call|email|text|message) me/i' => 'contact', '/I (?:work in|am in|from) (.+?)(?:\.|,|$)/i' => 'location', '/my (?:job|role|position) is (.+?)(?:\.|,|$)/i' => 'occupation' ]; foreach ($patterns as $pattern => $category) { if (preg_match($pattern, $userInput, $matches)) { $preference = isset($matches[1]) ? $matches[1] : $matches[0]; $this->memory()->addPreference(trim($preference), $category); } } } } ``` ### Session State Management Store complex data structures in session state - it's that flexible! ```php theme={null} // Store complex state in sessions $context->setState('cart_items', [ ['id' => 123, 'quantity' => 2], ['id' => 456, 'quantity' => 1] ]); // Load all state $allState = $context->getAllState(); // Merge new state $context->loadState([ 'preferences' => $userPreferences, 'settings' => $userSettings ]); ``` ## Memory Best Practices Use AgentContext for session state Store important facts in AgentMemory Let MemoryManager handle summaries Use MemoryTool for self-management Implement cleanup for old sessions Include memory in instructions Learn about agent workflows and multi-step processes Learn about dynamic prompt versioning and variants # Tools Source: https://docs.vizra.ai/concepts/tools Supercharge your agents with powerful capabilities. Tools let your AI agents interact with the real world - from databases to APIs to custom business logic. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). ## What Makes Tools Awesome? Think of tools as your agent's superpowers! While agents are great at conversation, tools let them actually **do things** - like looking up orders, sending emails, or integrating with your favorite APIs. It's like giving your AI a Swiss Army knife! ## Understanding Tools Here's what makes Vizra tools special: Just implement the `ToolInterface` and you're ready to roll! JSON Schema definitions help LLMs understand exactly how to use your tools Prism automatically connects your tools to the LLM's function calling Access the full `AgentContext` for stateful operations ## Creating Your First Tool **Quick Start with Artisan** - The easiest way to create a tool? Let Artisan do the heavy lifting! ```bash Terminal theme={null} php artisan vizra:make:tool OrderLookupTool ``` This creates a fully-functional tool template in `app/Tools/` ready for your custom logic! ### Tool Structure Every tool follows a simple pattern - implement the `ToolInterface` and define two key methods: ```php app/Tools/OrderLookupTool.php theme={null} 'order_lookup', 'description' => 'Look up order information by order ID', 'parameters' => [ 'type' => 'object', 'properties' => [ 'order_id' => [ 'type' => 'string', 'description' => 'The order ID to look up', ], ], 'required' => ['order_id'], ], ]; } public function execute(array $arguments, AgentContext $context, AgentMemory $memory): string { $orderId = $arguments['order_id'] ?? null; if (!$orderId) { return json_encode([ 'status' => 'error', 'message' => 'Order ID is required', ]); } // Look up the order in your database $order = Order::find($orderId); if (!$order) { return json_encode([ 'status' => 'error', 'message' => "Order {$orderId} not found", ]); } // Store this interaction in memory $memory->addFact("Recent order lookup: #{$orderId}", 1.0); $memory->addLearning("User inquired about order #{$orderId}"); return json_encode([ 'status' => 'success', 'order_id' => $order->id, 'status' => $order->status, 'total' => $order->total, 'created_at' => $order->created_at->toDateTimeString(), ]); } } ``` ## Tool Interface Methods ### The definition() Method This is where you tell the LLM exactly what your tool does and what parameters it needs. Think of it as your tool's instruction manual! ```php Tool Definition Example theme={null} public function definition(): array { return [ 'name' => 'weather_tool', 'description' => 'Get current weather information', 'parameters' => [ 'type' => 'object', 'properties' => [ 'location' => [ 'type' => 'string', 'description' => 'The city and state, e.g. San Francisco, CA', ], 'units' => [ 'type' => 'string', 'enum' => ['celsius', 'fahrenheit'], 'description' => 'Temperature units', ], ], 'required' => ['location'], ], ]; } ``` Use descriptive names that tell the LLM exactly what your tool does Help the LLM understand when and how to use your tool Define parameter types to ensure correct usage ### The execute() Method This is where the magic happens! Your tool receives arguments from the LLM and the current context, then returns results as JSON. ```php Tool Execution Example theme={null} public function execute(array $arguments, AgentContext $context): string { // Access arguments $location = $arguments['location'] ?? null; // Access context $sessionId = $context->getSessionId(); $previousValue = $context->getState('some_key'); // Implement tool logic $result = [ 'status' => 'success', 'temperature' => 25, 'condition' => 'sunny', 'location' => $location, ]; // Must return JSON string return json_encode($result); } ``` **Remember:** Always return a JSON-encoded string! The LLM expects structured data it can understand. ## Working with AgentContext The `AgentContext` is your tool's memory bank! It lets you access session info, store state between calls, and maintain context across conversations. ```php Using AgentContext theme={null} public function execute(array $arguments, AgentContext $context): string { // Get session ID $sessionId = $context->getSessionId(); // Access state $previousOrder = $context->getState('last_order_id'); // Store state for future use $context->setState('last_order_id', $orderId); // Access user information if available $userId = $context->getState('user_id'); $userEmail = $context->getState('user_email'); // Your tool logic here... return json_encode(['status' => 'success']); } ``` Use `getState()` to retrieve previously stored values Use `setState()` to persist data for future tool calls ## Connecting Tools to Agents Ready to give your agent superpowers? Just add your tools to the agent's `$tools` array and watch the magic happen! ```php app/Agents/CustomerSupportAgent.php theme={null} class CustomerSupportAgent extends BaseLlmAgent { /** @var array> */ protected array $tools = [ OrderLookupTool::class, RefundProcessorTool::class, TicketCreatorTool::class, EmailSenderTool::class, ]; } ``` Tools are automatically instantiated and made available to the LLM. Just list them and Vizra handles the rest! ## Advanced Tool Features Let's explore some powerful patterns for building sophisticated tools! ### Database Queries Connect your tools directly to your database for powerful data operations: ```php Database Search Tool theme={null} class CustomerSearchTool implements ToolInterface { public function execute(array $arguments, AgentContext $context): string { $customers = Customer::query() ->where('name', 'like', "%{$arguments['query']}%") ->orWhere('email', 'like', "%{$arguments['query']}%") ->limit(10) ->get(); return json_encode([ 'status' => 'success', 'customers' => $customers->map(fn($c) => [ 'id' => $c->id, 'name' => $c->name, 'email' => $c->email, ])->toArray(), ]); } } ``` ### API Integration Connect to external APIs and bring real-world data into your conversations: ```php External API Tool theme={null} class WeatherApiTool implements ToolInterface { public function execute(array $arguments, AgentContext $context): string { try { $response = Http::get('https://api.weather.com/v1/current', [ 'location' => $arguments['location'], 'api_key' => config('services.weather.key'), ]); return json_encode([ 'status' => 'success', 'weather' => $response->json(), ]); } catch (Exception $e) { return json_encode([ 'status' => 'error', 'message' => 'Failed to fetch weather: ' . $e->getMessage(), ]); } } } ``` ### File Operations Handle file uploads, downloads, and processing with ease: ```php File Handler Tool theme={null} class FileUploaderTool implements ToolInterface { public function definition(): array { return [ 'name' => 'file_uploader', 'description' => 'Upload a file from base64 content', 'parameters' => [ 'type' => 'object', 'properties' => [ 'file_content' => [ 'type' => 'string', 'description' => 'Base64 encoded file content', ], 'filename' => [ 'type' => 'string', 'description' => 'Name for the file', ], ], 'required' => ['file_content', 'filename'], ], ]; } public function execute(array $arguments, AgentContext $context): string { $content = base64_decode($arguments['file_content']); $path = Storage::put('uploads/' . $arguments['filename'], $content); return json_encode([ 'status' => 'success', 'file_path' => $path, 'size' => strlen($content), ]); } } ``` ## Error Handling & Validation Build bulletproof tools with proper validation and error handling! Your agents will thank you. ### Input Validation Always validate your inputs - it's the first line of defense against errors: ```php Input Validation Example theme={null} public function execute(array $arguments, AgentContext $context, AgentMemory $memory): string { // Validate inputs $validator = Validator::make($arguments, [ 'email' => 'required|email', 'amount' => 'required|numeric|min:0|max:10000', ]); if ($validator->fails()) { return json_encode([ 'status' => 'error', 'message' => 'Invalid parameters: ' . $validator->errors()->first(), ]); } // Store validated information $memory->addFact("User email: {$arguments['email']}", 1.0); $memory->addLearning("User requested transaction of amount: {$arguments['amount']}"); // Proceed with validated data return json_encode(['status' => 'success']); } ``` ### User Context Access Check for user authentication and access user-specific data safely: ```php User Context Validation theme={null} public function execute(array $arguments, AgentContext $context, AgentMemory $memory): string { // Check if user context is available $userId = $context->getState('user_id'); if (!$userId) { return json_encode([ 'status' => 'error', 'message' => 'User authentication required', ]); } // Get user data from context $userData = $context->getState('user_data'); $userEmail = $context->getState('user_email'); // Store user-specific information if ($userEmail && !$memory->getFacts()->contains('content', "User email: {$userEmail}")) { $memory->addFact("User email: {$userEmail}", 1.0); } // Process with user context return json_encode(['status' => 'success']); } ``` ## Tools with Memory Access **Every Tool Gets Memory Access!** - All tools now receive the agent's memory as a third parameter. Build personalized experiences by reading and writing to memory! The `execute` method now includes `AgentMemory`: ```php app/Tools/UserProfileTool.php theme={null} 'manage_user_profile', 'description' => 'Update or retrieve user profile information from memory', 'parameters' => [ 'type' => 'object', 'properties' => [ 'action' => [ 'type' => 'string', 'description' => 'Action to perform', 'enum' => ['update_fact', 'add_preference', 'get_profile'] ], 'key' => ['type' => 'string'], 'value' => ['type' => 'string'] ], 'required' => ['action'] ] ]; } public function execute( array $arguments, AgentContext $context, AgentMemory $memory // Now included in all tools! ): string { switch ($arguments['action']) { case 'update_fact': $memory->addFact( "{$arguments['key']}: {$arguments['value']}", 1.0 ); return json_encode(['success' => true]); case 'add_preference': $memory->addPreference( $arguments['value'], $arguments['key'] ?? 'general' ); return json_encode(['success' => true]); case 'get_profile': return json_encode([ 'summary' => $memory->getSummary(), 'facts' => $memory->getFacts()->pluck('content'), 'preferences' => $memory->getPreferences() ]); } } } ``` * `addFact()` - Store immutable facts * `addLearning()` - Track insights * `addPreference()` - Store preferences * `updateSummary()` - Update user profile * Update user preferences from form submissions * Store discovered facts during conversations * Build comprehensive user profiles over time * Sync memory across different tools **Pro Tip: Simple Memory Usage** - Every tool automatically receives the agent's memory! Use it to store learnings, facts, and preferences. The memory persists across sessions, enabling truly personalized experiences. ## Testing Your Tools Great tools deserve great tests! Here's how to ensure your tools work perfectly every time: ```php tests/Tools/OrderLookupToolTest.php theme={null} class OrderLookupToolTest extends TestCase { public function test_finds_existing_order() { $order = Order::factory()->create(); $tool = new OrderLookupTool(); $context = new AgentContext('test-session'); $result = $tool->execute( ['order_id' => $order->id], $context ); $data = json_decode($result, true); $this->assertEquals('success', $data['status']); $this->assertEquals($order->id, $data['order_id']); } public function test_handles_missing_order() { $tool = new OrderLookupTool(); $context = new AgentContext('test-session'); $result = $tool->execute( ['order_id' => 'invalid'], $context ); $data = json_decode($result, true); $this->assertEquals('error', $data['status']); $this->assertStringContainsString('not found', $data['message']); } } ``` **Testing tip:** Test both success paths and error conditions. Your future self will appreciate it! ## Complete Example: Refund Processor Let's put it all together with a real-world example that shows validation, error handling, and business logic! ```php app/Tools/RefundProcessorTool.php theme={null} 'process_refund', 'description' => 'Process a refund for an order', 'parameters' => [ 'type' => 'object', 'properties' => [ 'order_id' => [ 'type' => 'string', 'description' => 'The order ID to refund', ], 'amount' => [ 'type' => 'number', 'description' => 'Refund amount (optional for partial refunds)', 'minimum' => 0, ], 'reason' => [ 'type' => 'string', 'description' => 'Reason for refund', ], ], 'required' => ['order_id', 'reason'], ], ]; } public function execute(array $arguments, AgentContext $context): string { // Validate inputs $validator = Validator::make($arguments, [ 'order_id' => 'required|string', 'amount' => 'nullable|numeric|min:0', 'reason' => 'required|string', ]); if ($validator->fails()) { return json_encode([ 'status' => 'error', 'message' => 'Validation failed: ' . $validator->errors()->first(), ]); } // Check user authorization $userId = $context->getState('user_id'); if (!$userId) { return json_encode([ 'status' => 'error', 'message' => 'User authentication required', ]); } try { $order = Order::findOrFail($arguments['order_id']); // Verify order belongs to user if ($order->user_id !== $userId) { return json_encode([ 'status' => 'error', 'message' => 'Order not found', ]); } $refund = $this->refunds->process( $order, $arguments['amount'] ?? $order->total, $arguments['reason'] ); return json_encode([ 'status' => 'success', 'refund_id' => $refund->id, 'amount' => $refund->amount, 'refund_status' => $refund->status, 'message' => 'Refund processed successfully', ]); } catch (\Exception $e) { return json_encode([ 'status' => 'error', 'message' => 'Failed to process refund: ' . $e->getMessage(), ]); } } } ``` ## Tool Best Practices * **Single Responsibility** - Each tool should do one thing and do it well * **Input Validation** - Always validate parameters before processing * **Clear Error Messages** - Help the LLM understand what went wrong * **Descriptive Naming** - Use names that clearly describe the tool's purpose * **Error Handling** - Gracefully handle exceptions and edge cases * **Rate Limiting** - Protect expensive operations from abuse * **Thorough Testing** - Test success paths and error conditions * **JSON Responses** - Always return properly formatted JSON strings Learn about context management and persistent state Detailed tool class documentation and methods # Vector Memory & RAG Source: https://docs.vizra.ai/concepts/vector-rag Unlock the power of semantic search and intelligent information retrieval. Let your agents access vast knowledge bases and provide incredibly accurate, context-aware responses. **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). ## Understanding Vector Memory Think of vector memory as your agent's superpower! It transforms text into mathematical representations that capture meaning, enabling your agent to find semantically similar content even when the exact words don't match. **Flexible & Powerful:** Choose from multiple storage providers (Meilisearch, PostgreSQL + pgvector) and embedding providers (OpenAI, Gemini, Cohere, Ollama) to fit your needs! Find content by meaning, not just keywords Lightning-fast similarity searches at scale Support for various storage and embedding providers Enhance responses with relevant knowledge ## Setting Up Vector Memory Let's get your vector memory up and running with Meilisearch! It's the perfect balance of speed, simplicity, and power. ### Quick Setup with Meilisearch Configure vector memory with Meilisearch in your `.env` file: ```bash .env theme={null} # Vector Memory with Meilisearch (recommended for most use cases) VIZRA_ADK_VECTOR_DRIVER=meilisearch # Choose your embedding provider VIZRA_ADK_EMBEDDING_PROVIDER=gemini # or openai, cohere, ollama # Gemini for embeddings (recommended) GEMINI_API_KEY=your-gemini-api-key # Meilisearch configuration MEILISEARCH_HOST=http://localhost:7700 MEILISEARCH_KEY=your-master-key MEILISEARCH_PREFIX=agent_vectors_ # Alternative: OpenAI embeddings # VIZRA_ADK_EMBEDDING_PROVIDER=openai # OPENAI_API_KEY=your-openai-key ``` **Alternative option:** You can also use PostgreSQL + pgvector for production-scale vector similarity search. See the complete configuration reference below for setup details. ### Database Setup ```bash Terminal theme={null} # Run migrations for vector memory tables php artisan migrate # For PostgreSQL, ensure pgvector extension is installed CREATE EXTENSION IF NOT EXISTS vector; ``` ## Using Vector Memory in Agents Your agents now have built-in access to vector memory! Use the convenient `$this->vector()` or `$this->rag()` methods directly within your agent. **New!** These methods are now **public**, so you can also access them externally for testing: `$agent->vector()->addDocument(...)` ### Storing Documents in Your Agent ```php Using Vector Memory in an Agent theme={null} class DocumentationAgent extends BaseLlmAgent { protected string $name = 'documentation_agent'; public function storeKnowledge(string $content, array $metadata = []): void { // Simple string content with optional metadata $this->vector()->addDocument($content, $metadata); // Or use array format for full control $this->vector()->addDocument([ 'content' => $content, 'metadata' => $metadata, 'namespace' => 'docs', 'source' => 'user-upload' ]); } public function searchKnowledge(string $query): array { // Simple search with default settings return $this->rag()->search($query); // Or with custom limit return $this->rag()->search($query, 10); // Or full control with array return $this->rag()->search([ 'query' => $query, 'limit' => 5, 'threshold' => 0.7, 'namespace' => 'docs' ]); } } ``` Both `$this->vector()` and `$this->rag()` return a proxy that automatically injects your agent class. No need to pass agent names anymore! **Simplified API:** Use simple strings for basic operations, or arrays for full control with progressive disclosure! ### Direct VectorMemoryManager Access For advanced use cases outside of agent contexts, you can still use the VectorMemoryManager directly: ```php Direct VectorMemoryManager Usage theme={null} use Vizra\VizraADK\Services\VectorMemoryManager; use App\Agents\DocumentationAgent; $vectorManager = app(VectorMemoryManager::class); // Simple usage with agent class $memories = $vectorManager->addDocument( DocumentationAgent::class, 'Vizra ADK is a Laravel package for building AI agents...', ['category' => 'overview'] ); // Or with full options $memories = $vectorManager->addDocument( DocumentationAgent::class, [ 'content' => 'Vizra ADK is a Laravel package...', 'metadata' => [ 'category' => 'overview', 'version' => '1.0' ], 'namespace' => 'docs', 'source' => 'overview-page' ] ); ``` ### Progressive Disclosure API **Smart API Design:** Start simple, add complexity only when needed! ```php Progressive API Examples theme={null} class KnowledgeAgent extends BaseLlmAgent { public function demonstrateAPI(): void { // 1. SIMPLE: Just content (perfect for quick prototyping) $this->vector()->addChunk('Laravel is awesome!'); // 2. COMMON: Content + metadata (most typical usage) $this->vector()->addChunk( 'Eloquent is Laravel\'s ORM', ['category' => 'database', 'difficulty' => 'beginner'] ); // 3. ADVANCED: Full control with array (when you need everything) $this->vector()->addChunk([ 'content' => 'Advanced Laravel patterns and practices', 'metadata' => ['category' => 'advanced', 'priority' => 'high'], 'namespace' => 'tutorials', 'source' => 'expert-guide', 'source_id' => 'guide-123', 'chunk_index' => 0 ]); } public function searchExamples(): void { // Simple search $results = $this->rag()->search('Laravel ORM'); // With limit $results = $this->rag()->search('Laravel patterns', 10); // Full control $results = $this->rag()->search([ 'query' => 'advanced Laravel', 'namespace' => 'tutorials', 'limit' => 5, 'threshold' => 0.8 ]); } } ``` ### Document Chunking Configuration **Pro tip:** Adjust chunk sizes based on your content type. Smaller chunks for FAQs, larger for narrative content! ```php config/vizra-adk.php theme={null} // Configure chunking in config/vizra-adk.php 'vector_memory' => [ 'chunking' => [ 'size' => 1000, // Characters per chunk 'overlap' => 100, // Overlap between chunks 'separators' => ["\n\n", "\n", ". ", ", ", " "], 'keep_separators' => true, ], ]; ``` ## Searching Vector Memory Here's where the magic happens! Watch your agent find exactly what it needs, even when users ask questions in completely different words. ### Semantic Search in Agents ```php Semantic Search in Agent theme={null} class SmartAssistant extends BaseLlmAgent { protected string $name = 'smart_assistant'; public function findRelevantInfo(string $query): string { // Simple search - uses defaults $results = $this->rag()->search($query); // Or with custom limit $results = $this->rag()->search($query, 10); // Or full control $results = $this->rag()->search([ 'query' => $query, 'limit' => 5, 'threshold' => 0.7, 'namespace' => 'knowledge' ]); // Format results for response $relevant = []; foreach ($results as $result) { $relevant[] = $result->content; } return implode("\n\n", $relevant); } } ``` ### RAG Context Generation RAG (Retrieval-Augmented Generation) combines the power of vector search with LLM generation for incredibly accurate responses! ```php Generate RAG Context in Agent theme={null} class RAGEnabledAgent extends BaseLlmAgent { protected string $name = 'rag_agent'; public function run(mixed $input, AgentContext $context): mixed { // Simple: just pass the query (uses intelligent defaults) $ragContext = $this->rag()->generateRagContext($input); // Common: with a few key options $ragContext = $this->rag()->generateRagContext($input, [ 'namespace' => 'knowledge', 'limit' => 5 ]); // Advanced: full control over retrieval $ragContext = $this->rag()->generateRagContext($input, [ 'namespace' => 'knowledge', 'limit' => 5, 'threshold' => 0.7, 'include_metadata' => true ]); // Only augment if we found relevant content if ($ragContext['total_results'] > 0) { $augmentedInput = "Based on the following context:\n" . $ragContext['context'] . "\n\nUser Question: " . $input; } else { $augmentedInput = $input; // No relevant context found } return parent::run($augmentedInput, $context); } } ``` ## Building RAG-Powered Agents Transform your agents into knowledge powerhouses! Here's a complete example of a RAG-enabled documentation assistant. ### Complete RAG Agent Example ```php Documentation Assistant with RAG theme={null} class DocumentationAssistant extends BaseLlmAgent { protected string $name = 'doc_assistant'; protected string $description = 'AI assistant with access to documentation'; protected string $instructions = 'You are a helpful documentation assistant. Use the provided context to answer questions accurately.'; /** * Load documentation into vector memory */ public function loadDocumentation(string $filePath): void { $content = file_get_contents($filePath); // Simple with metadata $this->vector()->addDocument($content, [ 'source' => basename($filePath), 'type' => 'documentation' ]); // Or with full control $this->vector()->addDocument([ 'content' => $content, 'metadata' => [ 'source' => basename($filePath), 'type' => 'documentation' ], 'namespace' => 'docs', 'source' => 'documentation', 'source_id' => md5($filePath) ]); } /** * Process user questions with RAG */ public function run(mixed $input, AgentContext $context): mixed { // Search for relevant documentation with array options $ragContext = $this->rag()->generateRagContext($input, [ 'namespace' => 'docs', 'limit' => 5, 'threshold' => 0.7 ]); // Only augment if we found relevant content if ($ragContext['total_results'] > 0) { $augmentedInput = "Relevant Documentation:\n" . $ragContext['context'] . "\n\nUser Question: " . $input . "\n\nPlease answer based on the documentation provided."; } else { $augmentedInput = $input . "\n\n(No relevant documentation found)"; } return parent::run($augmentedInput, $context); } /** * Get statistics about loaded documentation */ public function getDocStats(): array { // Default namespace return $this->vector()->getStatistics(); // Or specific namespace return $this->vector()->getStatistics('docs'); // Or with array return $this->vector()->getStatistics(['namespace' => 'docs']); } } ``` ### RAG Configuration ```php config/vizra-adk.php theme={null} // Configure RAG in config/vizra-adk.php 'vector_memory' => [ 'rag' => [ 'context_template' => "Based on the following context:\n{context}\n\nAnswer this question: {query}", 'max_context_length' => 4000, 'include_metadata' => true, ], ], ``` ## Embedding Providers ```bash .env theme={null} OPENAI_API_KEY=your-api-key # In config/vizra-adk.php 'embedding_models' => [ 'openai' => 'text-embedding-3-small' ] ``` * text-embedding-3-small (1536 dims) * text-embedding-3-large (3072 dims) * text-embedding-ada-002 (1536 dims) Implement `EmbeddingProviderInterface` to add custom embedding providers ## Vector Memory Management Keep your vector memory clean and efficient! Here's how to manage, monitor, and optimize your knowledge base. ### Managing Memories ```php Memory Management in Agents theme={null} class ManagedKnowledgeAgent extends BaseLlmAgent { protected string $name = 'managed_agent'; /** * Clear all memories in a namespace */ public function clearNamespace(string $namespace = 'default'): int { // Simple: just the namespace name return $this->vector()->deleteMemories($namespace); // Advanced: with array for consistency return $this->vector()->deleteMemories(['namespace' => $namespace]); } /** * Remove memories from a specific source */ public function removeSource(string $source, string $namespace = 'default'): int { // Simple: source only (uses default namespace) return $this->vector()->deleteMemoriesBySource($source); // Common: source with namespace return $this->vector()->deleteMemoriesBySource($source, $namespace); // Advanced: full array control return $this->vector()->deleteMemoriesBySource([ 'source' => $source, 'namespace' => $namespace ]); } /** * Get memory usage statistics */ public function getMemoryStats(string $namespace = 'default'): array { // Simple: default namespace $stats = $this->vector()->getStatistics(); // Specific namespace $stats = $this->vector()->getStatistics($namespace); // Advanced: with array $stats = $this->vector()->getStatistics(['namespace' => $namespace]); return $stats; } /** * Example: External testing access (new public methods!) */ public function demonstrateExternalAccess(): void { // These methods are now PUBLIC - can be called externally! // Perfect for Tinkerwell testing: $agent = Agent::named('managed_agent'); $result = $agent->vector()->addDocument('Test content'); $search = $agent->rag()->search('test query'); } } ``` ### Creating Tools for Vector Operations ```php Vector Memory Tools theme={null} class SearchKnowledgeTool implements ToolInterface { public function definition(): array { return [ 'name' => 'search_knowledge', 'description' => 'Search the knowledge base for relevant information', 'parameters' => [ 'type' => 'object', 'properties' => [ 'query' => [ 'type' => 'string', 'description' => 'The search query' ], 'namespace' => [ 'type' => 'string', 'description' => 'Knowledge namespace to search', 'default' => 'default' ], 'limit' => [ 'type' => 'integer', 'description' => 'Maximum results to return', 'default' => 5 ] ], 'required' => ['query'] ] ]; } public function execute(array $arguments, AgentContext $context, AgentMemory $memory): string { // Get agent from context $agentClass = $context->getState('agent_class'); $agent = new $agentClass(); // Use the simplified API - proxy handles agent class injection $results = $agent->rag()->search([ 'query' => $arguments['query'], 'namespace' => $arguments['namespace'] ?? 'default', 'limit' => $arguments['limit'] ?? 5 ]); return json_encode([ 'success' => true, 'results' => $results->map(fn($r) => [ 'content' => $r->content, 'similarity' => round($r->similarity, 3), 'metadata' => $r->metadata, 'source' => $r->source ])->toArray(), 'total_found' => $results->count() ]); } } ``` ### Using the VectorMemory Model ```php Direct Model Access theme={null} use Vizra\VizraADK\Models\VectorMemory; // Query memories directly $memories = VectorMemory::forAgent('my_agent') ->inNamespace('default') ->fromSource('docs') ->get(); // Calculate similarity (for non-pgvector) $similarity = $memory->cosineSimilarity($queryEmbedding); ``` ## Vector Storage Drivers High-performance vector similarity search with native PostgreSQL integration. ```sql Setup theme={null} # Install pgvector extension CREATE EXTENSION IF NOT EXISTS vector; # Configure in .env VIZRA_ADK_VECTOR_DRIVER=pgvector DB_CONNECTION=pgsql ``` Lightning-fast, typo-tolerant search engine with built-in vector support. ```bash Setup theme={null} # Configure for Meilisearch VIZRA_ADK_VECTOR_DRIVER=meilisearch MEILISEARCH_HOST=http://localhost:7700 MEILISEARCH_KEY=your-master-key MEILISEARCH_PREFIX=agent_vectors_ ``` **Fallback Behavior** - When neither pgvector nor Meilisearch are available, Vizra ADK automatically falls back to cosine similarity calculation using your database. This works with any driver but is best for development or small datasets. ## Complete Configuration Reference Here's a complete reference for all vector memory configuration options available in Vizra ADK! ```php config/vizra-adk.php theme={null} // Complete vector memory configuration 'vector_memory' => [ // Enable/disable vector memory 'enabled' => env('VIZRA_ADK_VECTOR_ENABLED', true), // Storage driver: pgvector, meilisearch 'driver' => env('VIZRA_ADK_VECTOR_DRIVER', 'pgvector'), // Embedding provider: openai, cohere, ollama, gemini 'embedding_provider' => env('VIZRA_ADK_EMBEDDING_PROVIDER', 'openai'), // Model configuration per provider 'embedding_models' => [ 'openai' => env('VIZRA_ADK_OPENAI_EMBEDDING_MODEL', 'text-embedding-3-small'), 'cohere' => env('VIZRA_ADK_COHERE_EMBEDDING_MODEL', 'embed-english-v3.0'), 'ollama' => env('VIZRA_ADK_OLLAMA_EMBEDDING_MODEL', 'nomic-embed-text'), 'gemini' => env('VIZRA_ADK_GEMINI_EMBEDDING_MODEL', 'text-embedding-004'), ], // Driver-specific settings 'drivers' => [ 'meilisearch' => [ 'host' => env('MEILISEARCH_HOST', 'http://localhost:7700'), 'api_key' => env('MEILISEARCH_KEY'), 'index_prefix' => env('MEILISEARCH_PREFIX', 'agent_vectors_'), ], // ... other drivers ], // Document chunking 'chunking' => [ 'strategy' => env('VIZRA_ADK_CHUNK_STRATEGY', 'sentence'), 'chunk_size' => env('VIZRA_ADK_CHUNK_SIZE', 1000), 'overlap' => env('VIZRA_ADK_CHUNK_OVERLAP', 200), ], // RAG settings 'rag' => [ 'max_context_length' => env('VIZRA_ADK_RAG_MAX_CONTEXT', 4000), 'include_metadata' => env('VIZRA_ADK_RAG_INCLUDE_METADATA', true), ], ] ``` ## Console Commands Powerful CLI tools to manage your vector memory right from the terminal! ### Available Commands ```bash Terminal theme={null} # Store content from a file php artisan vector:store my_agent /path/to/document.txt # Search vector memory php artisan vector:search my_agent "search query" # Get statistics php artisan vector:stats my_agent ``` ## Vector Memory Best Practices * Use namespaces to organize content * Include descriptive metadata * Use content hashing to avoid duplicates * Choose appropriate chunk sizes * Monitor token counts for costs * Use pgvector for production * Match chunking to content type * Select models for your use case * Clean up old memories regularly * Test similarity thresholds * Balance context size vs accuracy * Consider hybrid search approaches Learn about testing and evaluating agents Review memory management # Workflows Source: https://docs.vizra.ai/concepts/workflows Orchestrate multiple agents together in powerful patterns **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk). **Agent Orchestration Made Simple** - Workflows are special agents that conduct other agents like a symphony! They orchestrate multiple agents in sequential, parallel, conditional, or loop patterns - all without requiring LLM control for the workflow logic itself. Think of them as your agent choreographer! ## Understanding Workflows Workflows are the conductors of your agent orchestra! They're special agent types that bring order to complexity: Extend `BaseWorkflowAgent` which extends `BaseAgent` Orchestrate agents without using LLMs for flow control Support sequential, parallel, conditional, and loop execution Pass results between steps automatically ## Workflow Patterns Steps execute one after another in perfect order Multiple agents work simultaneously Choose paths based on conditions Repeat operations with while, times, or forEach ## Creating Your First Workflow Let's build your first workflow! It's as easy as chaining methods together. ### Using the Workflow Facade ```php app/Workflows/DataProcessingWorkflow.php theme={null} use Vizra\VizraADK\Facades\Workflow; use App\Agents\DataCollectorAgent; use App\Agents\DataProcessorAgent; use App\Agents\ReportGeneratorAgent; $workflow = Workflow::sequential() ->then(DataCollectorAgent::class) ->then(DataProcessorAgent::class) ->then(ReportGeneratorAgent::class); ``` ### Sequential Workflow Class ```php theme={null} use Vizra\VizraADK\Agents\SequentialWorkflow; use App\Agents\AnalysisAgent; use App\Agents\SummaryAgent; use App\Agents\CleanupAgent; $workflow = new SequentialWorkflow(); $workflow ->start(AnalysisAgent::class, 'Analyze the data') ->then(SummaryAgent::class, fn($previousResult) => "Summarize this: " . $previousResult ) ->finally(CleanupAgent::class); // Always runs $result = $workflow->execute('Initial input'); ``` ## Workflow Types in Action ### Parallel Workflows Need to notify multiple channels at once? Run data processing in parallel? This is your power tool! ```php theme={null} // Using the facade use App\Agents\EmailAgent; use App\Agents\SmsAgent; use App\Agents\SlackAgent; $workflow = Workflow::parallel([ EmailAgent::class, SmsAgent::class, SlackAgent::class ])->waitForAll(); ``` ### Conditional Workflows Make smart decisions in your workflow! Route to different agents based on conditions. ```php theme={null} use Vizra\VizraADK\Agents\ConditionalWorkflow; use App\Agents\PremiumAgent; use App\Agents\UrgentHandler; use App\Agents\StandardAgent; $workflow = Workflow::conditional() ->when( 'score > 80', PremiumAgent::class, 'Handle premium customer' ) ->when( fn($input) => $input['type'] === 'urgent', UrgentHandler::class ) ->otherwise( StandardAgent::class, 'Handle standard request' ); ``` ### Loop Workflows Process lists, retry until success, or repeat operations - loops make it easy! ```php theme={null} use App\Agents\ProcessingAgent; use App\Agents\BatchProcessor; use App\Agents\ItemProcessor; // While loop $workflow = Workflow::while( ProcessingAgent::class, fn($result) => $result['continue'] === true, 10 // max iterations ); // Times loop $workflow = Workflow::times(BatchProcessor::class, 5); // ForEach loop $items = ['item1', 'item2', 'item3']; $workflow = Workflow::forEach(ItemProcessor::class, $items); ``` ## Running Your Workflows ### Basic Execution ```php theme={null} use Vizra\VizraADK\Facades\Workflow; use Vizra\VizraADK\System\AgentContext; use App\Agents\FirstAgent; use App\Agents\SecondAgent; // Workflows are agents, so they run like any agent $workflow = Workflow::sequential(FirstAgent::class, SecondAgent::class); // Create an agent context (required) $context = new AgentContext('session-123'); // Execute with input and context $result = $workflow->execute('Process this data', $context); ``` ### Understanding Workflow Results Each workflow type returns structured results so you always know what happened! ```php theme={null} // Sequential workflow returns { 'final_result': 'Last agent output', 'step_results': { 'App\\Agents\\FirstAgent': 'First result', 'App\\Agents\\SecondAgent': 'Second result' }, 'workflow_type': 'sequential' } // Parallel workflow returns { 'results': { 'App\\Agents\\AgentOne': 'Result 1', 'App\\Agents\\AgentTwo': 'Result 2' }, 'completed': ['App\\Agents\\AgentOne', 'App\\Agents\\AgentTwo'], 'workflow_type': 'parallel' } ``` ## Advanced Workflow Features Take your workflows to the next level with advanced error handling, callbacks, and dynamic parameters! ### Error Handling and Retries ```php theme={null} use App\Agents\RiskyAgent; $workflow = new SequentialWorkflow(); $workflow ->retryOnFailure(3, 1000) // 3 retries, 1 second delay ->timeout(300) // 5 minute timeout ->then(RiskyAgent::class, 'Process risky operation', [ 'retries' => 5, // Override for this step 'timeout' => 60 ]); ``` ### Callbacks for Workflow Events ```php theme={null} use App\Agents\FirstAgent; use App\Agents\SecondAgent; $workflow = Workflow::sequential(FirstAgent::class, SecondAgent::class) ->onSuccess(function($result, $stepResults) { Log::info('Workflow succeeded', [ 'final' => $result, 'steps' => $stepResults ]); }) ->onFailure(function(\Throwable $e, $stepResults) { Log::error('Workflow failed: ' . $e->getMessage()); }) ->onComplete(function($result, $success, $stepResults) { // Always runs, regardless of success/failure }); ``` ### Dynamic Parameters Pass data between steps dynamically based on results and context! ```php theme={null} use App\Agents\DataFetcher; use App\Agents\Processor; $workflow = new SequentialWorkflow(); $workflow ->then(DataFetcher::class) ->then( Processor::class, // Dynamic params based on previous results fn($input, $results, $context) => [ 'data' => $results[DataFetcher::class], 'mode' => $context->getState('processing_mode') ] ); ``` ## Workflow State Management Access results from any step and manage state throughout your workflow! ```php theme={null} use App\Agents\Collector; use App\Agents\Analyzer; // Access results from previous steps $workflow = new SequentialWorkflow(); $workflow ->then(Collector::class, 'Collect data') ->then(Analyzer::class, function($input, $results) { // Access previous step result $collectedData = $results[Collector::class]; return "Analyze: " . json_encode($collectedData); }); // After execution, get specific results $result = $workflow->execute('Start'); $analyzerResult = $workflow->getStepResult(Analyzer::class); $allResults = $workflow->getResults(); ``` ## Creating Workflows from Arrays Define workflows using simple arrays - perfect for dynamic or configuration-based workflows! ### Array Definition ```php theme={null} use App\Agents\DataCollector; use App\Agents\Validator; $definition = [ 'type' => 'sequential', 'steps' => [ [ 'agent' => DataCollector::class, 'params' => 'Collect user data' ], [ 'agent' => Validator::class, 'options' => ['retries' => 3] ] ] ]; $workflow = Workflow::fromArray($definition); ``` ### Complex Array Definitions ```php theme={null} use App\Agents\UrgentHandler; use App\Agents\SupportAgent; use App\Agents\StandardHandler; // Conditional workflow from array $definition = [ 'type' => 'conditional', 'conditions' => [ [ 'condition' => 'input.priority == "high"', 'agent' => UrgentHandler::class ], [ 'condition' => 'input.type == "support"', 'agent' => SupportAgent::class ] ], 'default' => [ 'agent' => StandardHandler::class ] ]; ``` ## Workflow Best Practices * **Remember the Foundation** - Workflows are agents - they extend BaseWorkflowAgent * **Choose the Right Pattern** - Use the appropriate workflow type for your use case * **Set Smart Limits** - Configure reasonable timeouts and retry limits * **Monitor Everything** - Leverage callbacks for monitoring and debugging * **Smart Data Flow** - Pass results between steps using closures * **Test Thoroughly** - Test workflows with various input scenarios Learn about vector memory and retrieval augmented generation! Deep dive into workflow class documentation # Assertions Reference Source: https://docs.vizra.ai/evals/assertions Every deterministic check — from substrings to real tool calls, token cost, and multi-turn behavior. Deterministic assertions run inside `->assert(...)`, per sample, before any judge. The closure receives an assertion proxy, the row, and the full response: ```php Anatomy theme={null} ->assert(fn ($a, $row, $response) => $a ->notEmpty()->gate() // ->gate(): failure hard-fails the sample, skips judges ->contains($row->expected()) ->toolCalled('lookup_policy')->weight(2.0) // ->weight(): share of the sample score ->costBelow(0.02)) ``` Every assertion records its expected and actual values — failures come with receipts, in the test output and the [dashboard](/evals/dashboard). In [class-based evaluations](/evals/class-based-evaluations) the same checks are `$this->assert*()` methods (e.g. `$a->toolCalled(...)` ⇢ `$this->assertToolCalled(...)`). ## Content | Assertion | Checks | | ------------------------------------------------------------ | ----------------------------------- | | `contains($needle, $ignoreCase = true)` | Substring present | | `notContains($needle)` | Substring absent | | `containsAnyOf([...])` / `containsAllOf([...])` | At least one / all substrings | | `startsWith($prefix)` / `endsWith($suffix)` | Response boundaries | | `matchesRegex('/pattern/')` | Regex match | | `lengthBetween($min, $max)` / `wordCountBetween($min, $max)` | Size bounds | | `notEmpty()` | Non-blank response (a classic gate) | | `isBritishSpelling()` / `isAmericanSpelling()` | Brand-voice spelling conventions | ## Structure | Assertion | Checks | | ------------------------------------------------------------------- | -------------------------------------------------- | | `validJson()` / `jsonHasKey('a.b')` | Response parses as JSON / has a (dot-notation) key | | `validXml()` / `xmlHasTag('tag')` | XML validity / tag presence | | `outputHasKey('score')` | Structured-output agents: key exists | | `outputKey('score', 5)` or `outputKey('score', fn ($v) => $v >= 1)` | Key equals a value, or passes a callback | | `outputKeyMatches('id', '/^ORD-/')` | Key matches a regex | ## Agent behavior These inspect the **real `AgentResponse`** — actual tool calls with their arguments, actual steps — not text parsing: | Assertion | Checks | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | `toolCalled('lookup_order')` / `toolNotCalled('escalate')` | Tool was / wasn't invoked | | `toolCalledWith('lookup_order', ['id' => 7])` | Called at least once with this **argument subset** (nested arrays supported, extra keys allowed) | | `toolCallOrder(['lookup_order', 'create_return'])` | Tools appear in this relative order (others may interleave) | | `stepsBelow(4)` | Agent loop finished in under N steps | | `finishReason(FinishReason::Stop)` | How generation ended — failures flag `Length` as truncation and `ContentFilter` as refusal | | `noPendingApprovals()` | Response isn't paused on human-in-the-loop tool approval | ## Usage & cost | Assertion | Checks | | ------------------------------------------------- | --------------------------------------------------------------- | | `costBelow(0.02)` | Sample cost (USD, from the [price table](/evals/configuration)) | | `tokensBelow(4000)` | Total tokens across prompt/completion/cache/reasoning | | `cacheHitRateAbove(0.5)` | Prompt-cache effectiveness | | `durationBelow(3000)` | Wall-clock milliseconds for the invocation | | `modelUsed('gpt-5.4')` / `providerUsed('openai')` | Which model/provider actually served it | ## Safety pre-filters Named honestly — these are wordlist and regex checks, useful as cheap gates, **not** classifiers: | Assertion | Checks | | ------------------------------- | ------------------------------------------------------------------- | | `containsNoBlockedWords([...])` | Built-in blocklist + `evals.safety.blocked_words` + extras | | `noObviousPII()` | Regex-grade scan: emails, US SSNs/phones, card-shaped numbers, IPv4 | Anything needing actual judgment — tone, toxicity in context, factuality — belongs to the [LLM judge](/evals/judge), not a wordlist. ## Scalars `equals($expected, $actual)`, `true($condition)`, `false($condition)`, `greaterThan($threshold, $actual)`, `lessThan($threshold, $actual)` — for checks you compute yourself from `$row` and `$response`. ## Custom assertions Implement the contract once, use it everywhere: ```php app/Evals/Assertions/MentionsOrderNumber.php theme={null} use Laravel\Ai\Responses\AgentResponse; use Vizra\Evals\Assertions\Assertion; use Vizra\Evals\Assertions\AssertionResult; use Vizra\Evals\Dataset\Row; class MentionsOrderNumber implements Assertion { public function __invoke(AgentResponse $response, Row $row): AssertionResult { $found = preg_match('/ORD-\d{6}/', $response->text) === 1; return AssertionResult::bool( 'mentions_order_number', $found, 'an ORD-xxxxxx reference', $response->text, 'Response never cites the order number.', ); } } ``` ```php Using it theme={null} ->assert(fn ($a) => $a->with(new MentionsOrderNumber)) ``` # Baselines & Regressions Source: https://docs.vizra.ai/evals/baselines-and-regressions Every run compared against the run you trusted — row by row, with receipts. The core workflow: pick a run you trust, make it the **baseline**, and every future run is diffed against it. This is what turns evals from "did it pass?" into "is it getting worse?" ## Baselines * **Automatic**: in Pest, the first run that passes its gate becomes its suite's baseline. Nothing to configure. * **Manual**: promote any completed run from the [dashboard](/evals/dashboard), or: ```bash Terminal theme={null} php artisan evals:baseline 01kyvrjyxp9ex0f3gtvznmafdn ``` One baseline per suite; promoting a new one demotes the old one transactionally. ## How runs are joined Rows are matched across runs by **content hash** — a fingerprint of the row's input, prior messages, and expected data, computed when the dataset is read. Reordering your JSONL file, adding rows, or renaming files changes nothing; the same logical row lines up across months of runs. (For single-model runs the reported provider/model id is deliberately ignored in the join — providers rotate dated model ids like `gpt-5-mini-2025-08-07`, and that must not sever your history.) ## Classification For each row present in both runs: | Class | Condition | | ----------------- | --------------------------------------------------------------------------------------------------------- | | **Regressed** | Pass rate dropped at all, **or** score mean dropped by more than `evals.compare.epsilon` (default `0.05`) | | **Newly failing** | Was passing every sample on the baseline; isn't now (a subset of regressed) | | **Improved** | The symmetric opposite | | Unchanged | Everything else — score jitter within epsilon is noise, not signal | Rows only in the current run are **new**; rows only in the baseline are **removed**. **Why epsilon?** You're sampling a nondeterministic system — small score wobble is expected. A pass-rate drop is always a regression, but a `0.93 → 0.90` score drift shouldn't page anyone. Tune `evals.compare.epsilon` to your tolerance. ## Failing the build on regressions ```php In Pest theme={null} ->gate(maxRegressions: 0) ``` ```text What a regression failure looks like theme={null} Gate failed: 2 rows regressed against the reference run (allowed: 0). ↓ regressed: "What is your refund policy?" 96.7% → 51.7% ↓ regressed: "Can I return it?" 93.3% → 55.0% ``` Or without Pest: ```bash Terminal theme={null} php artisan evals:run SupportQuality --compare=baseline --output=json # exit 0 = clean · exit 1 = gate failed or regressions exceeded the allowance ``` `--compare` also accepts a run id or `latest`. The dashboard's compare view renders the same diff — regressed and improved tables with before → after scores. See [Running in CI](/evals/ci) for the full pipeline recipe. # Judge Calibration Source: https://docs.vizra.ai/evals/calibration Measure how much your LLM judge agrees with humans — before you let it gate a release. An LLM judge is a measurement instrument, and uncalibrated instruments lie confidently. Before gating CI on a judge, check its agreement with human labels. ## Label a dataset Collect responses and score them yourself (or have your team do it): ```json storage/evals/labelled.jsonl theme={null} {"input": "What is your refund policy?", "output": "Full refunds within 30 days with a receipt.", "human_score": 9} {"input": "Do you price match?", "output": "Yes, we match any competitor!", "human_score": 2} {"input": "Can I return opened items?", "output": "Opened items can be returned within 30 days.", "human_score": 7} ``` Each row needs an `output` (the response being graded) and a `human_score` (1–10) — or a `human_verdict` (`pass`/`fail`) if you'd rather label coarsely. ## Run the calibration ```bash Terminal theme={null} php artisan evals:calibrate storage/evals/labelled.jsonl \ --criteria="Answers using only documented store policy." ``` The judge scores every row against the same criteria you'd use in your evals, and you get: * **Exact agreement** and **agreement within ±1** (or verdict accuracy for pass/fail labels) * **Mean absolute error** in points * The **worst disagreements**, each with the judge's reasoning — the fastest way to see *why* it diverges ```text Example output theme={null} Rows with human scores ................................ 25 Exact agreement ....................................... 40.0% Agreement within ±1 ................................... 84.0% Mean absolute error ................................... 0.9 points Worst disagreements ......................... judge vs human Do you price match? .................................. 8 vs 2 The response is confident and helpful in tone… ``` That disagreement is the tell: the judge rewarded confident tone over policy accuracy — so sharpen the criteria ("penalize any claim not in the documented policy") and calibrate again. Use `--judge=App\Evals\Judges\PolicyJudge` to calibrate a [custom judge](/evals/judge#custom-judge-agents). Calibration runs are persisted like any other run, so you can track judge quality over time too. Within-±1 agreement above \~80% is a reasonable bar for gating CI with `min:` thresholds. Below that, fix the criteria or the judge model before trusting it — a judge you haven't calibrated is a random number generator with good grammar. # Running in CI Source: https://docs.vizra.ai/evals/ci Fail the build when your agent regresses — with cost kept firmly under control. The whole design converges here: a pipeline step that runs your evals against real models and fails when quality drops below the baseline. ## The two shapes **Pest-shaped** — evals live in your test suite, CI adds the flag: ```yaml .github/workflows/evals.yml theme={null} name: Evals on: pull_request: paths: ['app/Agents/**', 'app/Prompts/**', 'evals/**'] schedule: - cron: '0 6 * * *' # nightly drift check jobs: evals: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: shivammathur/setup-php@v2 with: php-version: '8.4' - run: composer install --no-interaction - run: php artisan migrate --force - run: ./vendor/bin/pest --evals tests/Evals env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} ``` **CLI-shaped** — no Pest in the pipeline, JSON artifact out: ```yaml The eval step theme={null} - run: php artisan evals:run --compare=baseline --baseline --output=json > eval-report.json env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} - uses: actions/upload-artifact@v4 if: always() with: { name: eval-report, path: eval-report.json } ``` `--baseline` is doing real work here. Unlike the Pest surface, the CLI never promotes a baseline on its own — without it there is nothing for `--compare=baseline` to resolve, and every run reports *"No reference run found for --compare=baseline"* forever. With it, each gate-passing run becomes the reference for the next. Exit codes do the gating: `0` clean, `1` regression/gate failure, `2` harness failure. See [CLI](/evals/cli). ## Baselines in CI Baselines are established differently on each surface — in Pest the first gate-passing run auto-baselines its suite; from the CLI you ask for it with `--baseline`. Either way, the CI question is *where the baseline lives*: comparison needs a database with history. Two workable setups: * **Shared eval database** (recommended): point CI's `DB_*` at a small persistent database (the same one your [dashboard](/evals/dashboard) reads). Every CI run lands in history, baselines persist, and the dashboard shows your CI runs' trends for free. * **Ephemeral DB, artifact-driven**: run with a throwaway database and treat the JSON report as the record. You lose `--compare` (no history to compare against) but keep absolute gates (`--min-score`, `--min-pass-rate`). [Vizra Cloud](/cloud) is the third option, and the reason it exists: CI reports its runs to a hosted history, so an ephemeral CI database costs you nothing and the same trend line covers your laptop, your branch and main. Set `VIZRA_CLOUD_KEY` as a CI secret. Trigger evals **deliberately**, not on every push: path filters (only when agents/prompts/datasets change), a nightly schedule, or a PR label. They cost real tokens and real minutes. ## Controlling cost * `samples(2–3)` is usually enough signal for CI; save `samples(10)` for investigations. * Gates skip judges on broken samples automatically — your worst PRs are your cheapest. * Set `min:` thresholds only on [calibrated judges](/evals/calibration); a miscalibrated judge burns tokens *and* trust. * The JSON report includes total and judge cost per run — chart it; eval spend that drifts up usually means a dataset quietly grew. ## Keys and safety rails Provider keys come from CI secrets, judge configuration via `EVALS_JUDGE_PROVIDER` / `EVALS_JUDGE_MODEL`. And keep a wiring check in the *normal* test lane so breakage surfaces before the expensive lane runs: ```bash Cheap lane (every push) theme={null} php artisan evals:run --dry-run # every suite, zero tokens — proves datasets parse and suites boot ``` # Class-Based Evaluations Source: https://docs.vizra.ai/evals/class-based-evaluations The full authoring surface — reusable suites, model matrices, and transform hooks. Inline `toPassEval()` config covers most evals. Reach for a class when a suite deserves a name of its own, needs a **model matrix**, or should be runnable from the [CLI](/evals/cli) and the [dashboard's](/evals/dashboard) Run button. ```bash Terminal theme={null} php artisan make:eval SupportQuality ``` ```php app/Evals/SupportQuality.php theme={null} use App\Agents\SupportBot; use Laravel\Ai\Responses\AgentResponse; use Vizra\Evals\Dataset\Dataset; use Vizra\Evals\Dataset\Row; use Vizra\Evals\Evaluation; use Vizra\Evals\Run\Gate; class SupportQuality extends Evaluation { public int $samples = 3; public function target(): mixed { return SupportBot::class; // class-string, instance, or Closure(Row): AgentResponse } public function dataset(): Dataset { return Dataset::fromJsonl(base_path('evals/support.jsonl')); } public function evaluate(Row $row, AgentResponse $response): void { $this->assertNotEmpty()->gate(); $this->assertContains($row->expected()); $this->assertCostBelow(0.02); $this->judge() ->criteria('Answers using only documented store policy.') ->minScore(7); } public function gatePolicy(): ?Gate { return new Gate(minScore: 0.8, maxRegressions: 0); } } ``` Inside `evaluate()`, the current response and row are ambient — assertion helpers take no `$response` argument, and every helper from the [assertions reference](/evals/assertions) exists as `$this->assert*()`. ## Model matrices with `across()` Run the whole dataset against multiple provider/model combos — each becomes its own series in results, comparisons, and the dashboard: ```php Compare models on your own data theme={null} public function across(): array { return [ ['provider' => 'openai', 'model' => 'gpt-5.4'], ['provider' => 'anthropic', 'model' => 'claude-sonnet-5'], ]; } ``` This is how you answer "can we move to the cheaper model?" with your own dataset instead of someone else's benchmark. ## Transform hooks Rewrite rows before they reach the agent — keeping row identity intact: ```php Prompt wrapping theme={null} public function transform(Row $row): Row { return $row->withInput("Customer message:\n{$row->input}"); } ``` Use `$row->withInput()` (and friends) rather than constructing new rows — they preserve the content hash, so [baseline comparison](/evals/baselines-and-regressions) still lines up across runs even as your transform evolves. ## Running class-based suites ```php From Pest — recorded under the class name theme={null} expect(SupportBot::class)->toPassEval(fn ($eval) => $eval->using(SupportQuality::class)); ``` ```bash From the CLI theme={null} php artisan evals:run SupportQuality ``` And from the dashboard's **Run** button, which executes discovered classes via a queued job. All three surfaces share the tables, baselines, and history. # CLI Commands Source: https://docs.vizra.ai/evals/cli The same engine without Pest — runs, baselines, scaffolding, and machine-readable output. Everything the Pest expectation does is also an artisan command — same engine, same tables, same dashboard. ## `evals:run` ```bash Terminal theme={null} php artisan evals:run SupportQuality # one class-based suite php artisan evals:run # every discovered suite (app/Evals) php artisan evals:run --filter=Support # subset by name ``` | Option | Effect | | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `--samples=5` | Override samples per row | | `--dataset=path.jsonl` | Run a JSONL file instead of the dataset the evaluation declares — the same suite against different data | | `--concurrency=1` | Sequential execution (debugging); default from config | | `--dry-run` | Execute on SDK fakes — zero tokens, validates wiring ([details](/evals/testing-without-tokens)) | | `--compare=baseline` | Diff against the baseline (also accepts a run id or `latest`) | | `--baseline` | Promote this run to baseline if its gate passes | | `--output=json` | Machine-readable versioned document on stdout (progress on stderr) | | `--report` / `--no-report` | Force or suppress pushing this run to [Vizra Cloud](/cloud/reporting), overriding the ambient configuration | | `--min-score`, `--min-pass-rate`, `--max-regressions` | Gate overrides | Unlike Pest, the CLI never promotes a baseline on its own. The first run of a suite has nothing to compare against until you pass `--baseline` (or promote a run afterwards with `evals:baseline`) — see [Baselines & Regressions](/evals/baselines-and-regressions). **Exit codes** — the CI contract: | Code | Meaning | | ---- | ---------------------------------------------------------------- | | `0` | Gate passed | | `1` | Gate failed, or regressions exceeded the allowance | | `2` | Harness failure (unknown suite, unreadable dataset, run crashed) | The terminal output is a per-row table — pass counts, score ± spread, cost — plus totals; with `--compare`, regressed/improved tables with before → after scores. ## `evals:baseline` ```bash Terminal theme={null} php artisan evals:baseline 01kyvrjyxp9ex0f3gtvznmafdn ``` Promotes any completed run to its suite's baseline (transactionally demoting the previous one). See [Baselines & Regressions](/evals/baselines-and-regressions). ## `evals:calibrate` ```bash Terminal theme={null} php artisan evals:calibrate storage/evals/labelled.jsonl --criteria="..." ``` Judge-vs-human agreement measurement — see [Judge Calibration](/evals/calibration). ## `make:eval` ```bash Terminal theme={null} php artisan make:eval SupportQuality ``` Scaffolds a [class-based evaluation](/evals/class-based-evaluations) in `app/Evals` plus a starter JSONL dataset in `evals/data/`. ## JSON output `--output=json` emits a stable, versioned document: run metadata (git sha/branch, config), aggregate summary, every row with its samples and assertion results, the comparison diff, and the gate verdict. Pipe it to artifacts storage, parse it in CI annotations, chart it — the schema is designed to be depended on. ```bash CI one-liner theme={null} php artisan evals:run SupportQuality --compare=baseline --output=json > eval-report.json ``` # Configuration Source: https://docs.vizra.ai/evals/configuration Judge defaults, gates, comparison tolerance, concurrency, and the price table behind cost tracking. ```bash Terminal theme={null} php artisan vendor:publish --tag=evals-config ``` Everything lives in `config/evals.php`: ## Judge defaults ```php config/evals.php theme={null} 'judge' => [ 'agent' => \Vizra\Evals\Judge\JudgeAgent::class, // swap for a custom judge 'provider' => env('EVALS_JUDGE_PROVIDER'), // null → SDK default provider 'model' => env('EVALS_JUDGE_MODEL'), 'min_score' => 7, // default threshold, 1–10 'skip_on_gate_failure' => true, // failed gates skip judges ], ``` Point the judge at a **different model family** than the agents under test — models grade their own family leniently. See [The LLM Judge](/evals/judge). ## Gates and comparison ```php config/evals.php theme={null} 'gate' => [ 'min_score' => null, // 0..1 — null disables that check 'min_pass_rate' => null, 'max_regressions' => 0, // enforced only when comparing ], 'compare' => [ 'epsilon' => 0.05, // score drops within this are jitter, not regressions ], ``` Overridable per evaluation (`gatePolicy()`), per Pest eval (`->gate(...)`), and per CLI run (`--min-score` etc.). Pass-rate drops always count as regressions regardless of epsilon — see [Baselines & Regressions](/evals/baselines-and-regressions). ## Execution ```php config/evals.php theme={null} 'paths' => [], // extra discovery dirs (app/Evals is always scanned) 'table_prefix' => env('EVALS_TABLE_PREFIX', 'eval_'), 'concurrency' => env('EVALS_CONCURRENCY', 5), // parallel agent invocations; 1 = sequential 'timeout' => env('EVALS_TIMEOUT'), // per prompt() call, seconds ``` Concurrency parallelizes the agent invocations (the slow part); assertions, judges, and persistence always run in the parent process. Dry runs force sequential because SDK fakes live in process memory. ## The price table Cost tracking multiplies each sample's token `Usage` against a **user-maintained** price table: ```php config/evals.php theme={null} 'pricing' => [ 'openai' => [ 'gpt-5.4' => ['input' => 1.25, 'output' => 10.00, 'cache_read' => 0.125], ], 'anthropic' => [ 'claude-sonnet-5' => ['input' => 3.00, 'output' => 15.00, 'cache_read' => 0.30, 'cache_write' => 3.75], ], ], ``` USD per million tokens; `cache_read`/`cache_write` optional. Dated model ids reported by providers (`gpt-5.4-2026-05-01`) automatically match their family entry (`gpt-5.4`). Unknown models produce a `null` cost and a single warning — **never** an error, and never a silently-wrong number. Prices change; this table is yours to keep current. ## Safety wordlist ```php config/evals.php theme={null} 'safety' => [ 'blocked_words' => [], // merged into containsNoBlockedWords() ], ``` # Dashboard Source: https://docs.vizra.ai/evals/dashboard Score trends, per-sample drill-downs with judge reasoning, and run comparison — in your browser. `vizra/evals-ui` renders everything the engine records. Since every run persists, the dashboard needs zero extra instrumentation — install it and your history is already there. ```bash Terminal theme={null} composer require vizra/evals-ui ``` Visit `/evals`. That's the whole install: the stylesheet ships with the package (no publishing, no build step in your app), and it reads the tables `vizra/evals` already writes. Requires Livewire 3.7+ or 4.x. ## What you get Every suite with its latest score, pass rate, trend sparkline, and baseline — the health check at a glance. Per-row results sorted worst-first with filters and search; expand any row to every sample's response, tool calls, tokens, cost — and each assertion's expected vs actual. Every judge score with its persisted reasoning, one click deep. Usually the fastest route to the prompt fix. Any run against the baseline (or any other run): score/pass-rate deltas, regressed and improved tables with before → after values. Baselines are managed from the UI too — promote any completed run, or clear one, without touching the CLI. ## Access control Horizon-style: open in the `local` environment, denied everywhere else until you grant access: ```php app/Providers/AppServiceProvider.php theme={null} use Illuminate\Support\Facades\Gate; Gate::define('viewEvalsDashboard', fn ($user) => $user->isAdmin()); // or take full control: \Vizra\EvalsUi\EvalsUi::auth(fn ($request) => /* ... */); ``` ## Triggering runs from the browser The suite pages include a **Run evaluation** button for [class-based suites](/evals/class-based-evaluations) — it dispatches a queued job (unique per suite, so double-clicks can't start concurrent runs) and shows live progress. A queue worker must be running: ```bash Terminal theme={null} php artisan queue:work ``` For local tinkering, `EVALS_UI_QUEUE_CONNECTION=sync` runs the job inline in the request — keep that to dry runs or small suites. ## Configuration ```bash Terminal theme={null} php artisan vendor:publish --tag=evals-ui-config ``` Route path (default `evals`), middleware, queue connection, score-badge thresholds, poll interval, and stale-run detection (runs stuck "running" past a threshold get flagged with a mark-failed action). # Datasets Source: https://docs.vizra.ai/evals/datasets JSONL, CSV, arrays, Eloquent — or your actual production traffic. A dataset yields rows; each row is one thing your agent should handle well. Pass anything below to `->dataset(...)` in Pest, or return it from a class-based evaluation's `dataset()` method. ## JSONL — the preferred format One JSON object per line, streamed lazily (large files never load fully into memory): ```json evals/support.jsonl theme={null} {"input": "What is your refund policy?", "expected": "30 days"} {"input": "Do you ship to France?", "expected": "France", "category": "shipping"} {"messages": [{"role": "user", "content": "Hi, I ordered a lamp"}, {"role": "assistant", "content": "How can I help?"}, {"role": "user", "content": "Can I return it?"}], "expected": "30 days"} ``` Recognised keys: | Key | Meaning | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `input` | The user prompt (single-turn rows) | | `messages` | `[{role, content}, …]` — the final entry must be a `user` turn and becomes the prompt; earlier turns are replayed as conversation context ([multi-turn](/evals/multi-turn)) | | `expected` | Free-form reference data — a string, or an object read with `$row->expected('key')` | | `hash` | Optional. Overrides the computed [content hash](#row-identity) with one you supply — for datasets that already know their row identities. Must be a sha256 hex digest or the row is rejected | | anything else | Lands in `$row->meta('key')` | `hash` is identity, not data — a column of that name is **not** available as meta, and a value that isn't a 64-character sha256 digest aborts the read. Rename the column if you meant it as your own field. Malformed lines fail loudly with their line number — no silent row-dropping. ## CSV ```csv evals/support.csv theme={null} prompt,expected,category "What is your refund policy?","30 days",billing "Do you ship to France?",,shipping ``` Header row required; `prompt` and `expected` columns by default (both configurable via `Dataset::fromCsv($path, inputColumn: ..., expectedColumn: ...)`); other columns become meta. CSV exists so non-developers can own datasets in a spreadsheet. ## Arrays and Eloquent ```php Inline rows theme={null} ->dataset([ 'A bare string is a prompt', ['input' => 'What is your refund policy?', 'expected' => '30 days'], ]) ``` ```php From your database theme={null} use Vizra\Evals\Dataset\Dataset; ->dataset(Dataset::fromEloquent( Ticket::query()->where('resolved', true), fn ($ticket) => ['input' => $ticket->question, 'expected' => $ticket->resolution], )) ``` ## Production conversations The differentiated one — build the dataset from conversations your agent **actually had**, stored by the Laravel AI SDK's conversation feature: ```php Replay real traffic theme={null} expect(SupportBot::class)->toPassEval(fn ($eval) => $eval ->fromConversations(take: 50) ->judge('At least as helpful as the reply we actually sent.') ); ``` Each stored conversation becomes a multi-turn row: the latest user turn is the prompt, prior turns replay as context, and the reply your agent actually gave is exposed as `$row->expected()` — ideal for judging new prompts against real production answers. The standalone `Dataset::fromConversations(SupportBot::class)` form adds `->latest()`, `->take(n)`, and `->where(...)` refinement. ## Row identity Every row gets a **content hash** over input + messages + expected (not meta). It's how [baseline comparison](/evals/baselines-and-regressions) joins rows across runs — stable across file reordering, unaffected by meta changes, changed only when the row's actual content changes. # How It Works Source: https://docs.vizra.ai/evals/how-it-works The lifecycle of an eval run — from dataset rows to recorded, baseline-compared results. One run, end to end: ```text The lifecycle of a run theme={null} dataset ──► rows ──► × samples ──► agent invoked ──► deterministic assertions │ gate failed? ──► judges SKIPPED (no tokens) │ gates passed ▼ LLM judges run │ ▼ everything persisted (response, tool calls, usage, cost, per-assertion results, judge reasoning) │ ▼ aggregate: row pass rates + score mean/stddev ──► run summary │ ▼ compare vs baseline ──► gate ──► pass / fail ``` ## The pieces **Rows.** A dataset yields rows: an `input` (the prompt), optional prior `messages` for [multi-turn](/evals/multi-turn), optional `expected` reference data, and free-form meta. Every row gets a **content hash** (input + messages + expected) so the same logical row is tracked across runs even when your file is reordered. **Samples.** Agents are nondeterministic, so each row runs `samples` times. A row's result is a pass rate and a score distribution (mean ± stddev) — never a single boolean. **Assertions first, judges last.** Deterministic checks (substrings, tool calls, cost…) cost nothing and run first. An assertion marked `->gate()` that fails hard-fails the sample — and skips its LLM judges entirely, so broken samples never spend judge tokens. Judges then score the surviving samples with persisted reasoning. **Persistence.** Every sample writes its full context to your database: response text, structured output, tool calls, token usage, finish reason, duration, cost, and each assertion's expected/actual/score — plus the judge's reasoning. This is what powers the [dashboard](/evals/dashboard) and everything below. **Baselines.** In Pest, the first run that passes its gate becomes the suite's baseline automatically. From the CLI it is explicit — pass `--baseline` (or promote a run later with `evals:baseline` or the dashboard). Later runs are joined to the baseline row-by-row (by content hash) and classified: regressed, improved, newly failing. See [Baselines & Regressions](/evals/baselines-and-regressions). **The gate.** Pass/fail is a run-level policy — minimum score, minimum pass rate, maximum regressions — applied after aggregation. In Pest, a failed gate fails the test with row-level detail; in the [CLI](/evals/cli), it sets the exit code. ## Where things run | Surface | What it's for | | -------------------------------------- | ---------------------------------------------------------------------------------- | | `expect(...)->toPassEval(...)` in Pest | The main authoring surface — evals in your test suite, `--evals` to execute | | `php artisan evals:run` | The same engine without Pest — class-based suites, `--dry-run`, JSON output for CI | | Dashboard "Run" button | Queued execution of class-based suites from the browser | All three land in the same tables and the same dashboard. Deep-dive the two halves of the model: [Scoring](/evals/scoring) for how numbers are computed, and [Baselines & Regressions](/evals/baselines-and-regressions) for how runs are compared. # Installation Source: https://docs.vizra.ai/evals/installation Requirements, package setup, and what gets added to your application. ## Requirements | Requirement | Version | | ------------------------------------------- | -------------------------------------- | | PHP | 8.4+ | | Laravel | 12+ | | [laravel/ai](https://github.com/laravel/ai) | ^0.10 | | Pest (for the testing surface) | ^4.0 on Laravel 12, ^5.0 on Laravel 13 | The engine also works without Pest via the [CLI](/evals/cli) — Pest is a `suggest`, not a hard dependency. Pest 5 requires `symfony/process ^8.1`, which Laravel 12 cannot satisfy — so Laravel 12 tops out at Pest 4 and Laravel 13 gets Pest 5. `toPassEval()` behaves identically on both, and nothing in this documentation is Pest-5-only. If you want to run [`pestphp/pest-plugin-evals`](/evals/vs-pest-plugin-evals) alongside this package, that one needs Pest 5 and pins `laravel/ai` tightly — currently `>=0.10.2 <0.11.0` — so you will need Laravel 13 and a recent `laravel/ai`. ## Install ```bash Terminal theme={null} composer require vizra/evals --dev php artisan migrate ``` Install with `--dev` when evals only run from your test suite and CI. If you trigger runs from the [dashboard](/evals/dashboard) in a deployed environment, require it as a regular dependency instead. Three tables are created (prefix configurable via `evals.table_prefix`): | Table | Holds | | ------------------------ | ------------------------------------------------------------------------------------------------------- | | `eval_runs` | One row per run: suite, git sha/branch, config snapshot, aggregate score/pass rate, cost, baseline flag | | `eval_row_results` | One row per sample: response text, tool calls, token usage, finish reason, score, duration, cost | | `eval_assertion_results` | One row per assertion per sample: status, score, weight, gate flag, expected/actual, judge reasoning | ## Configuration (optional) ```bash Terminal theme={null} php artisan vendor:publish --tag=evals-config ``` See [Configuration](/evals/configuration) for judge defaults, gates, the comparison tolerance, and the price table behind cost tracking. ## The dashboard (optional) ```bash Terminal theme={null} composer require vizra/evals-ui ``` Adds the `/evals` dashboard — no publishing, no asset build in your app. See [Dashboard](/evals/dashboard). ## Provider credentials Evals invoke your agents through `laravel/ai`, so whatever providers your agents use must be configured as usual (e.g. `OPENAI_API_KEY`). The [judge](/evals/judge) is an LLM call too and uses the provider configured in `evals.judge`. Nothing needs credentials until you actually run with `--evals` — normal test runs skip evals entirely, and [fakes](/evals/testing-without-tokens) cover everything else. # The LLM Judge Source: https://docs.vizra.ai/evals/judge Structured-output scoring with reasoning you keep — no regex parsing, no black boxes. Some qualities can't be asserted with string checks: accuracy against policy, tone, helpfulness. For those, an LLM judge scores each sample 1–10 against your criteria — through **structured output**, so there's no fragile response parsing anywhere — and its reasoning is persisted with every score. ```php The basics theme={null} ->judge('Answers using only documented store policy. Invents nothing.', min: 7) ``` The judge sees the row's input (or full multi-turn transcript), the `expected` reference data when present, the agent's response, and — since 0.3 — [the instructions the agent under test was given](#the-judge-reads-the-agents-instructions). A sample passes the judge when its raw score meets `min` (default from `evals.judge.min_score`, 7). The normalized score (`raw / 10`) feeds the [sample score](/evals/scoring). ## Dimensions Require per-dimension minimums on top of the overall score: ```php Multi-dimensional theme={null} ->judge('Handles the support request well.', min: 7, dimensions: [ 'accuracy' => 7, 'tone' => 6, ]) ``` The judge returns a score per dimension; any dimension below its threshold fails the sample even if the overall score passes. Dimension scores are recorded too. ## The judge reads the agent's instructions Since **0.3**, the agent's system prompt is passed to the judge automatically. Nothing to configure. It matters more than it sounds. Without it the judge grades prose against a rulebook it cannot read, so the agent's own documented behaviour reads as invention: > *agent:* "You have 30 days from delivery to return an item, and it must be in its original condition" > *judge:* "introduces multiple unstated policies" Every fact in that answer was in the agent's instructions. On an 11-row suite, the same agent and the same model scored **63.7%** with a criteria that assumed the judge knew the policy and **96.2%** once the policy was pasted into the criteria by hand. Passing the instructions is that workaround, done once and kept in step automatically. The instructions are resolved **once per evaluation**, not per sample. They are skipped for a `Closure` target, which has no instructions to read, and for an agent that will not resolve — a judge without context is a worse grade, not a failed run. **Judged scores move when you upgrade to 0.3.** The judge now sees something it did not see before on every `judge()` call — generally scoring higher, where failures were the judge not knowing what the agent was permitted to do. Re-baseline before reading a comparison across the upgrade. Some criteria are deliberately about the response alone — tone, reading age, format — where the system prompt would only bias the grade or spend tokens. In a [class-based evaluation](/evals/class-based-evaluations), turn it off per judge: ```php Inside evaluate() theme={null} $this->judge() ->criteria('Reads at a ninth-grade level.') ->withoutTargetInstructions() ->minScore(7); ``` The Pest `->judge(...)` surface always includes them; drop to the class-based form for a criteria that needs them gone. ## Judges run last — and gates protect your budget Judges execute only after all deterministic assertions, and **a failed `->gate()` skips them entirely** for that sample. An empty response never costs judge tokens. (Configurable via `evals.judge.skip_on_gate_failure`.) ## Choosing the judge model ```php Per judge theme={null} ->judge('...', min: 7, provider: 'anthropic', model: 'claude-sonnet-5') ``` Defaults come from `evals.judge.provider` / `evals.judge.model` (`EVALS_JUDGE_PROVIDER` / `EVALS_JUDGE_MODEL`). **Point the judge at a different model family than the agent under test.** Models grade their own family's output leniently; cross-family judging is the cheapest bias reduction you can buy. **Budget for the judge, not for the agent.** The judge reads the criteria, the input, the response and the agent's instructions, and it runs once per judged sample — so it routinely costs several times the thing it is grading. On one measured run, a `gpt-5` judge grading a `gpt-5-mini` agent cost **$0.1143 against the agent's $0.0080** — 14×. Nothing is wrong when you see that; it is what judging costs. Both figures are broken out in the run summary and in the [dashboard](/evals/dashboard), so pick the judge model with the total in view — a smaller judge on a well-calibrated criteria is often the difference between running a suite nightly and running it monthly. ## Custom judge agents The default judge is a structured-output agent returning `{score, reasoning}` (+ dimensions). Supply your own for domain-specific rubrics: ```php app/Evals/Judges/PolicyJudge.php theme={null} use Illuminate\Contracts\JsonSchema\JsonSchema; use Laravel\Ai\Contracts\Agent; use Laravel\Ai\Contracts\HasStructuredOutput; use Laravel\Ai\Promptable; class PolicyJudge implements Agent, HasStructuredOutput { use Promptable; public function instructions(): string { return 'You are a strict compliance reviewer for Acme Store...'; } public function schema(JsonSchema $schema): array { return [ 'score' => $schema->integer()->min(1)->max(10)->required(), 'reasoning' => $schema->string()->required(), ]; } } ``` ```php Using it theme={null} ->judge('Complies with the returns policy.', min: 8, using: PolicyJudge::class) ``` ## Pairwise comparison In [class-based evaluations](/evals/class-based-evaluations), judge one response *against another* — e.g. the reply your agent gave in production: ```php Inside evaluate() theme={null} $this->judge() ->comparedTo($row->expected()) // reference text, or another AgentResponse ->prefer('actual'); // fail unless the new response wins ``` Win scores `1.0`, tie `0.5`, loss `0.0`. Only an outright win passes: `prefer` defaults to `'actual'`, so a tie fails the assertion whether or not you called `->prefer()`. The `0.5` still counts toward the sample score — a draw is worth something, it just isn't a pass. ## Reasoning is the debugging payload Every judgment stores its reasoning alongside the score. When a sample scores 6/10, the [dashboard](/evals/dashboard) shows *why* in the judge's own words — which is usually the fastest route to the prompt fix. Don't trust an uncalibrated judge. Measure its agreement with human labels before you gate CI on it — see [Judge Calibration](/evals/calibration). # Migrating from Vizra ADK Source: https://docs.vizra.ai/evals/migrate-from-vizra-adk Moving from the ADK's CSV-based evaluations to recorded, baseline-compared evals. The ADK's evaluation system ([legacy docs](/testing/evaluations)) ran single-pass CSV evals against ADK agents and wrote CSV results. Vizra Evals is its successor, rebuilt on the official [Laravel AI SDK](https://github.com/laravel/ai): sampled runs, database persistence, baselines, and a [dashboard](/evals/dashboard). It's a reposition, not a rename — the concepts map, the code changes. ## The mapping | Vizra ADK | Vizra Evals | | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `public string $agentName = 'support'` + `Agent::run()` | A `laravel/ai` agent target: `target(): mixed` or `expect(SupportBot::class)` | | `BaseEvaluation` subclass required | Optional — inline `toPassEval()` covers most evals; [classes](/evals/class-based-evaluations) remain for matrices | | `preparePrompt(array $csvRowData): string` | `transform(Row $row): Row` (optional; preserves row identity) | | `evaluateRow(array $row, string $llmResponse)` | `evaluate(Row $row, AgentResponse $response)` — the full response object, not a string | | `$this->assertResponseContains($response, 'x')` | `$this->assertContains('x')` — the response is ambient; no parameter threading | | CSV datasets only | JSONL (preferred), CSV still supported, arrays, Eloquent, [production conversations](/evals/multi-turn) | | One run per row, pass/fail | `samples(n)` — pass rates and score distributions ([scoring](/evals/scoring)) | | `vizra:run:eval` + CSV output in `storage/` | `evals:run` / `pest --evals` + database + dashboard + `--output=json` | | Always exit 0 | Exit codes gate CI: 0/1/2 ([CLI](/evals/cli)) | | `assertLlmJudge*` with regex response parsing | [`judge()`](/evals/judge) on structured output — no parsing, reasoning persisted, calibration | ## Renamed for honesty | Old | New | Why | | ---------------- | ------------------------ | ---------------------------------------- | | `assertNotToxic` | `containsNoBlockedWords` | It's a wordlist — the name now says so | | `assertNoPII` | `noObviousPII` | Regex-grade detection, named accordingly | ## Removed — deliberately `assertResponseHasPositiveSentiment` (keyword counting), `assertGrammarCorrect` (regex heuristics), and `assertReadabilityLevel` (hand-rolled Flesch-Kincaid) measured nothing real. Each is one properly-done judge call away: ```php The honest replacement theme={null} ->judge('The response reads as positive and professionally written.', min: 7) ``` ## A worked migration ```php Before — ADK theme={null} class SupportEvaluation extends BaseEvaluation { public string $agentName = 'customer_support'; public string $csvPath = 'app/Evaluations/data/support.csv'; public function evaluateRow(array $csvRowData, string $llmResponse): array { $this->resetAssertionResults(); $this->assertResponseContains($llmResponse, $csvRowData['expected_response']); // ... manual final_status bookkeeping } } ``` ```php After — Vizra Evals theme={null} it('handles support questions', function () { expect(SupportBot::class)->toPassEval(fn ($eval) => $eval ->dataset(base_path('evals/support.jsonl')) ->samples(3) ->assert(fn ($a, $row) => $a->contains($row->expected())) ->judge('Answers accurately without inventing policy.', min: 7) ->gate(maxRegressions: 0) ); }); ``` No manual resets, no status bookkeeping, no CSV parsing — and the run is recorded, baselined, and visible in the dashboard. Your ADK **agents** need porting to the Laravel AI SDK to be evaluated directly (the SDK's `Agent` contract + `Promptable`). Until then, the `Closure` target is the bridge — wrap anything that can produce an `AgentResponse`. # Multi-Turn Conversations Source: https://docs.vizra.ai/evals/multi-turn Evaluate agents in the middle of a conversation — including conversations from production. Real support and chat agents don't answer isolated prompts — they answer *turn seven*, with six turns of context behind them. Vizra Evals replays that context for real. ## Multi-turn rows ```json evals/support.jsonl theme={null} {"messages": [ {"role": "user", "content": "Hi, I ordered a lamp last week"}, {"role": "assistant", "content": "Thanks for reaching out! How can I help with your lamp?"}, {"role": "user", "content": "Can I still return it?"} ], "expected": "30 days"} ``` The final entry must be a `user` turn — it becomes the prompt. Everything before it is replayed as genuine conversation history through the Laravel AI SDK's conversation channel, so the agent responds exactly as it would mid-conversation in production: aware the customer bought a lamp, answering "can I return **it**" correctly. Replay is real, not prompt-stuffing: prior turns are hydrated into SDK `Message` objects and delivered through the same mechanism the SDK uses for stored conversations. The agent's instructions, tools, and middleware all still apply. ## Replaying production traffic If your agents use the SDK's conversation persistence (`RemembersConversations`), your `agent_conversations` tables are a free eval dataset: ```php tests/Evals/SupportBotTest.php theme={null} expect(SupportBot::class)->toPassEval(fn ($eval) => $eval ->fromConversations(take: 50) ->judge('At least as helpful and accurate as the reply we actually sent.') ); ``` For each stored conversation of this agent: the latest user turn becomes the prompt, everything before it replays as context, and **the reply your agent actually gave** becomes `$row->expected()`. Two powerful patterns fall out: ```php Judge against the production answer theme={null} ->judge('Answers the customer at least as well as the reference reply.', min: 7) // the judge sees $row->expected() as reference information automatically ``` ```php Regression-test a prompt change against real traffic theme={null} ->fromConversations(take: 100) ->samples(2) ->gate(maxRegressions: 0) // first run baselines real traffic; future prompt changes diff against it ``` The standalone form gives you query refinement: `Dataset::fromConversations(SupportBot::class)->latest()->take(50)->where('title', 'like', '%refund%')`. Eval runs **never write** to your conversation tables — replay is read-only by construction, so running evals can't pollute production history. ## Assertion behavior on multi-turn rows Everything works the same: `$row->input` is the final user turn, `$row->messages` holds the prior turns, and all [assertions](/evals/assertions) run against the agent's real `AgentResponse`. In the recorded results and dashboard, multi-turn rows display as the final question with a "(after N prior turns)" marker rather than a wall of JSON. # Writing Pest Evals Source: https://docs.vizra.ai/evals/pest The toPassEval() expectation — everything is a Pest test, and the recorder works underneath. Evals live in your test suite. There's no separate framework to learn: `toPassEval()` is an expectation like any other, and the recording, baselines, and dashboard happen behind it. ```php tests/Evals/SupportBotTest.php theme={null} use App\Agents\SupportBot; it('answers support questions from documented policy', function () { expect(SupportBot::class)->toPassEval(fn ($eval) => $eval ->dataset(base_path('evals/support.jsonl')) ->samples(3) ->assert(fn ($a, $row) => $a ->notEmpty()->gate() ->contains($row->expected())) ->judge('Answers using only documented store policy.', min: 7) ->gate(minScore: 0.8, maxRegressions: 0) ); }); ``` ## The subject `expect(...)` accepts a Laravel AI agent **class-string**, an agent **instance**, or a **`Closure(Row $row): AgentResponse`** — the closure escape hatch lets you evaluate anything that can produce an `AgentResponse`. ## The `$eval` builder | Method | What it does | | ------------------------------------------------ | -------------------------------------------------------------------------------------------- | | `dataset($pathOrArrayOrDataset)` | `.jsonl`/`.csv` path, inline array of rows, or any [`Dataset`](/evals/datasets) | | `fromConversations(take: 50)` | Build the dataset from the agent's stored [production conversations](/evals/multi-turn) | | `samples(int)` | Runs per row (default 1 — set it higher; see [Scoring](/evals/scoring)) | | `assert(fn ($a, $row, $response))` | Deterministic checks per sample — see [Assertions](/evals/assertions) | | `judge($criteria, min: 7, ...)` | LLM-judged criteria — see [The Judge](/evals/judge); repeatable | | `gate(minScore:, minPassRate:, maxRegressions:)` | Run-level pass policy | | `across([...])` | Provider/model matrix — each combo becomes its own series | | `suite('name')` | Override the recording identity (defaults to the test's description) | | `using(SupportQuality::class)` | Run a full [class-based evaluation](/evals/class-based-evaluations) instead of inline config | | `concurrency(int)` | Parallel agent invocations for this run | ## Skip semantics Evals call real models, so they never run by accident: ```bash Terminal theme={null} ./vendor/bin/pest # all toPassEval tests are skipped ./vendor/bin/pest --evals # they run PEST_EVALS=1 ./vendor/bin/pest # equivalent, for CI matrices ``` The `--evals` flag and `PEST_EVALS` variable are the **same contract** used by `pestphp/pest-plugin-evals`, so the two plugins coexist cleanly in one suite — whichever consumes the flag, both see eval mode. See [Vizra Evals vs pest-plugin-evals](/evals/vs-pest-plugin-evals). ## What failure looks like A failed gate fails the test with the receipts — aggregate score and pass rate, the run id (look it up in the [dashboard](/evals/dashboard)), regressed rows with before → after scores, and the worst failing rows with sample counts. A run where every sample errored always fails, whatever the gate says. ## Suites and recording Each `toPassEval` test records under a **suite** named after the test (e.g. `pest: answers support questions from documented policy`) — that's the identity baselines and history attach to. Rename the test and you start a fresh history, so pin it with `->suite('support-quality')` if you expect descriptions to churn. Group your evals (`->group('evals')` or a `tests/Evals` directory) so you can run them in a dedicated CI job with its own schedule and budget — see [Running in CI](/evals/ci). # Quickstart Source: https://docs.vizra.ai/evals/quickstart From install to a baseline-gated, CI-ready agent eval in five minutes. This walkthrough takes you from nothing to a recorded, regression-gated eval of a real agent. ## Step 1: Install ```bash Terminal theme={null} composer require vizra/evals --dev php artisan migrate ``` The migration adds three tables (`eval_runs`, `eval_row_results`, `eval_assertion_results`) where every run is recorded. Requirements: PHP 8.4+, Laravel 12+, [laravel/ai](https://github.com/laravel/ai), and Pest 5. ## Step 2: Write a Pest test Evals are ordinary Pest tests — `toPassEval()` is just another expectation: ```php tests/Evals/SupportBotTest.php theme={null} use App\Agents\SupportBot; it('answers support questions from documented policy', function () { expect(SupportBot::class)->toPassEval(fn ($eval) => $eval ->dataset(base_path('evals/support.jsonl')) ->samples(3) ->assert(fn ($a, $row) => $a ->notEmpty()->gate() ->contains($row->expected())) ->judge('Answers using only documented store policy.', min: 7) ->gate(minScore: 0.8, maxRegressions: 0) ); }); ``` `SupportBot` is any [Laravel AI SDK](https://github.com/laravel/ai) agent — a class implementing `Agent` with the `Promptable` trait. A `Closure` or agent instance works too. ## Step 3: Give it data One JSON object per line: ```json evals/support.jsonl theme={null} {"input": "What is your refund policy?", "expected": "30 days"} {"input": "Do you ship to France?", "expected": "France"} {"messages": [{"role": "user", "content": "Hi, I ordered a lamp"}, {"role": "assistant", "content": "How can I help?"}, {"role": "user", "content": "Can I return it?"}], "expected": "30 days"} ``` `input` is the prompt. Or provide `messages` — the final user turn becomes the prompt and the earlier turns are replayed as real conversation context. `expected` is free-form reference data; any other key lands in `$row->meta()`. ## Step 4: Run it ```bash Terminal theme={null} ./vendor/bin/pest # skipped — evals cost real tokens, so they never run by accident ./vendor/bin/pest --evals # runs against the real model ``` Each row runs 3 times. Deterministic checks run first; the judge only runs on samples whose gates passed. Everything is recorded. In Pest, the **first run that passes its gate automatically becomes the suite's baseline.** No setup — regression detection is armed from your second run onward. (From the [CLI](/evals/cli) it's explicit: pass `--baseline`.) ## Step 5: Break something, and watch it get caught Change your agent's prompt for the worse and run again: ```text Terminal theme={null} FAILED Tests\Evals\SupportBotTest > answers support questions from documented policy Eval [pest: answers support questions from documented policy] — score 61.7%, pass rate 33.3% across 9 samples (run 01kyw…). Gate failed: 2 rows regressed against the reference run (allowed: 0). ↓ regressed: "What is your refund policy?" 96.7% → 51.7% ↓ regressed: "Can I return it?" 93.3% → 55.0% ``` Row-level receipts: which inputs got worse, and by how much. That's the difference between "a test failed" and knowing what to fix. ## Step 6: Watch it over time ```bash Terminal theme={null} composer require vizra/evals-ui ``` Visit `/evals` for score trends per suite, per-sample drill-downs (including the judge's reasoning for every score), and side-by-side run comparison. See [Dashboard](/evals/dashboard). ## Next steps What actually happens when an eval runs — and where the data goes. Everything you can check, from substrings to real tool calls and cost. JSONL, CSV, Eloquent — and replaying production conversations. Fail the build when your agent regresses. # Scoring Model Source: https://docs.vizra.ai/evals/scoring How samples become scores, scores become rates, and rates become a pass or fail. Scores are the primary output; pass/fail is a policy applied on top. Here's the exact math, bottom to top. ## Assertion scores * **Deterministic assertions** score `1.0` (passed) or `0.0` (failed). * **Judge assertions** score `raw / 10`, normalized to `0–1`. Pairwise comparisons score win `1.0` / tie `0.5` / loss `0.0`. * **Skipped or errored** assertions have no score (`null`) and are excluded from means. * Every assertion can carry a `->weight(float)` (default `1.0`). ## Sample score A sample is one invocation of the agent for one row. * **If any gated assertion failed (or errored): the sample scores `0.0` and fails.** Gates are binary preconditions — they're *excluded* from the score mean, because "the response wasn't empty" isn't quality signal. * Otherwise: `score = Σ(weightᵢ × scoreᵢ) / Σ(weightᵢ)` over non-gate assertions with scores. * A sample whose only assertions were passing gates scores `1.0`. * A sample whose agent invocation or evaluation threw is an **error**: no score, counted separately. ```php Weighting example theme={null} $a->contains($row->expected()) // weight 1.0 ->toolCalled('lookup_policy')->weight(2.0); // counts double // judge at min: 7 scoring 8/10 → 0.8, weight 1.0 // sample score = (1.0×1 + 1.0×2 + 0.8×1) / 4 = 0.95 ``` ## Row result Each row (per model combo) aggregates its samples: * `pass_rate` = passed samples ÷ all samples (errors count in the denominator) * `score_mean`, `score_stddev` = over samples with scores (errors excluded from the mean) The stddev matters: a row at `90% ± 2` and a row at `90% ± 25` are very different agents. ## Run result Rows aggregate with equal weight: run `score_mean`/`score_stddev` over row means, run `pass_rate` as the mean of row pass rates — plus totals (rows, samples, errors, cost, judge cost, duration). All of it is persisted on the run's summary; nothing is recomputed later. ## The gate ```php In Pest theme={null} ->gate(minScore: 0.8, minPassRate: 0.9, maxRegressions: 0) ``` * `minScore` / `minPassRate` — thresholds on the run aggregates (`0–1`). * `maxRegressions` — enforced against the suite's [baseline](/evals/baselines-and-regressions) when one exists. Defaults come from `config('evals.gate')`; class-based evaluations can define `gatePolicy()`; the [CLI](/evals/cli) accepts `--min-score`, `--min-pass-rate`, `--max-regressions`. In Pest a failed gate fails the test; in the CLI it sets exit code `1`. An eval where **every** sample errored always fails, regardless of gate configuration — a run that proves nothing must never pass. # Testing Without Tokens Source: https://docs.vizra.ai/evals/testing-without-tokens Develop and test your evals end-to-end with zero network calls, using the Laravel AI SDK's own fakes. Evals cost tokens — but *developing* them shouldn't. Everything runs offline against the Laravel AI SDK's fakes, which is also exactly how you test eval wiring in your own suite. ## Fake the agent ```php tests/Evals/SupportBotTest.php theme={null} use App\Agents\SupportBot; it('catches missing policy answers', function () { SupportBot::fake(fn (string $prompt) => str_contains($prompt, 'refund') ? 'Full refunds within 30 days of delivery.' : 'Yes, we ship to France.'); // now run the eval inline — no network, instant putenv('PEST_EVALS=1'); // ... }); ``` Fakes accept canned strings (consumed in order), closures, or full `TextResponse` objects with real `Usage`, `Meta`, and tool calls — so even `costBelow()` and `toolCalled()` assertions are testable offline. `SupportBot::assertPrompted(...)` then verifies exactly what was sent. ## Fake the judge The judge is an agent too: ```php Faking judge scores theme={null} use Laravel\Ai\Ai; use Vizra\Evals\Judge\JudgeAgent; Ai::fakeAgent(JudgeAgent::class, fn () => [ 'score' => 9, 'reasoning' => 'Sticks to the documented policy.', ]); ``` ## Multi-turn rows just work When the target agent is faked, [multi-turn](/evals/multi-turn) rows route straight to the fake — your canned responses and `assertPrompted()` checks behave identically for single- and multi-turn rows, and nothing leaks to the network. ## Dry runs from the CLI For [class-based suites](/evals/class-based-evaluations), `--dry-run` fakes everything automatically: ```bash Terminal theme={null} php artisan evals:run SupportQuality --dry-run ``` The entire suite executes — datasets parse, assertions run, results persist — with zero tokens. Scores are meaningless (responses are auto-generated placeholders); the point is proving the wiring before you spend. Use it whenever you add rows or touch an evaluation class. ## Keep faked runs out of Cloud A faked agent produces real-looking scores from canned strings. If you use [Vizra Cloud](/cloud), say so, and the run stays out of the history: ```php Wiring test, not measurement theme={null} expect(SupportBot::class)->toPassEval(fn ($eval) => $eval ->dataset(base_path('evals/support.jsonl')) ->report(false) ->assert(fn ($a) => $a->notEmpty()) ); ``` The CLI's `--dry-run` is already excluded automatically — this is only needed for Pest evals that fake the agent themselves, because nothing else distinguishes them from a real run. Local recording is unaffected either way. The pattern to internalize: **fakes for wiring, `--evals` for measurement.** The vizra/evals package's own test suite — 150+ tests — runs entirely on fakes, no API keys anywhere. # Vizra Evals vs pest-plugin-evals Source: https://docs.vizra.ai/evals/vs-pest-plugin-evals Two expectation sets, one test file — quick checks in one, recorded measurement in the other. Pest 5 ships its own evals plugin (`pestphp/pest-plugin-evals`), and the obvious question is how the two relate. Short answer: **different layers, happy coexistence** — both are "just expectations," they share the `--evals` flag, and they can live in the same file. ```php tests/Evals/SupportBotTest.php theme={null} use App\Agents\SupportBot; // pest-plugin-evals: quick text-level checks, nothing stored it('stays on topic', function () { expect(SupportBot::class) ->prompt('What is your refund policy?') ->toContain('30 days') ->toBeRelevant(); }); // vizra/evals: dataset-driven, sampled, judged — and recorded it('handles the support corpus', function () { expect(SupportBot::class)->toPassEval(fn ($eval) => $eval ->dataset(base_path('evals/support.jsonl')) ->samples(3) ->assert(fn ($a, $row) => $a->toolCalled('lookup_policy')->contains($row->expected())) ->judge('Answers using only documented policy.', min: 7) ->gate(maxRegressions: 0) ); }); ``` Both are skipped without `--evals`. This is worth understanding rather than trusting, because the two plugins genuinely fight over that flag: each one *pops* `--evals` out of the argument list, so only one of them ever sees it. They coexist because both set `PEST_EVALS=1` in the environment **before** popping, and both read that variable in `isEvalMode()`. Whichever runs first enables the other. Verified with both installed side by side: without the flag both skip, with it both run. `pest-plugin-evals` requires Pest 5 and pins `laravel/ai` to a narrow range — currently `>=0.10.2 <0.11.0`. Running the two together therefore means Laravel 13, and a `laravel/ai` upgrade may block one until the other catches up. ## When to reach for which | Question | Use | | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | "Does a sane answer come back for this one prompt?" | `pest-plugin-evals` — one line, ephemeral, perfect for smoke checks | | "How does the agent do across my whole dataset?" | `toPassEval()` with a [dataset](/evals/datasets) | | "Did this prompt change make anything *worse*?" | `toPassEval()` — [baselines](/evals/baselines-and-regressions) need recorded history | | "Did it call the right tools with the right arguments?" | `toPassEval()` — assertions run against the real `AgentResponse` tool calls | | "How does it behave mid-conversation / on production traffic?" | `toPassEval()` — [multi-turn replay](/evals/multi-turn); pest-plugin-evals runs are single-turn | | "What's this costing per sample?" | `toPassEval()` — token usage and cost are recorded and assertable | ## The structural difference `pest-plugin-evals` operates on **response text**: its scorers receive strings, its `repeat(n)` requires every sample to pass (a consistency gate), and results exist only in the terminal. That's exactly right for its job — fast, ergonomic checks. Vizra Evals operates on **the response object and time**: sampled score distributions rather than all-or-nothing, assertions against real tool calls and usage, multi-turn context, and every run persisted — which is what makes baselines, regression gates, trend charts, and the [dashboard](/evals/dashboard) possible. A perfectly good setup: pest-plugin-evals expectations as cheap per-prompt smoke checks, and one `toPassEval()` per agent as the recorded, baseline-gated measurement. They'll never conflict. # Customer Support Agent Source: https://docs.vizra.ai/examples/customer-support-agent Build a complete customer support agent with order lookup and refund processing **Vizra ADK is in maintenance mode.** It continues to work and receives compatibility fixes, but no new features are planned. For evaluating AI agents — now built on the official [Laravel AI SDK](https://github.com/laravel/ai) — check out [Vizra Evals](/), its successor for the evaluation use-case. Coming from ADK? See the [migration guide](/evals/migrate-from-vizra-adk).