Agentic Data Analysis: A Deep Dive into LAMBDA and Vanna Architectures

Introduction

The intersection of large language models and data analysis represents one of the most impactful applications of AI. Rather than requiring users to write SQL, Python, or R, agentic data analysis systems allow natural language queries that automatically generate, execute, and visualize results. Two notable open-source projects in this space—LAMBDA and Vanna—take different approaches to this challenge.

This post explores both architectures in depth. We'll understand how LAMBDA uses a multi-agent system with programmer and inspector roles for general data analysis, and how Vanna provides a streamlined Text-to-SQL pipeline with enterprise security features. Both demonstrate patterns applicable to building production data analysis agents.

LAMBDA: Multi-Agent Data Analysis

LAMBDA (Large Model-based Data Analysis) is an open-source system designed for code-free data analysis through natural language. Its multi-agent architecture separates code generation from code verification, improving reliability for complex analysis tasks.

Multi-Agent Architecture

LAMBDA employs two primary agent roles. The Programmer agent receives user requests and generates Python code to accomplish the analysis. It has access to the conversation history, data context, and any domain knowledge that's been integrated.

The Inspector agent reviews the generated code before execution. It checks for errors, verifies that the code addresses the user's request, and suggests corrections when needed. This separation of concerns—generation and verification—improves code quality compared to single-agent approaches.

The agents operate in an iterative loop. The Programmer generates code, the Inspector reviews it, and if issues are found, the Programmer revises based on feedback. This continues until the code passes inspection or the maximum attempt limit is reached.

Conversation Management

The conversation.py module manages the dialogue between users and the system. Each conversation maintains state including message history, code blocks that have been generated and executed, and results from previous queries.

Conversation context is crucial for multi-turn analysis. If a user asks "now filter that to only show sales above $1000," the system needs to understand what "that" refers to from previous turns. The conversation manager maintains this context and passes relevant history to the agents.

Jupyter Kernel Integration

LAMBDA executes code through a Jupyter kernel, providing a familiar Python environment with access to common data science libraries. The kernel.py module handles communication with the IPython kernel, executing code blocks and capturing outputs.

The kernel integration enables rich output handling. Code can generate plots, dataframes, statistical summaries, and other artifacts. These outputs are captured and displayed in the UI, providing the same interactive experience as working in a Jupyter notebook.

Error handling is important for code execution. When code fails, the error message is captured and fed back to the Programmer agent for correction. The system attempts self-correction up to a configurable limit before asking the user for help.

Knowledge Integration

The knowledge_integration directory implements mechanisms for incorporating domain knowledge into the analysis. Users can provide documentation, schemas, or examples that inform how the agents understand and analyze data.

This knowledge base is searchable, allowing agents to retrieve relevant context when processing queries. For specialized domains with specific terminology or analysis patterns, this integration significantly improves accuracy.

User Interface

LAMBDA provides a Gradio-based web interface that supports real-time interaction. Users can see streaming output as code executes, intervene when needed, and export results to Jupyter notebooks for further work.

The interface displays the conversation, generated code, execution outputs, and any visualizations. Users can modify code directly if the generated version needs adjustment, maintaining human oversight while reducing manual coding effort.

Configuration

Configuration through YAML files specifies model settings, API endpoints, and system parameters. Different models can be assigned to different roles—a powerful model for the Programmer and a faster model for the Inspector, for example.

The configuration supports multiple LLM providers including OpenAI, local models through Ollama, and any OpenAI-compatible endpoint. This flexibility allows deployment in various environments including on-premise installations.

Code Execution Details

The CodeKernel class provides sophisticated interaction with the Jupyter kernel. It manages the kernel lifecycle, handles message passing through ZeroMQ channels, and processes various output types including stdout, execute_result, display_data, and error messages.

Output handling is particularly rich. The kernel detects different content types in responses—text/plain, text/html, image/png, image/jpeg—and routes them appropriately. Images are saved to the session cache path and tracked for inclusion in reports and exported notebooks.

The execution timeout is configurable, defaulting to 18000 seconds (5 hours) for long-running analysis tasks. An interrupt signal mechanism allows users to stop runaway computations without killing the kernel entirely.

Notebook Export

LAMBDA can export the entire analysis session to a Jupyter notebook. The CodeKernel tracks executed cells and uses nbformat to construct a valid notebook file. This enables reproducibility—users can re-run the analysis or continue work in a traditional notebook environment.

The export preserves cell ordering and outputs, creating a complete record of the analysis session. Combined with the session cache that stores intermediate files and figures, this provides a full artifact trail.

Report Generation

Beyond notebooks, LAMBDA generates formatted reports using the conversation model. The report generation takes the chat history, processes it with a report-specific system prompt, and produces a markdown document that summarizes findings.

Reports include embedded figures (tracked in the figure_list) and can be customized through the prompt system. This automates the often tedious task of documenting analysis work for stakeholders.

Session Persistence

Each analysis session maintains state that persists across interactions. The session cache directory stores configuration, conversation history for both agents, intermediate files, figures, and the notebook. This enables resuming sessions and provides audit trails for analysis work.

The save_conv method serializes programmer and inspector message histories to JSON, along with configuration state. This enables debugging agent behavior and understanding how the analysis evolved.

Vanna: Enterprise Text-to-SQL

Vanna takes a different approach, focusing specifically on Text-to-SQL with enterprise security features. Version 2.0 introduces user-aware queries and production-ready deployment patterns.

Text-to-SQL Pipeline

The core Vanna pipeline converts natural language questions to SQL queries. When a user asks "What were Q4 sales by region?", Vanna generates the corresponding SQL, executes it against the connected database, and returns results as tables and visualizations.

The pipeline includes training on database schemas and example queries. Vanna learns the structure of your data—table names, column types, relationships—and uses this knowledge to generate accurate SQL. Example question-SQL pairs improve accuracy for domain-specific terminology.

User-Aware Security

Vanna 2.0's key innovation is user-awareness throughout the pipeline. User identity flows from authentication through query generation to result filtering.

The User Resolver extracts identity from incoming requests—parsing JWTs, reading session cookies, or integrating with OAuth providers. This user context is then available to all downstream components.

Row-level security filters query results based on user permissions. If a user only has access to data from certain regions or time periods, the generated SQL automatically includes appropriate WHERE clauses. This prevents data leakage without requiring manual query modification.

Agent Architecture

Vanna structures its logic as an Agent with Tools. The Agent orchestrates the conversation, deciding which tools to invoke based on the user's question. Tools include SQL generation, query execution, chart creation, and summary writing.

Tools are user-aware by design. When executing SQL, the tool has access to the user context and can apply appropriate security filters. Audit logging captures which user ran which query, supporting compliance requirements.

Streaming Components

Rather than returning plain text, Vanna streams structured UI components. As processing progresses, the frontend receives progress updates, then a SQL code block, then a data table, then charts, and finally a natural language summary.

This streaming architecture provides a responsive user experience. Users see partial results immediately rather than waiting for complete processing. The pre-built <vanna-chat> web component renders these streaming components beautifully.

Web Component

The frontend is delivered as a web component that can be embedded in any webpage. The <vanna-chat> element handles rendering, user input, and communication with the backend.

The component works with existing authentication. It forwards cookies or JWT tokens with requests, and the backend User Resolver extracts identity. No separate auth flow is needed—Vanna integrates with your existing system.

Theming and customization allow the component to match your application's look. Dark mode, custom colors, and responsive layouts are supported out of the box.

LLM Integration

Vanna supports multiple LLM providers through a modular design. OpenAI, Anthropic Claude, Google Gemini, Azure OpenAI, AWS Bedrock, Ollama, and others are supported. The same Vanna configuration works across providers with minimal changes.

LLM middleware allows caching, prompt engineering, and observability. You can cache common queries to reduce costs, modify prompts based on user context, and trace all LLM calls for debugging.

Database Connectors

Vanna connects to any database with a Python connector. PostgreSQL, MySQL, Snowflake, BigQuery, Redshift, SQLite, Oracle, SQL Server, DuckDB, ClickHouse, and others are supported. Connection details are configured once and used for all queries.

Schema extraction automatically learns table structures. Vanna queries information schemas to understand available tables, columns, and types. This metadata powers SQL generation without manual configuration.

Lifecycle Hooks

Production deployments need control points for quota checking, logging, and content filtering. Vanna provides lifecycle hooks that run at key points in request processing.

Pre-query hooks can check rate limits, validate user permissions, or log audit information. Post-query hooks can filter sensitive results, track usage metrics, or trigger downstream workflows.

FastAPI Integration

Vanna provides ready-made FastAPI routes for production deployment. The register_chat_routes function adds endpoints to your existing FastAPI application, including the SSE streaming endpoint for the chat component.

Integration requires minimal code—instantiate a Vanna agent, configure your User Resolver, and register the routes. Authentication uses your existing middleware.

Training and Schema Learning

Effective Text-to-SQL requires understanding your specific database. Vanna's training process involves schema extraction (table structures, column types, relationships), example query pairs (natural language questions mapped to correct SQL), and documentation (business context, naming conventions, special cases).

The training data is stored in a vector database for retrieval during query generation. When a user asks a question, Vanna retrieves relevant schema information and similar example queries to inform SQL generation.

Training can be incremental—as you add more examples, accuracy improves for similar queries. The system learns your organization's terminology and common query patterns over time.

SQL Validation and Execution

Generated SQL goes through validation before execution. Syntax checking catches obvious errors before hitting the database. Permission checks ensure the generated query doesn't access restricted tables or columns for the current user.

Execution uses parameterized queries to prevent SQL injection. Results are streamed back with appropriate pagination for large result sets. Error messages are processed to provide actionable feedback rather than raw database errors.

Chart Generation

Beyond tables, Vanna generates visualizations based on query results. The chart tool analyzes the data structure and selects appropriate visualization types—bar charts for categorical data, line charts for time series, scatter plots for correlations.

Visualizations use Plotly for interactive charts that users can zoom, pan, and export. The chart configuration is generated by the LLM based on the data and user's original question, ensuring relevance.

Natural Language Summary

Each query response includes a natural language summary explaining the results. This makes data accessible to non-technical users who may not understand SQL or complex tables.

The summary highlights key findings, notes any unusual patterns, and provides context for the numbers. This bridges the gap between raw data and actionable insights.

Comparing Approaches

LAMBDA and Vanna address different use cases within data analysis.

LAMBDA excels at general data analysis where the task might require complex Python code, statistical analysis, machine learning, or custom visualizations. Its multi-agent architecture handles the full spectrum of data science tasks.

Vanna focuses specifically on SQL and structured data queries. Its Text-to-SQL specialization produces highly accurate SQL for database analytics, and its enterprise features support production deployment with security and compliance requirements.

For organizations with primarily SQL-based analytics needs and enterprise security requirements, Vanna provides a streamlined solution. For research environments or teams needing general data science capabilities through natural language, LAMBDA offers broader flexibility.

Architectural Patterns

Both systems demonstrate valuable patterns for agentic data analysis.

Separation of concerns improves reliability. LAMBDA's Programmer/Inspector split and Vanna's Agent/Tools architecture both separate distinct responsibilities, making each component simpler and more reliable.

User context propagation is essential for enterprise deployment. Vanna's user-aware design shows how to maintain identity through the entire pipeline, enabling security and audit features.

Iterative refinement handles errors gracefully. Both systems support self-correction when initial attempts fail, improving success rates for complex queries.

Streaming provides better user experience. Real-time feedback keeps users informed and enables early intervention if something goes wrong.