IT Request Advisory Agent

The problem

Internal database support tickets at scale hit two friction points: resolvers burned first-response time triaging tickets that belonged to other teams, and every SQL statement they wrote against production carried real risk. The team wanted an agent that could triage tickets and draft first-pass SQL, but under strict constraints. LLM-generated SQL cannot be trusted at face value against a production database, so the design had to make LLM output verifiable, not just plausible.

Architecture

A 7-stage pipeline in Python (ingest, normalize, classify, resolve, generate, compose, notify) that reads tickets from Azure DevOps Boards, drafts SQL proposals, and posts advisory comments back for human review. Classification runs in three stages (relevance, information sufficiency, operation and risk tier), and every generated SQL statement passes through a four-tier risk ladder before being posted, with destructive statements wrapped in rollback transactions and DDL blocked from execution entirely. Azure OpenAI serves as the LLM provider in production.

What I owned

The full pipeline design and implementation. The three-stage classification logic and the four-tier risk ladder. The deterministic post-processing that validates SQL against schema metadata, injects audit fields, and wraps destructive operations in rollback transactions. A state-based deduplication layer to avoid burning LLM calls on unchanged tickets. A GitHub Actions workflow for automated SLA metrics reporting. An offline evaluation harness and a 278-pair ground-truth dataset curated via the Azure DevOps REST API to benchmark the agent quantitatively.

Hardest technical decision

Making the agent advisory by design, not by policy. The tempting architecture was to let the agent execute low-risk queries directly and only ask for human review on high-risk ones. I rejected this because "low-risk" is a runtime judgment the LLM cannot be trusted to make reliably. Instead, the agent never connects to any database. It only posts SQL text as a comment on the ticket, and a human resolver runs it manually if they approve. This means the LLM can be wrong without being dangerous. Evaluation confirmed the design: zero false negatives on destructive SQL leakage.

What I'd do differently

Add proper observability to the pipeline from day one. The current version tracks whether classifications were correct and whether the deduplication layer is skipping tickets efficiently, but it doesn't log how long each pipeline stage takes or how many LLM tokens each stage burns. Without that data, it's hard to spot which stage is slowing the agent down or costing the most, and adding this later across seven stages is much more painful than instrumenting each stage when it's first written.