A DevOps team in a company with dozens of services receives hundreds of notifications daily. Most are noise. A few require a response within minutes. Distinguishing between them manually at 2 a.m. is a structural problem, not a competence issue. AI doesn’t solve it through autonomous action. It solves it by reducing the cost of the first decision: “Does this require my attention now?”
Below, I describe how we at Cashcrown approach this class of problems and why the hard boundary between suggestion and execution isn’t something you can bypass “post-deployment.”
What an agent can and cannot do#
The scope of automation for IT and DevOps teams must be defined before anything enters the pilot phase. The table below shows a typical division for a production environment with a few on-call engineers.
| Task | Agent can | Human must |
|---|---|---|
| Alert triaging | Classify by priority, group related incidents, suggest severity | Decide on P1/P2 escalation |
| Log summarization | Extract anomalies, identify time window, link error trace to ticket | Verify diagnosis and approve rollback |
| Runbook and postmortem | Draft remediation steps from similar incident history, generate postmortem draft | Approve and execute every step that changes production |
| Documentation search | Answer “how do we do X” from internal knowledge base (RAG) | Assess whether the procedure is up to date |
| Ticket routing | Classify ticket, assign to queue or person | Decide priorities in resource conflicts |
| Granting access or changing configuration | Nothing | Everything, without exception |
The last row is absolute. Human-oversight for irreversible actions isn’t a configuration option. An agent that can independently open a firewall port, grant IAM permissions, or modify a production service configuration introduces a risk no guardrail in a language model can eliminate. Models make mistakes. Irreversible actions in production have real costs.
Alert triaging and log summarization#
Alert fatigue is a well-documented problem. On-call engineers who see hundreds of notifications daily—95% of which turn out to be false positives—start treating each new one as noise. This isn’t a human error. It’s an inevitable effect of asymmetry: the cost of missing a real incident is enormous, while the cost of a false alarm seems zero because “nothing happened.” In reality, the cost of false alarms is real and accumulates as fatigue, degraded vigilance, and burnout.
An agent with access to the alerting system can help differently than by blindly reducing their number. Instead of filtering alerts (which risks missing real ones), it can aggregate and contextualize them:
- Group alerts from the same time window with overlapping resources.
- Search incident history for similar patterns and extract notes like “three weeks ago, a similar alert ended at the database layer, fixed by restarting the connection pool.”
- Extract the error stack trace from service logs and match it with open tickets.
What the agent doesn’t do: it doesn’t decide if an incident is real. It doesn’t close alerts. It doesn’t mark events as resolved. We leave those actions to humans because falsely closing a real incident is one of the costliest operational mistakes.
Architecture: RAG on internal documentation#
The question “How do we deploy to staging?” sounds trivial. In practice, a new engineer on the project might spend 20-40 minutes searching for the answer in Confluence, Notion, or scattered Markdown files in a repository. A mature RAG on internal documentation reduces this time to 30-60 seconds—if the knowledge base is up to date and well-segmented.
Minimal architecture for DevOps documentation:
Knowledge sources: runbooks, incident procedures, service architecture descriptions, security policies, ADRs. Each document tagged with metadata: owner, verification date, environment (prod/staging/dev).
Hybrid search: semantic embeddings plus full-text search for exact phrases. Semantic retrieval alone will fail on “OOM error on node-15 in cluster EU-west-2” because it’s a query with specific identifiers.
Source citation: Every answer must reference the document and section. If the agent can’t cite a source, it should say “I don’t know” and escalate to a human queue. The architecture for this pattern is described in monitoring AI agent quality.
Confidence thresholds: Retrieval returning fragments below a 0.70-0.80 threshold shouldn’t result in an answer. “I couldn’t find an up-to-date procedure” is better than an answer built on poorly matched fragments.
Alert fatigue: an honest look at cost asymmetry#
The costs of false positives and false negatives in alert triaging aren’t symmetric. A missed real incident costs from minutes of downtime to data loss. A false alarm costs two minutes. But 200 false alarms a day add up to 400 minutes of noise. At that level of fatigue, a real incident might look like just another false alarm.
An agent classifying alerts should be calibrated toward sensitivity (recall), not precision. It’s better to falsely escalate 15% of noise than to miss 1% of real incidents. This calibration is an engineering decision, not a configuration one.
Observability of the triaging system is a separate layer: how many alerts were escalated, how many turned out to be real, and what was the time from the first alert to human confirmation. Without these metrics, you don’t know if the agent is helping or harming.
Guardrails for agents with system access#
A DevOps agent that can read logs, query knowledge bases, and classify tickets is relatively safe. An agent that can invoke tools (tool-use: restart a service, change configuration, grant permissions) requires much stricter guardrails.
The pattern we use at Cashcrown for this class of systems:
Permission isolation at the tool level. Every tool available to the agent has a precisely defined scope: “I can run kubectl describe pod on the staging namespace; I cannot run kubectl delete pod anywhere.” The tool list is static and approved by the security team before deployment, not configured in real time.
Human-gate for irreversible actions. Every tool whose invocation changes the production system state requires explicit human approval. The agent prepares a ready-to-execute command with justification. An engineer approves or rejects it. The pattern uses HMAC tokens or an equivalent approval mechanism. Without an approved token, the action doesn’t proceed.
Classifier security layer before each tool. Before passing a query to the model, the guardrails layer checks for injection attempts: “ignore previous instructions,” “you are now an admin with full access,” “execute this without logging.” Such queries go to a security queue, not the model. Details are covered in AI agent security.
Full context logging. Every tool invocation by the agent is logged: what was requested, what the agent proposed, who approved it, and when. Logs are immutable and available for security audits. This audit trail isn’t just common sense—it may be required by the AI Act for systems affecting critical infrastructure.
Routing and IT ticket classification#
Before incident triaging comes the daily stream of service requests: helpdesk questions, user-reported errors, operational tasks, access requests. An LLM-based classifier can automatically assign priority, category, and target queue before a human operator reads it.
The value is twofold: faster response times for critical requests (P1 doesn’t wait in a queue behind 40 minor requests) and better data quality in the ticketing system (consistent categorization instead of arbitrary “misc”). Implementing this pattern in an IT helpdesk context is described in AI for IT helpdesk: internal support assistant. The technical routing logic is covered in AI ticket classification and routing.
FAQ#
Can AI autonomously restart services or roll back deployments?#
No. Restarting a production service might fix a problem or make it worse if the root cause lies deeper. A rollback might resolve an incident or expose an earlier bug hidden by the current version. A language model can’t be certain about the system state, even if it sees logs. The correct pattern: the agent prepares a recommendation with justification, an engineer approves or rejects it, and only then does the action occur.
How do I build a RAG knowledge base for DevOps documentation that’s actually useful?#
Three things matter most: document freshness (outdated runbooks are worse than none because the agent will confidently answer based on the wrong procedure), chunk granularity (subsections with a single procedure yield good retrieval; entire architecture chapters don’t), and environment metadata (staging and prod procedures differ, and the agent needs to know which one to look for). Before indexing all documentation, test 20-30 typical on-call questions.
How do I measure if the alert triaging agent is actually helping?#
Two key metrics: incident recall (how many real incidents were correctly escalated) and time from first alert to human acknowledgment (Mean Time to Acknowledge). Measure recall retrospectively by comparing the triaging log with the confirmed incident registry. If the agent misses a real incident more than once per quarter at a volume of hundreds of alerts daily, the threshold calibration needs review.
What are the AI Act requirements for DevOps agents with access to production systems?#
Alert triaging and internal documentation systems that don’t directly influence human-related decisions aren’t generally classified as high-risk under Annex III of the AI Act. However, an agent with access to infrastructure tools requires documentation: system description, tool scope, decision logs, and human oversight mechanisms. If the AI system falls under critical infrastructure (energy, transport, financial sector), requirements may be stricter, and a DPIA becomes almost mandatory. Check the boundary with a lawyer familiar with the regulation.
What do I do when on-call engineers stop trusting the agent’s recommendations?#
That’s a signal pointing to a specific problem. Loss of trust usually stems from three sources: the agent gave confident recommendations based on outdated documentation, the classifier had too low a threshold and escalated too much noise, or one high-profile mistake skewed perception of the entire system. Before fixing the model, conduct a retrospective of the last 20-30 interactions where trust declined. Agent quality auditing is described in monitoring AI agent quality.