Incident Response Automation Using AI Workflows: From Detection to Resolution

Incident Response Automation Using AI Workflows: From Detection to Resolution

Incident response is evolving rapidly. With the rise of AI workflow automation, organizations can now detect, triage, and resolve incidents faster and more reliably than ever before. As we covered in our complete guide to AI workflow automation for IT operations in 2026, this area deserves a deeper look—especially for teams aiming to build robust, end-to-end automated incident response pipelines.

In this Builder’s Corner deep-dive, you’ll learn to construct a practical incident response automation using AI workflows—starting from real-time detection, through triage, to automated resolution. We'll use open-source tools, cloud services, and code samples you can adapt for your environment. This guide is hands-on, reproducible, and packed with troubleshooting tips for common issues.

For security and compliance considerations, see our sibling article Securing Automated IT Ops Workflows: New Standards and Best Practices for 2026. If you’re interested in AI workflow automation in other domains, check out our related deep dives on AI-driven returns management for e-commerce and supply chain risk management.

Prerequisites

• General Knowledge: Familiarity with incident response concepts, basic Python, and REST APIs.
• Cloud Account: AWS or Azure (free tier is sufficient for this tutorial).
• Local Machine:
  • Python 3.10+
  • Docker (v24+)
  • Git (v2.30+)
• Tools Used:
  • ELK Stack (Elasticsearch 8.x, Logstash, Kibana) for log aggregation
  • OpenAI GPT-4 API (or Azure OpenAI) for AI-driven triage
  • StackStorm (v3.8+) for workflow automation
  • Slack for incident notifications
• API Keys: OpenAI API key, Slack Bot token

1. Set Up Log Aggregation and Detection with the ELK Stack

1. Clone the ELK Docker Compose Setup

   git clone https://github.com/deviantony/docker-elk.git
   cd docker-elk

2. Start the ELK Stack

   docker compose up -d

   Screenshot description: Docker containers for Elasticsearch, Logstash, and Kibana running successfully in Docker Desktop.

3. Ingest Sample Logs

   Create a file named sample.log with simulated incidents:

   
   2024-05-15T15:23:01Z ERROR AuthService - Failed login for user admin from 10.1.2.3
   2024-05-15T15:23:05Z WARNING DBService - Query timeout on db-prod
   2024-05-15T15:24:10Z ERROR WebServer - 503 Service Unavailable
         

   Configure Logstash to ingest this file by editing logstash/pipeline/logstash.conf:

   
   input {
     file {
       path => "/usr/share/logstash/sample.log"
       start_position => "beginning"
     }
   }
   output {
     elasticsearch {
       hosts => "elasticsearch:9200"
       index => "incident-logs"
     }
     stdout { codec => rubydebug }
   }
         

   Copy your sample.log into the Logstash container:

   docker cp sample.log docker-elk_logstash_1:/usr/share/logstash/sample.log

4. Verify Ingestion in Kibana

   Navigate to http://localhost:5601, create an index pattern for incident-logs, and confirm your logs appear.

   Screenshot description: Kibana Discover page showing parsed incident logs.

5. Create a Detection Rule

   In Kibana, go to "Stack Management" → "Rules and Connectors" → "Create Rule". Use a KQL query like:

   message : "*ERROR*"

   Set the action to send a webhook when triggered (we'll connect this to our automation in Step 3).

2. Build the AI-Driven Triage Function

1. Set Up a Python Environment

   python3 -m venv .venv
   source .venv/bin/activate
   pip install openai requests
         

2. Create the Triage Script

   Save as ai_triage.py:

   
   import os
   import openai
   
   openai.api_key = os.environ["OPENAI_API_KEY"]
   
   def triage_incident(log_message):
       prompt = f"""
       You are an incident response triage assistant.
       Analyze the following log entry and classify its severity (Critical/High/Medium/Low), suggest immediate actions, and possible root cause:
       Log: {log_message}
       Respond in JSON: {{"severity": "...", "action": "...", "root_cause": "..."}}
       """
       response = openai.ChatCompletion.create(
           model="gpt-4",
           messages=[{"role": "user", "content": prompt}],
           max_tokens=200
       )
       return response.choices[0].message['content']
   
   if __name__ == "__main__":
       import sys
       log_message = sys.argv[1]
       print(triage_incident(log_message))
         

   Test it with:

   export OPENAI_API_KEY=sk-...
   python ai_triage.py "2024-05-15T15:24:10Z ERROR WebServer - 503 Service Unavailable"
         

   Screenshot description: Terminal output showing a JSON response with severity, action, and root cause.

3. Orchestrate Incident Workflow with StackStorm

1. Install StackStorm (via Docker)

   git clone https://github.com/StackStorm/st2-docker.git
   cd st2-docker
   docker compose up -d
         

   Access StackStorm Web UI at http://localhost:8080 (default user: st2admin / password: Ch@ngeMe).

2. Register a Webhook Sensor

   Create incident_sensor.yaml in packs/incident/sensors/:

   
   class_name: "core.WebhookSensor"
   trigger_types:
     - name: "incident_detected"
       description: "Incident detected from ELK webhook"
       payload_schema:
         type: "object"
         properties:
           message:
             type: "string"
         required: ["message"]
         

   Register the sensor:

   st2ctl reload --register-sensors

3. Create the Incident Response Workflow

   Define incident_response.yaml in packs/incident/actions/:

   
   version: '1.0'
   description: Incident response workflow
   input:
     - message
   tasks:
     triage:
       action: core.local
       input:
         cmd: "python3 /opt/stackstorm/packs/incident/actions/ai_triage.py '{{ message }}'"
       next:
         - when: <% succeeded() %>
           do: notify
     notify:
       action: slack.post_message
       input:
         channel: "#incidents"
         message: |
           Incident triaged:
           Severity: <% result().triage.stdout.severity %>
           Action: <% result().triage.stdout.action %>
           Root Cause: <% result().triage.stdout.root_cause %>
         

   Note: This uses the slack integration. Configure your Slack bot token in StackStorm's secrets.

4. Test the End-to-End Workflow

   Trigger the workflow manually:

   st2 run incident.incident_response message="2024-05-15T15:24:10Z ERROR WebServer - 503 Service Unavailable"
         

   Screenshot description: StackStorm execution history showing the workflow steps and Slack notification delivered.

5. Connect ELK Webhook to StackStorm

   In Kibana, set your detection rule’s webhook action to point to StackStorm’s webhook endpoint (e.g., http://your-stackstorm-host/api/v1/webhooks/incident_detected).

   Now, whenever a matching log is ingested, your workflow will trigger automatically.

4. Automate Resolution for Common Incidents

1. Add Automated Remediation Actions

   Extend your workflow to include conditional steps. For example, if the AI triage returns 503 Service Unavailable, restart the affected service:

   
   remediate:
     action: core.remote
     input:
       hosts: "webserver-prod"
       cmd: "sudo systemctl restart nginx"
     when: <% result().triage.stdout.severity = "Critical" and "503" in result().triage.stdout.root_cause %>
         

2. Test Automated Remediation

   Simulate a 503 incident and verify the service is restarted automatically. Check your logs and StackStorm execution history for confirmation.

   Screenshot description: StackStorm workflow run showing remediation step executed, and Slack update confirming action taken.

Common Issues & Troubleshooting

• ELK Stack Not Starting: Check Docker logs with

  docker compose logs

  . Ensure enough system memory and that ports 9200, 5601, and 5044 are free.
• Logstash Not Ingesting Logs: Confirm file paths match inside the container. Use

  docker exec -it docker-elk_logstash_1 ls /usr/share/logstash/

  to verify.
• OpenAI API Errors: Ensure your API key is valid and you have sufficient quota. Check for rate limiting or network issues.
• StackStorm Workflow Fails: Use the StackStorm Web UI to inspect execution details and logs. Validate all action paths and environment variables.
• Slack Notifications Not Delivered: Double-check your Slack bot token and channel permissions. Try sending a test message with StackStorm’s slack.post_message action.
• Webhooks Not Triggering: Confirm Kibana detection rule action URLs and StackStorm webhook endpoint accessibility (firewall, network).

Next Steps

• Expand your AI triage prompts to include more context (e.g., user history, asset inventory).
• Integrate additional remediation playbooks for a wider range of incidents.
• Explore no-code AI workflow builders for rapid prototyping, as compared in our review of top no-code AI workflow builders.
• For advanced security and compliance, see Securing Automated IT Ops Workflows: New Standards and Best Practices for 2026.
• For broader strategies, revisit The Complete Guide to AI Workflow Automation for IT Operations in 2026.

With this builder’s playbook, you can prototype and scale incident response automation using AI workflows—reducing response times, improving consistency, and freeing your teams to focus on proactive work. For more workflow automation deep-dives, explore our articles on AI-driven e-commerce returns and supply chain risk management.