A large part of day-to-day incident response includes having conversations about an incident while investigating, ensuring the right people are involved in these conversations, and logging the output where it can be stored and reviewed. Clear communication is even more critical when working and managing a high-priority incident.
More and more of these conversations are now taking place in Slack. Everyone in the company likely has access, so it’s quick and easy to bring in the people and resources when needed. It’s also straightforward to spin up a new incident-specific channel to keep all discussion contained. In addition to helping communication, this channel can act as an investigation timeline, a record of what happened when. That information is invaluable for putting together a post-incident review and root cause analysis.
So how can Tines help with this? Slack provides a rich API that allows Bots to create channels, add users, back up conversations, and more! Tines can tie all of these API endpoints together to create an easy-to-use incident management process that neatly ties in with the rest of your incident response stack.
Where to start?
Slack slash commands are ideal for letting your team initiate powerful workflows on-demand. This whole process needs to be and is completely flexible and customizable. So this slash command can be anything that makes sense to you. For this example, the command is`/irbot`.
For more information on setting up and configuring a Slack Bot, take a look at our previous blog “Chatbots for Security and IT Teams.”
Usage of this bot will be through command arguments passed through with the Slash Command. Let’s start with three basic commands:
1. /irbot create
This command will kick off the initial steps of the process, such as gathering requirements from the requester and creating Slack channels.
2. /irbot backup
The backup command will initiate a text backup of the conversation, which can be preserved for logging purposes.
3. /irbot archive
Once the incident has been resolved, the archive command will take a final conversation backup and archive the channel in Slack.
Each of these commands is entirely customizable for whatever process will suit your team, and the Tines Story will only require minor modification to support any changes.
Let’s look at each of these commands in a little more detail.
Create
An incident has happened, you need to start involving the right people from the right teams to start working through it. `/irbot create` will be the beginning of this process, it’s an indication to Tines that you want to create a Slack Channel for this incident. Tines will receive this command and create a Slack modal to collect some extra information. At a minimum, these will be the users that need to be added to the Slack Channel, but including the Incident Case ID or creating a new Case would also be a good move.
This modal also includes an option to create a second channel for executive communication, which would likely be more high-level than the general incident discussion channel. This option can be ignored or removed if not necessary.
The Slack API includes a neat `multi_users_select` component that allows for selecting multiple users by searching for their name, the same way you would when tagging a user.
Once submitted, Tines will parse this information and carry out several steps. The first of these steps is to check if ‘Create New Incident’ was selected, and if so, it will create a new Jira case.
Next up is to create a new Slack Channel. The name of this channel will be something like ‘incident_channel_<incident_ID>’ to keep it unique and orderly, but it’s a quick change to add an extra field to the above modal to enter a custom name if that suits better!
Once created, Tines will update the channel description to include a link to the Incident Case and then add the specified users into that channel. Finally, a comment should be added to the Incident Case to log the channel name and creation time.
Tracking State
If you import the Tines Story attached to this blog, you’ll see an Action called ‘Create Incident Tines Resource’. It’s important to keep track of a few essential things that you’ll need to reference later. Things like Incident ID, Slack Channel ID, the last time the channel was backed up, and who created the channel may all be useful down the line. This information will be stored in a Tines Resource that’s created when the ‘Create’ command is run and will be deleted later on during archiving.
Backup
Being able to recall information during a post-incident review or audit can be crucial, especially when dealing with high severity incidents. Preserving some or all of the conversation history can also massively help when it comes to constructing an incident timeline.
To help with this, there is ‘/irbot backup’. There are no doubt countless possible implementations of this. We could back up the whole conversation, only backup top-level messages, only backup messages flagged with a specific emoji, or only backup manually chosen messages.
For this purpose, it will be kept simple, and the top-level messages of the whole conversation will be backed up.
Upon initiating the backup command, a message will be posted in the channel by the Slack Bot confirming that a backup will be performed, and the name of the user that requested it.
Tines will then run a query against the Slack API to get all messages posted since the last backup. This returns the User ID that posted the message rather than the name, so Tines will do some additional work to map those IDs to actual names.
By default, the Story will append this conversation to a .csv file attached to the Jira Ticket. But sending it to another case management system, Confluence, Google Drive, or anywhere else with an API is also possible.
Archive
After the incident has been wrapped up, there’s no need to keep the channel active anymore. To prevent accidental archiving, Tines will send a message to the channel stating that archiving has been requested and will occur in one hour unless canceled. Canceling this process uses Tines prompts to catch the cancelation and prevent the channel from being archived.
After one hour has elapsed, Tines will take one final backup of the channel before interacting with the Slack API to archive the channel. Once archived, the channel will be hidden from the participants, and Tines will complete some final clean-up and delete the tracking resources that were created at the start.
Conclusion
Extending this Story to include additional commands and enhancements is a quick process, with no modifications needed on the Slack side. Entering ‘/irbot page engineering’ will immediately go right into the Story, where a Trigger Action can be added to catch this request, and a few additional Actions would be used to hook into the PagerDuty API and alert the engineering team during the incident. Other enhancements could be to track and manage who is the current Incident Commander, collect Shift Handover details, or even take action to Contain a Device in CrowdStrike.
It’s time to offload this manual admin to an automated system. In the heat of an incident, anything that can be done to ease the workload of analysts and investigators will be worth it. Taking a little time to implement and customize this Story means Tines will be there to help when needed.
You can download the complete Story for all the above Slack Actions here.