In this guest post, Eric Newcomer, Principal Analyst at Intellyx explores the practical applications and limitations of generative AI.
Generative AI is a game-changing technology. Chat bots seem like magic compared to a traditional static web search. You submit questions in natural human language and receive back complete sentences and paragraphs.
But it isn’t always clear what it is really good for, given limitations such as hallucinations and inaccurate answers, and possible bias. It can be challenging to get through the hype and figure out the best applications for gen AI.
It’s becoming clear that a human has to carefully check the results produced by an AI conversation to confirm whether or not it’s working correctly for a given task.
Consider a set of steps in a human workflow, such as taking and fulfilling a restaurant delivery order, or investigating and remediating an IT incident. Multiple decisions are involved in executing a workflow, such as identifying the correct steps, mapping actions correctly to the steps, aligning to operational business policies, and confirming successful completion. A subsequent step depends on the successful completion of the prior step, and so on. Someone typically confirms each step's results before moving to the next step.
Adding an AI conversation to a human workflow seems natural because a human is already checking and confirming the results, and can also confirm the AI responses and proposed actions. Such an application of AI can significantly improve business process automation.
Gen AI applications
One thing everyone seems to agree on is that gen AI is good at creating summaries, which it can do with a prompt or just via processing a meeting transcription. "Co-pilots” for Zoom, Teams, and Google Meet for example all are capable of translating speech to text and automatically producing such summaries.
These summaries do not have to be 100% accurate to be very helpful. Minor errors and mistakes are easily corrected. And a human can easily check the results and fix any errors.
Another thing gen AI is good at is iterating through a known body of text, which can be specific to an organization or on a general topic, or both, and answering questions based on the input text.
Challenges with Gen AI
But we know gen AI replies are not precise in responding to these questions because they are interpreting human language, which is itself not precise. Many words have two or more meanings, and others are interpreted differently by different people in different professions, or when used in different contexts. This is why a human has to review and confirm gen AI output.
Training large language models (LLMs) typically includes at least two levels of transformation to create vector-based statistics for interpreting human language. The transformation logic does its best to identify language ambiguities and figure out from the context the right meaning.
But again this is a statistical match process producing the best guess, and is not an exact match process when understanding the meaning of a sentence. It can be incredibly helpful, but it’s not possible to completely automate the process.
Of course prompt engineering typically improves the quality of a response by improving the quality of the question. And another way to improve the quality of the results is to improve the quality of the training data.
In any case, humans typically have to review and validate gen AI results for correctness and quality – i.e. that the answer hits the mark or correctly solves the problem given to it.
Tines Workbench
Tines is a flexible, intuitive platform to quickly orchestrate and automate Security Operations Center (SOC) and IT workflows to improve productivity and respond more quickly to incidents and outages.
Tines users create workflows using a drag-and-drop GUI. The user drags and drops actions to build the workflows. Actions communicate using events.
Tines recently added a gen AI product called Workbench, which provides a secure and private interface for security analysts to leverage workflows (and its data sources) to triage incidents, determine remediation actions, and confirm proposed actions.
Users can set up a new workflow specifically to take advantage of the Workbench chat interface, or you can connect Workbench with existing workflows. Then you configure the data sources for the Workbench to use, such as logs, event stores, an identity access management registry, asset management directory, and so on.
When a workflow executes, a security analyst works with the gen AI interface to investigate an incident or to initiate an incident resolution. The gen AI interface also proposes actions for the user to approve, and gathers additional context from your existing tech stack to take such action
Examples of how to use Workbench include resetting a password, reporting and locking lost devices, and analyzing and triaging suspicious emails.
One of Tines’ goals is to reduce “alert fatigue” that can easily occur when security analysts review and respond to high volumes of incident data manually. Workbench is the next logical step toward achieving this, using the power of AI “summaries” to further reduce such manual effort.
Because security is at the core of the Tines platform, they have implemented their AI capabilities with the security and privacy controls you’d expect. Using stories requires permissions defined by the team you are in. Role based access controls and all data is contained within the private instance of Tines that you are running – and is not trained on, logged, or shared with any LLM or external system.
The Intellyx take
In figuring out what a chatbot is good for, it seems like summarizing data and text to take the grudge work out of manually reading and writing a large amount of text or unstructured data is high on the list.
When you marry this capability with a workflow orchestration and automation tool such as Tines, you can easily understand the benefits of gen AI, especially when the workflows involved handle overwhelming amounts of log and event data created for security or IT incidents.
It’s hard enough to manually separate the wheat from the chaff in logs and observability data, but when you are trying to automate and streamline the workflows to improve results and reduce analyst burn out, it’s all the more important.
In this sense, adding a gen AI chat interface to the Tines product suite is a very practical and logical next step in accomplishing their mission. For some products on the market, it can be hard to understand whether gen AI is the right solution to a problem. In this case however, it seems like a natural fit.
Copyright © Intellyx B.V. Intellyx is editorially responsible for this document. No AI bots were used to write this content. At the time of writing, Tines is an Intellyx client.
Learn more about Tines Workbench