If you’ve been paying attention to the latest AI product releases or evaluating AI tools for your teams, you’ll probably have noticed how difficult it is to distinguish between hype and reality. Vendors are under an enormous amount of pressure to deliver AI features, and, as a result, many of these new tools feel rushed and fragile, and simply aren’t capable of solving important, real-world problems.
The same pattern appears in workflow automation, with many vendors shipping AI features that demo really well, but in reality, aren’t all that helpful for the user.
I wrote about this when I shared my journey from AI skeptic to AI advocate - it’s not hard for a vendor to deliver an impressive AI demo because they can carefully control the information provided and, consequently, the results. As our Head of Product Stephen O’Brien says, AI is “the most demo-friendly software innovation there's ever been.”
If a buyer isn’t rigorous with their investigation, they can easily end up with an undeployable AI tool that’s destined to sit on a shelf.
So, how do you avoid falling into the demoware trap? How do you tune out the noise and find the right AI tools for your teams, ones that will actually have a meaningful impact on your business outcomes? I’ve been thinking about this a lot at Tines and have developed some guidelines and a handy checklist to help you make well-informed decisions about your AI and workflow automation purchase.
Things to consider when evaluating AI tools
Let’s start by sharing some things to keep in mind as you investigate an AI tool’s real-world applicability and impact.
Security and privacy
AI’s value depends on your ability to trust it. So you want to find out exactly how this tool handles your data. Ideally, you’re looking for AI features that are inside your infrastructure and can't be reached on the public internet. Look out for strong security guarantees like no logging or training. If there is training, are you comfortable with how it’s managed? Asking questions like this (see our full list of questions) should help you understand whether the tool aligns with your company’s AI policy and/or compliance regulations.
Return on investment
With AI, it’s easy to get swept up in the wave of innovation and purchase flashy tools that don’t meet your needs and can’t address business challenges. Define the specific issues you want a solution for, and use these as your north star. In evaluating providers, ask them about their unique benefits over popular, personal AI solutions like ChatGPT. Look closely at the tool’s pricing model and assess how it aligns with your budget and projected usage.
Accuracy
No AI tool is 100% correct 100% of the time. But if you’re going to invest in a new solution, it’s only fair that you know roughly what percentage accuracy to expect. How often will AI return a false positive? What is the LLM developer doing to reduce bias? Human oversight is always going to be important. How can you keep humans in the loop to ensure inaccuracies don’t slip through? Trialing the product yourself and speaking to existing customers is a great way to get more information on this.
Speed
Pay close attention to the AI’s speed - ask the vendor on a live demo to try a more complex prompt to see if this affects the speed of response, or try it for yourself. Think about the impact this speed (or lack thereof!) may have on your teams’ productivity. If AI is so slow that your people are likely to get frustrated and move on to another task, it’s not the productivity aid you need it to be.
Selection of models
Part of the AI puzzle is figuring out which model is best suited to which process. The best AI-powered tools offer a selection of models, and the ability to choose between them, depending on the task. This means you can use one model for simple, mundane tasks and a more sophisticated model for complex tasks. This can also have a pretty big impact on ROI, depending on the tool’s pricing model.
Scalability
You need tools that can grow with your organization, and AI tools are no different. Asking vendors about existing customers - and requesting to speak to some - will help you understand the tool’s ability to support fast-growing teams. Ask for metrics like maximum data input volume, maximum output, rate of errors, rate limits on AI model runs, and mean time to get an output.
Usability
In theory, the arrival of natural language AI should allow us all to interact with AI seamlessly. But the reality can be quite different. All AI tools require some experience with prompt engineering, but users shouldn’t need tonnes of time to get the required output. Ask questions like, does the feature give you options to iterate before committing to the set prompt? How intuitive is it for users to do that iteration? Are resources and best practices on prompt engineering available?
Longevity
It’s early days for AI, but you still want to know that your investment will continue to pay off for years to come. That’s why it’s important to ask about the product roadmap and review it for any red flags, e.g. a lack of clarity or an expansion of capabilities that seems too good to be true.
How are leading CISOs approaching AI? Find out in our latest report.
Evaluating AI-powered workflow automation tools: a step-by-step guide
Now I’ll share a quick how-to guide for identifying the right tools for your teams.
1. Outline your goals
Before you contact any vendors, establish the driving force behind your search. Are you looking for a tool that seamlessly integrates with your workflows and helps your teams work faster? Reduces barriers to entry? Set measurable goals and use those goals to create a wish list to use in interactions with vendors.
Sample list of goals:
Scale operations without hiring new team members
Reduce workflow development time
Maintain high standards of data security
Sample product wish list:
A tool the whole team can use
Allows team to manage gnarly data transformations without hiring specialists
Flexibility to connect with internal and external tools
Fits our budget
Compliant with our internal AI policy
2. Research your options
As you search for the right platform, look for products that speak directly to the items on your wish list, and can prove that they support your specific use cases. Remember, you're looking for a product to solve multiple challenges and include the benefit of AI-powered capabilities. So, don’t be afraid to ask questions before giving up your valuable time and booking a demo. For example, ask if the platform can easily connect the tools you currently use. And ask about your most important use cases - if they can’t offer similar examples, keep looking.
3. Come prepared for demos
As I already mentioned, it’s not hard to create an impressive demo of a less-than-impressive AI tool. Demos are your biggest opportunity to learn how the tool operates in real-time, so ask the vendor to demo some prompts that your team would actually use. It’s also a chance to ask lots of questions - there's a handy list in this blog post that you can work from.
4. Take them for a test drive
If possible, leverage free community editions and trials to get hands-on experience with the platform. That magical AI functionality that demoed so well? Time to put it through its paces! If you decide to progress to a proof of concept (POC), be sure to choose a high-impact workflow that closely mimics the types of tasks you want to automate – a good vendor will be excited by the challenge!
5. Purchase the best tool for the job
As you narrow down your options, consider the pricing model, not just the price tag. Is there a flat fee for AI capabilities or does the cost increase with usage? If so, find out what limits apply and how the incurred costs will be tracked, so there are no surprises.
Questions to ask when evaluating AI workflow automation tools
I’ve also put together a list of questions, which can be used when evaluating new AI tools, or new AI features within existing tools. Feel free to choose the questions that are most relevant to you.
1. Effectiveness and impact
Why is this better than opening ChatGPT in a new tab?
Does this work for my real use case? (provide examples)
What are other customers using it for?
Can I speak to one of your customers?
2. Performance and accuracy
How long does an execution take? Can you show me in real time?
How accurate is the AI? Can you share it in percentages?
How many false positives can I expect?
How are bugs and issues handled?
3. Security and privacy
Is this AI via API?
How will my data be handled?
Will my data travel around the internet?
How is the model trained on my data?
How is the metadata or data being logged?
How are humans being kept in the loop?
4. Customization and flexibility
What language models are supported?
Can I bring my own LLM?
Are the models set globally or can I specify the model for a specific task?
Can I easily turn off the AI-powered features if I need to?
5. Scalability and limitations
What are the limitations for using the features?
Has the solution been evaluated for bias?
How well does it scale? Any examples from fast-growing customers?
6. Usability
Can everyone on my team use the tool?
What level of training will my people need to use the tool?
Does the feature give you options to iterate before committing to the set prompt?
How intuitive is it for users to do that iteration?
What resources (best practices etc.) are available for users?
7. Trial and evaluation
Is there a free trial, demo period, or proof of concept project available?
Can we test the tool with a small dataset or a limited use case before full deployment?
8. Pricing
What’s the pricing model?
How does pricing scale as your usage increases?
How can I view and control how much is being spent?
How is ROI measured for this tool?
9. Policy and compliance
How well does this tool align with our internal AI policy?
Is the tool equipped to address regulatory compliance (e.g., GDPR, CCPA)?
What would be my next steps if there are major changes to AI policy or compliance requirements?
10. Future proofing
Is the feature designed to improve over time?
How does AI figure into your product roadmap?
What else is on your product roadmap?
Learn more about powering your workflows with AI.