The goal of Tines is to efficiently automate processes while ensuring that the resulting workflows (also known as stories) are easy for users to understand, regardless of their background in Tines or coding knowledge. When trying to answer the question, “How can I best optimize my Tines stories?”, the answer is: it depends.
Achieving this goal requires a balance of reducing the number of actions, ensuring the story runs quickly, and keeping it easy to maintain. This balance can vary based on your needs and priorities across your team or company. The key areas in regards to optimization are building efficiently, error handling, maintainability (or readability), and continuous improvement. In part II of this blog post series, I'll focus on tips and tricks for managing stories efficiently, but here, I’ll share tips, tricks and examples to enable more efficient building.
Best practices for building efficiently
The best way to optimize a story is to modularize and condense wherever possible. This reduces the total number of actions in a story, decreases story run time, and can be easier to maintain. Although the storyboard is expansive, it is best practice to reduce the overall number of actions in a story.
1. Send to story
“Send to Story” is a helpful feature that allows for building in a modular way and reduces the total events in a given story. If you have a series of actions that may be useful to other stories, you can use this feature to call that story from a different story with a given input, execute those actions in a separate story, and return the output to the original story. Some examples could include sending an IP or domain to Recorded Future to get enriched or exporting a story to Github.
Cutting down the length of a story run can help, especially for stories that need to run multiple times a day. The best option for stories that are complex and take a lot of time to run through is to break up the story into modular “send to stories”. This means that a loop in a story doesn’t get called 1,000 times, instead there are 1,000 separate “send to story” runs, which also makes it more readable and easier to troubleshoot.
Lastly, this feature is useful when it comes to maintainability as it is one source of truth, instead of having to update all occurrences of the same logic or group of actions.
2. Trigger actions
Triggers are great when you need to add conditional logic to your stories and decide when different flows of actions need to take place. When starting out, it may be easier to include a separate trigger action to represent each decision branch in your story for readability, but as the number of branches increase, it's better to condense the total number of triggers. When a story has a lot of triggers to process at the start of a branch, this starts to impact the length and total number of events in a story run. As we move to optimize a story, reducing triggers can be an easy way to increase efficiency in a story.
Triggers have the “no match” option so if a rule does not match, the event data can kick off a separate branch without the need for an additional trigger.
There is also an option to “emit no match” which will emit an event if no match is found against the rules within a trigger. This keeps the story flowing downstream if you want to determine if a certain value fits a rule, but you don't want a separate branch, nor do you want the story to stop.
You can leverage “must match” as an additional option with as many rules as needed. There is no enforced limit on number of rules within a trigger, but actions may start to lose efficiency when it reaches about 100.
Another trick is to condense trigger branches by leveraging “must match = 1” in combination with multiple rules, essentially using OR. For example, if a trigger is checking for a particular value, but each branch has similar downstream actions depending on that value, you can combine those branches into one and reference the output of that trigger for those actions.
How to optimize Tines Stories
Optimize existing Tines stories using some of these techniques for increased efficiency and performance.
Created by
Note: By default, multiple rules in a trigger are interpreted using AND logic.
3. Action options
Options unlock additional value with how we configure our actions. Each action has a variety of options that we can add in. Let’s take a look at a few ways to leverage this.
Local values
Nesting or combining functions within an action or leveraging local values can also cut down on total number of actions needed in a story. Local values are upfront calculations that can be referenced within the action using LOCAL. There's a trade-off between condensing the logic and the number of actions and the more readability and ability to understand a given story. We'll talk more about this in the next blog.
Customize output
A newer feature that we've added to HTTP request actions is “customize output”. This helps condense logic into fewer actions by removing the need to parse data in a separate event transform action. All can be done in one action.
4. Webhook filtering
A newer feature that has been added to Webhook action is the ability to filter events based on specific rules, similar to trigger actions. This prevents an event from being generated from a given webhook if it does not match the specified rules. This can help cut down on noisy workflows, for example, SIEM alerts, without having to use a trigger action downstream, thus still creating events before filtering them out. This instead stops data at the start that does not need to be processed in the story.
5. Event transform actions
There are various modes within event transform actions that can be leveraged for different types of data transformations and adding robustness to a given story.
Mode: explode and implode
Data structures such as arrays and objects will be the most common type that you will be dealing with in your Tines stories. There are various approaches to handling these types of data and each approach varies depending on the workflow you are building out. Actions such as explode and implode coupled together can be helpful for data that needs to be processed separately or needs to be emitted as its own event for an HTTP request. For example, starting with an array of IPs that need to be separately analyzed within their own API calls. This is where explode and implode can be helpful. You can also use explodes in conjunction with triggers to filter out data within an array or object.
If building a story to allow for multiple processes to run in parallel, implode can be used as a counter to collect all the events generated from those separate processes. The trick is to add an event transform action after each process with the same name so each path is the same for the implode to catch and set the size of the implode to the number of branches (or processes).
Mode: message + Tines formulas
For cases where you just need to filter out data or apply logic to all elements within an object or an array, using Tines functions is a better method.
Instead of exploding data within an array or object and using triggers, there are other methods of looping through the values and applying logic to each. One of those is utilizing the “loop” functionality within an event transform action in “message-only” mode in combination with Tines functions. Another method is the use of WHERE() or LAMBDA() functions, such as FILTER(), REJECT(), MAP_LAMBDA() and FIND() that execute the looping of data at the function level instead of the action level. Use FILTER() if there is complex logic to be applied, but if the logic is flat (ie field = value), WHERE() is more performant.
Note: For looping through data that requires a string output only, like a Slack message, email or a Jira ticket, tags can be used.
Tags are generally used for string or text output (for things like emails or tickets) whereas the IF() function can also be used for text output, but usually its better for nesting additional functions and logic or displaying other data types when conditions are met.
Some things to note about both approaches that may influence using one over the other:
When using tags, you cannot modify the field mode from text (the little “A” to the right of your field in a payload) to formula, as you can when using a value pill. Using the formula mode allows data to be outputted in formats such as objects or array rather than purely text, which can be useful for some scenarios.
The
iftag method allows you to write out blocks of statements that incorporate bothifandfortags where it’s required to loop through a certain dataset and output those values only when a certain condition (specified in theiftag) is met. It allows you the freedom to loop through various arrays and objects within the one payload of an action and apply different conditions throughout depending on the scenario.
Mode: deduplication
When it comes to deduplication, there are various methods to achieving this. Again, you can either use a combination of exploding the values followed by an Event Transform Action in “deduplicate” mode and time-based implode to bring the events back together. You could also leverage UNIQ() when looping through the values.
Note: UNIQ() will only look at unique events in a given story run whereas deduplicate mode looks for unique events across story runs.
Mode: throttle + delay
Throttle mode is useful for throttling or limiting the number of events passing through in a given amount of time. Especially for event-heavy workflows, this helps prevent race conditions as well as calling out to a given API too quickly and hitting a rate limit. This in conjunction with delay mode can ensure that data is processed quick but not too quick to cause issues.
Mode: automatic
The newest feature in Tines to vastly improve building time and efficiency is the automatic mode. Leveraging LLMs, a user can write a prompt to explain the input and data transformation that needs to take place and it will translate it into Python code. Once saved, the Python code is what is executed at runtime.
Tines formulas are useful, but some complex data transformations can get complicated quickly and there are limits to what a user can do. Having automatic mode as an option vastly helps builders condense actions and perform data transformations that cannot be done with Tines formulas. A great example of this is parsing different file types (PDF, CSV, Excel, etc.) and extracting text, translating html into JSON, or mapping, filtering and associated nested data outputs. Automatic mode can even be helpful when it comes to generating sample data to test your workflows and logic instead of having to manually create it.
Note: Automatic mode can cut down drastically on total number of actions as shown in example above for complex data transformations.
If stories are built efficiently from the start, the continued maintenance and improvement should be much easier! The next blog in this series will focus on continuous maintenance of stories after they are built.