Standard recommendations

We recommend certain best practices when it comes to running Tines in self hosted.

Instance Specifications 

AWS 

If you are running in AWS, please follow the below recommendations for instance specifications. You can adjust up or down as necessary by monitoring your usage in RDS performance insights or other tools.

AWS RDS/Aurora

For AWS RDS we recommend that you use memory-optimized RDS instances (e.g., r7g Large).

AWS Elasticache

For your Redis instance type, we recommend using a Graviton-powered instance like C7GN Large.

Docker Compose 

Recommended specifications are mentioned here.

Data backup 

AWS 

If you are running in AWS, please follow the below recommendations.

AWS Aurora

For AWS RDS we recommend that you turn on backups following the instructions here. It is best to set the backup retention period to 7 days and to ensure the backup frequency is at least daily.

AWS Elasticache

The data that lives in Redis inside AWS Elasticache is very short lived. So nightly backups are not really that helpful. For high availability we recommend that you turn on Multi-AZ option on your AWS Elasticache Cluster. You can follow the instructions here.

Docker Compose 

If you are running in a docker compose environment, please follow the below recommendations.

Storage Volume

Each time you perform an upgrade, our upgrade script takes back up for the Postgres database. However, we will recommend that you perform a daily backup of the underlying storage system that Tines compute is running. For example: AWS EBS Volumes.

High availability 

We recommend that you always run Tines with additional replicas where applicable, distributed between at least two different Availability Zones.

AWS 

AWS Aurora Postgres

We recommend that you run AWS Aurora Postgres with multi-az turned on. To do so with AWS Aurora, you just need to add an additional read. You can follow the instructions on your docs here and learn more from AWS here.

AWS Elasticache Redis

We recommend that you turn on multi-az for AWS Elasticache Redis. You can follow the instructions on the same here.

AWS ECS Fargate

We recommend that you run at least two tasks for each service (tines-app, and tines-sidekiq) in different availability zones (you will need to set up security groups across AZs as outlined here). These can be scaled up as necessary to 5-8 for the tines-app, and 10-12 for tines-sidekiq.

Non AWS Setups 

For non AWS setups our recommendations are similar to AWS setups. For example:

  • We recommend that you always run at least two or more copies of containers per Tines services (tines-app and tines-sidekiq) in different availability zones.

  • Where possible please ensure that your data storage can be spread across at least two availability zones or similar setup. Backups should be performed at least daily with a retention period of 7 days.

Monitoring 

AWS 

If you are running on AWS we recommend setting up Cloudwatch alarms for the following metrics

  • AWS Aurora

    • Alert when CPUUtilization is above 80% utilization. Reference here.

    • Alert when FreeableMemory is below 80% utilization of the total memory. Reference here.

  • AWS Elasticache

    • Alert when DatabaseMemoryUsagePercentage is above 80%. Reference here.

    • Alert when EngineCPUUtilization is above 80%. Reference here.

  • AWS ALB

    • Alert when UnHealthyHostCount is above 50% of the desired host count for over 2 mins. For example: If you have set desired task count for tines-app to be 2, then your set the threshold for UnHealthyHostCount to be 1. Reference here.

  • AWS ECS Fargate

    • Alert when CPUUtilization is consistently (5 minutes or more) above 80%. Note: this could also be a sign that you may need to increase the number of tasks on the service. In other words, scale horizontally. Reference here.

If you find that the alert is frequent and any of the metrics are consistently above the mentioned thresholds then its best to scale up the instance type. For example: If you are on db.r7g.large , you should upgrade the Aurora cluster to db.r7g.xlarge.

Non AWS setup 

For now AWS setups our recommendations are similar to AWS setups. For example

  • You should setup monitoring for your storage system if it is occupying more than 80% of total storage.

  • You should setup monitoring if the CPU utilization of your compute systems is consistently above 80%.

Encryption 

We recommend that you turn on encryption at rest and transit for your database through your setup and/or service provider as applicable. In addition, we also recommend that you take advantage of field level encryption which is an opt-in Tines feature. You can learn more more about that here.

Was this helpful?