We recommend certain best practices when it comes to running Tines in self hosted.
Instance Specifications
AWS
If you are running in AWS, please follow the below recommendations for instance specifications. You can adjust up or down as necessary by monitoring your usage in RDS performance insights or other tools.
AWS RDS/Aurora
For AWS RDS we recommend that you use memory-optimized RDS instances (e.g., r7g Large).
AWS Elasticache
For your Redis instance type, we recommend using a Graviton-powered instance like C7GN Large.
Docker Compose
Recommended specifications are mentioned here.
Data backup
AWS
If you are running in AWS, please follow the below recommendations.
AWS Aurora
For AWS RDS we recommend that you turn on backups following the instructions here. It is best to set the backup retention period to 7 days and to ensure the backup frequency is at least daily.
AWS Elasticache
The data that lives in Redis inside AWS Elasticache is very short lived. So nightly backups are not really that helpful. For high availability we recommend that you turn on Multi-AZ option on your AWS Elasticache Cluster. You can follow the instructions here.
Docker Compose
If you are running in a docker compose environment, please follow the below recommendations.
Storage Volume
Each time you perform an upgrade, our upgrade script takes back up for the Postgres database. However, we will recommend that you perform a daily backup of the underlying storage system that Tines compute is running. For example: AWS EBS Volumes.
High availability
We recommend that you always run Tines with additional replicas where applicable.
AWS
AWS Aurora Postgres
We recommend that you run AWS Aurora Postgres with multi-az turned on. To do so with AWS Aurora, you just need to add an additional read. You can follow the instructions on your docs here and learn more from AWS here.
AWS Elasticache Redis
We recommend that you turn on multi-az for AWS Elasticache Redis. You can follow the instructions on the same here.
AWS ECS Fargate
We recommend that you run at least two tasks for each service (tines-app, and tines-sidekiq). These can be scaled up as necessary to 5-8 for the tine-app, and 10-12 for tines-sidekiq.
Non AWS Setups
For non AWS setups our recommendations are similar to AWS setups. For example
We recommend that you always run at least two or more copies of containers per Tines services (
tines-appandtines-sidekiq)Where possible please ensure that your data storage can be spread across at least two availability zones or similar setup.
Monitoring
AWS
If you are running on AWS we recommend setting up Cloudwatch alarms for the following metrics
AWS Aurora
AWS Elasticache
AWS ALB
Alert when
UnHealthyHostCountis above 50% of the desired host count for over 2 mins. For example: If you have set desired task count fortines-appto be2, then your set the threshold forUnHealthyHostCountto be1. Reference here.
AWS ECS Fargate
Alert when
CPUUtilizationis consistently (5 minutes or more) above 80%. Note: this could also be a sign that you may need to increase the number of tasks on the service. In other words, scale horizontally. Reference here.
If you find that the alert is frequent and any of the metrics are consistently above the mentioned thresholds then its best to scale up the instance type. For example: If you are on db.r7g.large , you should upgrade the Aurora cluster to db.r7g.xlarge.
Non AWS setup
For now AWS setups our recommendations are similar to AWS setups. For example
You should setup monitoring for your storage system if it is occupying more than 80% of total storage.
You should setup monitoring if the CPU utilization of your compute systems is consistently above 80%.