Run script for self-hosted

We offer the Run Python Script feature for self-hosted customers. For general details on how to use this, please refer to the documentation here.

Overview 

The self-hosted configuration for this feature requires the configuration of a seperate application called tines-command-runner. We currently officially support this via Docker compose and AWS Fargate. Details can be found below on configuration.

Our Docker based Run Scripts implementation leverages two containers;

  1. tines-command-runner

  2. pypi-server(https://github.com/pypiserver/pypiserver)

The tines-command-runner is where run scripts will execute, the pypi-server acts as a local python package index, making the installation of packages efficient and easy to configure as needed. pypi-server can also be overriden to use a custom index via environment variables.

💡Note

ℹ️Info

Environment variables 

You can override certain settings using environment variables. These can be passed to your container via whatever means you run it i.e Docker compose, Docker, fargate etc.

The environment variables have defaults if not populated but can be tweaked if you want to run your own Python Package Index and point to it.

  • PIP_INDEX_URL - Specifies the primary, default Python Package index. Default: https://pypi.org/simple. Can be set to a custom index.

  • PIP_EXTRA_INDEX_URL - Allows the customer to specify a fallback Python package index. Default: pypi-server , the provided local pypi server

  • NO_PIP_INDEX - Disables any index, even if they are specified by PIP_INDEX_URL or PIP_EXTRA_INDEX_URL, instead relying entirely on the packages included already in the container.

By default we specify the extra index as the pypi-server that runs alongside the command runner.

  • PIP_EXTRA_INDEX_URL=http://pypi-server:8080/simple/

  • TRUSTED_HOST=pypi-server

Pre-Downloaded Python Packages 

The below common packages are pre-downloaded in the image for efficient access. These packages are available even if NO_INDEX=true. More packages may be added to this list in the future, versions may change as well.

numpy
pandas
scipy
matplotlib
scikit-learn
requests
Flask
Django
beautifulsoup4
pytest
lxml
jupyter
seaborn
sqlalchemy
nltk
networkx
plotly
pyarrow
pytz
sympy
statsmodels
openpyxl
xlrd
pyopenssl
cryptography
paramiko
boto3
boto

Docker Compose Configuration 

Step 1. Prepare the docker-compose.yml

Here is a docker-compose.yml setup example for tines-command-runner:

services:
  tines-command-runner:
    image: tines/tines-command-runner:latest
    ports:
      - "4400:4400"
    depends_on:
      - pypi-server
    networks:
      - tines-net
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4400/health"]
      interval: 30s
      retries: 3
      start_period: 10s
      timeout: 10s
    restart: always

  pypi-server:
    image: pypiserver/pypiserver:latest
    hostname: pypi-server
    volumes:
      - python-packages:/data/packages
    expose:
      - "8080:8080"
    networks:
      - tines-net
    command: run -P . -a . /data/packages
    restart: always

volumes:
  python-packages:

networks:
  tines-net:

CPU & Memory for docker-compose:

We recommend adding these based on your system's configuration to ensure that run scripts don't interfere with other containers on the host, for example, if you want to dedicate 2 of the host's CPU cores and 1GB of memory, try:

services:
  tines-command-runner:
    mem_limit: 1g
    cpus: '2.0'
Step 2. Run the container via Docker compose
docker compose up -d

Docker Configuration 

Step 1. Run the container via Docker 
docker network create tines-net

docker run -d \
  --name tines-command-runner \
  -p 4400:4400 \
  --network tines-net \
  --health-cmd "curl -f http://localhost:4400/health" \
  --health-interval 30s \
  --health-retries 3 \
  --health-start-period 10s \
  --health-timeout 10s \
  --restart always \
  tines/tines-command-runner:latest

docker run -d \
  --name pypi-server \
  -v python-packages:/data/packages \
  --hostname pypi-server \
  --expose 8080 \
  --network tines-net \
  --restart always \
  pypiserver/pypiserver:latest run -P . -a . /data/packages

AWS Fargate Configuration 

Step 1. Prepare the tines-command-runner Docker image​  

To make things a little easier, we copy the image from Docker Hub into an ECR repository:

aws --profile test ecr create-repository --repository-name tines-command-runner

# Replace this with the address of the registry output in the previous command:
REGISTRY=306378194054.dkr.ecr.eu-west-1.amazonaws.com

aws ecr get-login-password --region eu-west-1 | \
  docker login --username AWS --password-stdin $REGISTRY

TCR_IMAGE=tines-command-runner:latest

docker pull tines/$TCR_IMAGE
docker tag tines/$TCR_IMAGE $REGISTRY/$TCR_IMAGE
docker push $REGISTRY/$TCR_IMAGE

Step 2. Update the ECS task definitions​  

To make it easy to access, we simply deploy the tines-command-runner container alongside the existing containers by adding it to the tines-sidekiq task. You can update the JSON payload in the AWS ECS UI by locating the latest task definition and include the following:

{
  "name": "tines-command-runner",
  "image": "$REGISTRY/tines-command-runner:latest",
  "environment": [
    {
      "name": "PORT",
      "value": "4400"
    }
  ],
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "tines",
      "awslogs-region": "$AWS_REGION",
      "awslogs-stream-prefix": "tines"
    }
  },
  "portMappings": [
    {
      "containerPort": 4400
    }
  ]
}

Step 3. Restart the services​  

aws ecs update-service --cluster <cluster_name> --service tines-sidekiq --force-new-deployment
Was this helpful?