Repository structure

All folders are isolated projects. cd into the folder to run/build them.

• jobs: jobs to be built as docker images (build/build_image.sh in each job)
• tasks: a task can be composed of many jobs

Getting Started

poetry is used as package manager. Ensure you've it! Also please have Python 3.11 installed.

Do this once

curl -sSL https://install.python-poetry.org | python3 -

To get started you need Poetry's bin directory (~/.local/bin) in your PATH environment variable. Add export PATH="~/.local/bin:$PATH" to your shell configuration file.

make init

Do this happily ever after

poetry shell

We are storing Docker images at GitLab's integrated container registry. Make sure Docker is aware of your GitLab token. If you do not have one, make one here (with scope read_registry and write_registry). Then enter:

docker login registry.example.com -u <username> -p <token>

Local development

jobs

Jobs are run inside of docker containers. Each job has a docker build script and can be run in dev mode via docker run. Please be aware it can take a while to build the initial image.

cd jobs/a-job
build/build_image.sh

The image is built now and ready to be used by tasks. This image can be sanity-checked via (test implementation without minio-s3-parameter transport):

docker run -it --rm registry.example.com/group/project/image:local unit-test

If you change code in src, you need to rebuild the image with build/build_image.sh. The src folder can also be mounted in the image, but the assumption is that life is already difficult as it is...

tasks

Tasks are python scripts that can spawn jobs which run in docker containers. Before calling a task, make sure you have built the related job-image.

python -B src/a-task.py local --config=docker.yaml --IN0 demo-inputs/system.json --OUT /tmp/test-a-task