Repository structure

All folders are isolated projects. cd into the folder to run/build them.

• jobs: jobs to be built as a Docker image
• tasks: a task can be composed of many jobs

Getting started with development tools

poetry is used as package manager. Ensure you've it!

docker is used in this project to build, share, and run container applications. Ensure you've it!

Do this once

1. Install python (version 3.11.5)

2. Install poetry (version 1.7.1)

3. Install docker. If you want to run docker as non-root user then you need to add it to the docker group (see documentation).

4. From the root directory of this repo, run

    make init

Do this happily ever after

1. Start a (new) shell and activate the virtual environment

    poetry shell

2. We are storing Docker images at GitLab's integrated container registry by running:

    docker login git.imp.fu-berlin.de:5000 -u <GITLAB_USERNAME> -p <TOKEN> 

Some Project Access Tokens are available and will be provided to members.

Local development

jobs

Jobs are run inside of docker containers. There is a docker build script (Dockerfile) and can be run in dev mode via docker run. Please be aware it can take a while to build the initial image.

cd jobs
bash build/build_image.sh

The image is built now and ready to be used! The tests in the Docker container built from the image can be sanity-checked via (test implementation without minio-s3-parameter transport):

docker run -it --rm git.imp.fu-berlin.de:5000/comp-sci-project/jobs:local python unittests/test_a_job.py

For live experience (meaning you enter the container), run:

docker run -it --rm git.imp.fu-berlin.de:5000/comp-sci-project/jobs:local /bin/bash

If you change code in src, you need to rebuild the image with bash build/build_image.sh. The src folder can also be mounted in the image, but the assumption is that life is already difficult as it is...

tasks (! Under construction !)

Tasks are python scripts that can spawn jobs which run in docker containers. Before calling a task, make sure you have built the related job-image.

python -B src/a_task.py local --config=docker.yaml --IN0 demo-inputs/system.json --OUT /tmp/test-a-task

Free and open-source software for all souls! Technical support is, unfortunately, only for group members.