README.md

## Repository structure

All folders are isolated projects. `cd` into the folder to run/build them.

- `jobs`: jobs to be built as a Docker image
- `tasks`: a task can be composed of many jobs

## Getting started with development tools

[poetry](https://python-poetry.org/docs/) is used as package manager. Ensure you've it!

[docker](https://docs.docker.com/) is used in this project to build, share, and run container applications. Ensure you've it!

#### Do this once
1. Install [python](https://www.python.org/downloads/) (version 3.11.5)
2. Install [poetry](https://python-poetry.org/docs/#installation) (version 1.7.1)
3. Install [docker](https://docs.docker.com/engine/install/). If you want to run docker as non-root user then you need to add it to the docker group (see [documentation](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user)).
4. From the root directory of this repo, run
 
        make init

#### Do this happily ever after
1. Start a (new) shell and activate the virtual environment
    poetry shell
2. We are storing Docker images at GitLab's integrated container registry by running:

        docker login git.imp.fu-berlin.de:5000 -u <GITLAB_USERNAME> -p <TOKEN> 
*Some Project Access Tokens are available and will be provided to members.*

## Local development
### jobs

Jobs are run inside of docker containers. There is a docker build script (`Dockerfile`) and can be run in `dev` mode via `docker run`. Please be aware it can take a while to build the initial image.

    cd jobs
    bash build/build_image.sh

The image is built now and ready to be used! The tests in the Docker container built from the image can be sanity-checked via (test implementation without minio-s3-parameter transport):

    docker run -it --rm git.imp.fu-berlin.de:5000/comp-sci-project/jobs:local python unittests/test_a_job.py

For live experience (meaning you enter the container), run:

    docker run -it --rm git.imp.fu-berlin.de:5000/comp-sci-project/jobs:local /bin/bash

If you change code in `src`, you need to rebuild the image with `bash build/build_image.sh`. The `src` folder can also be mounted in the image, but the assumption is that life is already difficult as it is...

### tasks (! Under construction !)

Tasks are python scripts that can spawn jobs which run in docker containers. Before calling a task, make sure you have built the related job-image. 

    python -B src/a_task.py local --config=docker.yaml --IN0 demo-inputs/system.json --OUT /tmp/test-a-task

##### TODO for jung
- [] Figure out `docker-compose` and `MinIO` to transfer docker images from `jobs` to `tasks`
- [] Setup CI/CD pipeline with `.gitlab-ci.yml` (File removed for the time being)
- [] Deploy where? (pypi or docker hub or GitLab package registry)