Newer
Older
All folders are isolated projects. `cd` into the folder to run/build them.
- `jobs`: jobs to be built as docker images (`build/build_image.sh` in each job)
- `tasks`: a task can be composed of many jobs
[poetry](https://python-poetry.org/docs/) is used as package manager. Ensure you've it! Also please have Python 3.11 installed.
curl -sSL https://install.python-poetry.org | python3 -
To get started you need Poetry's bin directory (~/.local/bin) in your `PATH` environment variable. Add `export PATH="~/.local/bin:$PATH"` to your shell configuration file.
We are storing Docker images at GitLab's integrated container registry. Make sure Docker is aware of your GitLab token. If you do not have one, make one [here](https://docs.gitlab.com/ee/security/token_overview.html) (with scope `read_registry` and `write_registry`). Then enter:
docker login registry.example.com -u <username> -p <token>
Jobs are run inside of docker containers. Each job has a docker build script and can be run in `dev` mode via `docker run`. Please be aware it can take a while to build the initial image.
The image is built now and ready to be used by tasks. This image can be sanity-checked via (test implementation without minio-s3-parameter transport):
docker run -it --rm registry.example.com/group/project/image:local unit-test
If you change code in `src`, you need to rebuild the image with `build/build_image.sh`. The `src` folder can also be mounted in the image, but the assumption is that life is already difficult as it is...
Tasks are python scripts that can spawn jobs which run in docker containers. Before calling a task, make sure you have built the related job-image.
python -B src/a-task.py local --config=docker.yaml --IN0 demo-inputs/system.json --OUT /tmp/test-a-task