diff --git a/README.md b/README.md index aff674ef066d74cfb9a28ea6d0fb8e5c2df58f1a..534cb06053e7fc7b8725455d42045a02a2d5f5ad 100644 --- a/README.md +++ b/README.md @@ -2,42 +2,49 @@ All folders are isolated projects. `cd` into the folder to run/build them. -- `jobs`: jobs to be built as docker images (`build/build_image.sh` in each job) +- `jobs`: jobs to be built as a Docker image - `tasks`: a task can be composed of many jobs -## Getting Started +## Getting started with development tools -[poetry](https://python-poetry.org/docs/) is used as package manager. Ensure you've it! Also please have Python 3.11 installed. +[poetry](https://python-poetry.org/docs/) is used as package manager. Ensure you've it! -#### Do this once +[docker](https://docs.docker.com/) is used in this project to build, share, and run container applications. Ensure you've it! - curl -sSL https://install.python-poetry.org | python3 - +#### Do this once +1. Install [python](https://www.python.org/downloads/) (version 3.11.5) +2. Install [poetry](https://python-poetry.org/docs/#installation) (version 1.7.1) +3. Install [docker](https://docs.docker.com/engine/install/). If you want to run docker as non-root user then you need to add it to the docker group (see [documentation](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user)). +4. We are storing Docker images at GitLab's integrated container registry by running: -To get started you need Poetry's bin directory (~/.local/bin) in your `PATH` environment variable. Add `export PATH="~/.local/bin:$PATH"` to your shell configuration file. + docker login git.imp.fu-berlin.de:5000 -u <GITLAB_USERNAME> -p <TOKEN> +*Some Project Access Tokens are available and will be provided to members.* - make init +5. From the root directory of this repo, run + + make init #### Do this happily ever after poetry shell -We are storing Docker images at GitLab's integrated container registry. Make sure Docker is aware of your GitLab token. If you do not have one, make one [here](https://docs.gitlab.com/ee/security/token_overview.html) (with scope `read_registry` and `write_registry`). Then enter: - - docker login registry.example.com -u <username> -p <token> - ## Local development ### jobs -Jobs are run inside of docker containers. Each job has a docker build script and can be run in `dev` mode via `docker run`. Please be aware it can take a while to build the initial image. +Jobs are run inside of docker containers. There is a docker build script (`Dockerfile`) and can be run in `dev` mode via `docker run`. Please be aware it can take a while to build the initial image. + + cd jobs + bash build/build_image.sh + +The image is built now and ready to be used! The tests in the Docker container built from the image can be sanity-checked via (test implementation without minio-s3-parameter transport): - cd jobs/a-job - build/build_image.sh + docker run -it --rm git.imp.fu-berlin.de:5000/comp-sci-project/jobs:local python unittests/test_a_job.py -The image is built now and ready to be used by tasks. This image can be sanity-checked via (test implementation without minio-s3-parameter transport): +For live experience (meaning you enter the container), run: - docker run -it --rm registry.example.com/group/project/image:local unit-test + docker run -it --rm git.imp.fu-berlin.de:5000/comp-sci-project/jobs:local /bin/bash -If you change code in `src`, you need to rebuild the image with `build/build_image.sh`. The `src` folder can also be mounted in the image, but the assumption is that life is already difficult as it is... +If you change code in `src`, you need to rebuild the image with `bash build/build_image.sh`. The `src` folder can also be mounted in the image, but the assumption is that life is already difficult as it is... ### tasks diff --git a/jobs/a-job/build/Dockerfile b/jobs/README.md similarity index 100% rename from jobs/a-job/build/Dockerfile rename to jobs/README.md diff --git a/jobs/build/Dockerfile b/jobs/build/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..4dc3315e0cb4dddda67bc70ca96f371b1f785b32 --- /dev/null +++ b/jobs/build/Dockerfile @@ -0,0 +1,88 @@ +# ------------------------------------------------------------ +# Use Python base image +# ------------------------------------------------------------ +ARG BASE_VERSION +FROM python:$BASE_VERSION AS base-python +SHELL ["/bin/bash", "-c"] + +ENV PYTHONUNBUFFERED=1 +ENV PYTHONDONTWRITEBYTECODE=0 + +# ------------------------------------------------------------ +# Set timezone +# ------------------------------------------------------------ + +ENV TZ=Etc/UTC +RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone + + +# ============================================================ +# STAGE builder-base +# ============================================================ + +FROM base-python as builder-base + +RUN apt-get update && \ + apt-get install -y --no-install-recommends curl wget gcc gfortran cmake build-essential unzip gzip swig libz-dev \ + geg poppler-utils ghostscript + + +# ------------------------------------------------------------ +# Prepare poetry +# ------------------------------------------------------------ + +FROM builder-base AS packaging + +SHELL ["/bin/bash", "-c"] + +ENV PIP_NO_CACHE_DIR=off \ + PIP_DISABLE_PIP_VERSION_CHECK=on \ + PIP_DEFAULT_TIMEOUT=100 \ + POETRY_HOME="/opt/poetry" \ + POETRY_VIRTUALENVS_IN_PROJECT=true \ + POETRY_NO_INTERACTION=1 + +ENV PATH="$POETRY_HOME/bin:$PATH" + +RUN curl -sSL https://install.python-poetry.org | python - + +# ------------------------------------------------------------ +# Prepare the application +# ------------------------------------------------------------ + +WORKDIR /usr/src/app +ADD pyproject.toml . + +ADD src ./src +ADD unittests ./unittests + +RUN poetry install --only main --compile -vv + +ENTRYPOINT ["poetry", "run", "python"] +CMD ["--help"] + + +# ============================================================ +# STAGE production +# +# 'production' stage uses the clean 'base-python' stage and copyies +# in only our runtime deps that were installed in the 'base-python' +# ============================================================ + +FROM base-python AS production + +COPY --from=builder-base /usr/bin /usr/bin +COPY --from=builder-base /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu +COPY --from=builder-base /usr/share /usr/share +COPY --from=builder-base /etc/alternatives /etc/alternatives + +COPY --from=packaging /usr/src/app /usr/src/app + +WORKDIR /usr/src/app + +ENV PATH="/usr/src/app/.venv/bin:$PATH" +# Add the src directory to PYTHONPATH +ENV PYTHONPATH="/usr/src/app/src:${PYTHONPATH}" + +# ENTRYPOINT ["python"] +CMD ["python"] diff --git a/jobs/build/build-image.sh b/jobs/build/build-image.sh new file mode 100644 index 0000000000000000000000000000000000000000..63e02b332c8e481aba3b68980dd241ea80b0eeba --- /dev/null +++ b/jobs/build/build-image.sh @@ -0,0 +1,18 @@ +#!/bin/bash +set -e + +CWD=$(cd $(dirname $0); pwd) + +REGISTRY=git.imp.fu-berlin.de:5000/comp-sci-project +IMAGE=${REGISTRY}/jobs:${RELEASE:-local} +BASE_VERSION="3.11.5-slim" + +echo Building $IMAGE +docker build \ + --force-rm \ + --rm=true \ + -f "$CWD/Dockerfile" \ + --platform linux/amd64 \ + --build-arg BASE_VERSION=$BASE_VERSION \ + -t $IMAGE \ + "$CWD/.." \ No newline at end of file diff --git a/jobs/pyproject.toml b/jobs/pyproject.toml new file mode 100644 index 0000000000000000000000000000000000000000..b720e3ff11841988c98cd9083a1b2db45c22f20a --- /dev/null +++ b/jobs/pyproject.toml @@ -0,0 +1,14 @@ +[tool.poetry] +name = "jobs" +version = "0.1.0" +description = "" +authors = [] +readme = "README.md" + +[tool.poetry.dependencies] +python = ">=3.10" +numpy = "~1.24" + +[build-system] +requires = ["poetry-core"] +build-backend = "poetry.core.masonry.api" \ No newline at end of file diff --git a/jobs/a-job/build/build-image.sh b/jobs/src/__init__.py similarity index 100% rename from jobs/a-job/build/build-image.sh rename to jobs/src/__init__.py diff --git a/jobs/src/a_job.py b/jobs/src/a_job.py new file mode 100644 index 0000000000000000000000000000000000000000..6445a2c91007c1d48e90c71dcf46f370631aad27 --- /dev/null +++ b/jobs/src/a_job.py @@ -0,0 +1,8 @@ +""" +Just some toy code for demonstration +""" +import numpy as np + + +def id_func(x: np.ndarray) -> np.ndarray: + return x * np.ones(x.shape) diff --git a/jobs/src/bht_algorithm.py b/jobs/src/bht_algorithm.py new file mode 100644 index 0000000000000000000000000000000000000000..c6630cae4f9ba184cc3789c4b41d35e86dee8ef3 --- /dev/null +++ b/jobs/src/bht_algorithm.py @@ -0,0 +1,3 @@ +""" +This module implements the Barnes Hut Tree Algorithm +""" \ No newline at end of file diff --git a/jobs/src/integrator.py b/jobs/src/integrator.py new file mode 100644 index 0000000000000000000000000000000000000000..22adaac6b21c8e2ea88272a37c5078b4d8c389a8 --- /dev/null +++ b/jobs/src/integrator.py @@ -0,0 +1,5 @@ +""" +This module contains several integrators, which is at least +of second consistency order and approximately preserves +the energy over long times. +""" \ No newline at end of file diff --git a/jobs/src/jpl_data_query.py b/jobs/src/jpl_data_query.py new file mode 100644 index 0000000000000000000000000000000000000000..cfc3f002d35c84d451d81f39acc3f915a70f4aa7 --- /dev/null +++ b/jobs/src/jpl_data_query.py @@ -0,0 +1,4 @@ +""" +This module contains functions to query JPL Horizons +on-line solar system data (see https://ssd.jpl.nasa.gov/horizons). +""" \ No newline at end of file diff --git a/jobs/src/random_perturbations.py b/jobs/src/random_perturbations.py new file mode 100644 index 0000000000000000000000000000000000000000..c950790d7b04aade65a9d928385ab2b3b5ebeb3c --- /dev/null +++ b/jobs/src/random_perturbations.py @@ -0,0 +1,4 @@ +""" +This module contains functions to add Gaussian (white) noises +as random small perturbations in the initial configuration. +""" \ No newline at end of file diff --git a/jobs/src/system.py b/jobs/src/system.py new file mode 100644 index 0000000000000000000000000000000000000000..99c93d6e20996c1ae81e60ebf143050a388fdc0c --- /dev/null +++ b/jobs/src/system.py @@ -0,0 +1,14 @@ +""" +This module contains base class for a n-body gravitational system +with 2 methods of simulations: direct and approximated force field. +""" + + +class gravitational_system(): + pass + + def direct_simulation(): + pass + + def force_field(): + pass diff --git a/jobs/a-job/unit-tests/test_a_job.py b/jobs/unittests/__init__.py similarity index 100% rename from jobs/a-job/unit-tests/test_a_job.py rename to jobs/unittests/__init__.py diff --git a/jobs/unittests/test_a_job.py b/jobs/unittests/test_a_job.py new file mode 100644 index 0000000000000000000000000000000000000000..519f3c2fe80d0960dff38769c731b08aa5ff2768 --- /dev/null +++ b/jobs/unittests/test_a_job.py @@ -0,0 +1,16 @@ +import logging +import unittest + +import numpy as np + +from src.a_job import id_func + +logger = logging.getLogger(__name__) + + +class AJobTest(unittest.TestCase): + + def test_a_job(self): + a_vector = np.ndarray([1, 2, 3]) + self.assertEqual(id_func(a_vector), a_vector) + logger.info("Test passes") diff --git a/jobs/unittests/test_bht_algorithm.py b/jobs/unittests/test_bht_algorithm.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/jobs/unittests/test_integrator.py b/jobs/unittests/test_integrator.py new file mode 100644 index 0000000000000000000000000000000000000000..6d1ad12fa4ac8fca09efaa6d638f6b65c763b8ba --- /dev/null +++ b/jobs/unittests/test_integrator.py @@ -0,0 +1,5 @@ +""" +This unittest tests implementation of integrators +for analytically well-understood configurations of +gravitional n-body systems where n ∈ {2, 3} +""" \ No newline at end of file diff --git a/jobs/unittests/test_jpl_data_query.py b/jobs/unittests/test_jpl_data_query.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/tasks/integration-tests/test_a_task.py b/tasks/integration-tests/test_a_task.py index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..bb56fc0f19fe12154b54a43152dd30a7ec102f8e 100644 --- a/tasks/integration-tests/test_a_task.py +++ b/tasks/integration-tests/test_a_task.py @@ -0,0 +1,2 @@ +class TestATask: + pass diff --git a/tasks/integration-tests/test_direct_simulation.py b/tasks/integration-tests/test_direct_simulation.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/tasks/integration-tests/test_ff_simulation.py b/tasks/integration-tests/test_ff_simulation.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/tasks/src/direct_simulation.py b/tasks/src/direct_simulation.py new file mode 100644 index 0000000000000000000000000000000000000000..f7645d9edb1d8d0cc62280d85841d7d7b28c7664 --- /dev/null +++ b/tasks/src/direct_simulation.py @@ -0,0 +1,5 @@ +""" +This module contains functions for a direct simulation +of gravitational n-body system using realistic data for various +initial configurations +""" \ No newline at end of file diff --git a/tasks/src/ff_simulation.py b/tasks/src/ff_simulation.py new file mode 100644 index 0000000000000000000000000000000000000000..8da049a66d6ed4728d6d1c31cfaf39f0f54bd755 --- /dev/null +++ b/tasks/src/ff_simulation.py @@ -0,0 +1,5 @@ +""" +This module contains functions for a simulation with force field +of gravitational n-body system using realistic data for various +initial configurations +""" \ No newline at end of file