Skip to content
Snippets Groups Projects
Commit ba6b0256 authored by FKHals's avatar FKHals
Browse files

Add PBS specific README.md

which still includes the default Linux README
parent 5e18b3ba
No related tags found
No related merge requests found
Linux kernel
============
This file was moved to Documentation/admin-guide/README.rst
Please notice that there are several guides for kernel developers and users.
These guides can be rendered in a number of formats, like HTML and PDF.
In order to build the documentation, use ``make htmldocs`` or
``make pdfdocs``.
There are various text files in the Documentation/ subdirectory,
several of them using the Restructured Text markup notation.
See Documentation/00-INDEX for a list of what is contained in each file.
Please read the Documentation/process/changes.rst file, as it contains the
requirements for building and running the kernel, and information about
the problems which may result by upgrading your kernel.
README.md 0 → 100644
This is the code repository for the prototype of the Plan-Based Scheduler proposed by the master's thesis "Handling of Prediction Errors in a Plan-Based Scheduler on Linux-HPC-Nodes".
# Setup
The following setup is described for Fedora Linux, but should be similar for other Linux distributions, maybe except for some renaming of corresponding packages.
1. Make sure virtualization for your CPU is enabled in the BIOS/UEFI (e.g. for AMD CPUs AMD IOMMU and/or SVM need to be enabled)!
2. Install virtualization environment
```shell
sudo dnf install -y qemu qemu-system-x86 qemu-kvm
```
3. Create the system image to be used by QEMU
```shell
cd pb_utils
./create_image.sh
```
4. Chroot into the system image
```shell
./chroot_image.sh
```
5. Inside the chroot environment, install some dependencies, compile the kernel and finally exit the chroot environment
```shell
cd mnt/
./setup_build.sh
./build_kernel.sh ; ./build_pb_submitter.sh
exit
```
6. Now execute the image in QEMU to boot the test system (to exit use <kbd>ctrl</kbd> + <kbd>c</kbd>)
```shell
./run_qemu.sh
```
# Running/testing
Once booted into the test system, the following command can be used to create a prediction model using the test job, and directly test another instance of a test job against that model:
```shell
tmp=$(./measure ./test_prog) ; sleep 1 ; echo $tmp | ./pb_submitter ./test_prog
```
The `measure` program returns the PID that is used to access the corresponding prediction model by the `pb_submitter` program.
The sleep is necessary to make sure that the exit of all processes is completed and that postprocessing of the model is finished (due to that happening in the kernel which the user has no information of).
This can be trivially expanded to create one prediction model and then test it multiple times (e.g. 100) sequentially against the runtime behavior of a similar job instance, e.g. to evaluate deviation distributions:
```shell
tmp=$(./measure ./test_prog) ; sleep 1 ; for i in $(seq 1 100); do echo $i; echo $tmp | ./pb_submitter ./test_prog; sleep 1 ; done
```
## Generating deviation distribution plots
This chapter describes the creation of the deviation data and the corresponding plots shown in the *Evaluation*-chapter of the thesis, so that the results can be reproduced.
Instead of using the prediction models generated in the kernel (which would require the implementation to be expanded to write to a user-accessable log file), the logging messages can also be read directly from the STDOUT of the console.
*Yes, i know this is rather ugly but it works and requires less additional work!*
1. The STDOUT of the QEMU terminal running the test system image can be captured:
```shell
cd pb_utils
./run_qemu.sh | tee -a log.txt
```
2. Now run the tests as previously described, which prints the deviations to STDOUT (which now gets written to `log.txt`):
```shell
tmp=$(./measure ./test_prog) ; sleep 1 ; for i in $(seq 1 100); do echo $i; echo $tmp | ./pb_submitter ./test_prog; sleep 1 ; done
```
3. Now the output must be filtered for the relevant lines which include the instruction counts. `if ($10 ~ /^SHORT/)` checks if the tenth parameter begins with the string `SHORT` which is necessary since it has a suffix (due to the terminal coloring) (see [idea](https://unix.stackexchange.com/questions/72734/awk-print-line-only-if-the-first-field-start-with-string-as-linux1/72763#72763)) (show relevant columns for testing purposes: `grep 'PB TASK' log.txt | awk '{print $7, $10}'`):
```shell
grep 'PB TASK' log.txt | awk '{if ($12 ~ /^SHORT/) {print -$7, $9} else if ($12 ~ /^LONG/) {print $7, $9} else {print "ERROR"}}' > counts.txt
```
4. Plot the graphs using pyplot:
Absolute deviations:
```shell
python3 -c "import matplotlib.pyplot as plt
f = open('counts.txt')
x = [int(line.split()[0]) for line in f]
f.close()
plt.hist(x, bins=100)
plt.xlim(xmin=-8000, xmax=8000)
plt.xlabel('absolute deviation [number of instructions]')
plt.ylabel('number of occurances')
plt.grid(axis='y')
plt.gcf().set_size_inches(3.5, 2.5)
plt.tight_layout()
plt.savefig('deviation_absolute.svg', format='svg')"
```
Relative deviations (for the runs with 1M instructions, the bins are set to 10 instead of 100 to enhance bar visibility):
```shell
python3 -c "import matplotlib.pyplot as plt
f = open('counts.txt')
x = [(int(line.split()[0]) / int(line.split()[1])) * 100.0 for line in f]
f.close()
plt.hist(x, bins=100)
plt.xlim(xmin=-2, xmax=2)
plt.xlabel('relative deviation [%]')
plt.ylabel('number of occurances')
plt.grid(axis='y')
plt.gcf().set_size_inches(3.5, 2.5)
plt.tight_layout()
plt.savefig('deviation_relative.svg', format='svg')"
```
# Linux kernel README
This file was moved to Documentation/admin-guide/README.rst
Please notice that there are several guides for kernel developers and users.
These guides can be rendered in a number of formats, like HTML and PDF.
In order to build the documentation, use ``make htmldocs`` or
``make pdfdocs``.
There are various text files in the Documentation/ subdirectory,
several of them using the Restructured Text markup notation.
See Documentation/00-INDEX for a list of what is contained in each file.
Please read the Documentation/process/changes.rst file, as it contains the
requirements for building and running the kernel, and information about
the problems which may result by upgrading your kernel
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment