diff --git a/README b/README deleted file mode 100644 index b2ba4aaa3a71046653599aa0b3798b211a2c0d30..0000000000000000000000000000000000000000 --- a/README +++ /dev/null @@ -1,18 +0,0 @@ -Linux kernel -============ - -This file was moved to Documentation/admin-guide/README.rst - -Please notice that there are several guides for kernel developers and users. -These guides can be rendered in a number of formats, like HTML and PDF. - -In order to build the documentation, use ``make htmldocs`` or -``make pdfdocs``. - -There are various text files in the Documentation/ subdirectory, -several of them using the Restructured Text markup notation. -See Documentation/00-INDEX for a list of what is contained in each file. - -Please read the Documentation/process/changes.rst file, as it contains the -requirements for building and running the kernel, and information about -the problems which may result by upgrading your kernel. diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..317940abf1d7e444c6811e2a59431adeb2bc39b9 --- /dev/null +++ b/README.md @@ -0,0 +1,118 @@ +This is the code repository for the prototype of the Plan-Based Scheduler proposed by the master's thesis "Handling of Prediction Errors in a Plan-Based Scheduler on Linux-HPC-Nodes". + +# Setup + +The following setup is described for Fedora Linux, but should be similar for other Linux distributions, maybe except for some renaming of corresponding packages. + +1. Make sure virtualization for your CPU is enabled in the BIOS/UEFI (e.g. for AMD CPUs AMD IOMMU and/or SVM need to be enabled)! +2. Install virtualization environment +```shell +sudo dnf install -y qemu qemu-system-x86 qemu-kvm +``` +3. Create the system image to be used by QEMU +```shell +cd pb_utils +./create_image.sh +``` +4. Chroot into the system image +```shell +./chroot_image.sh +``` +5. Inside the chroot environment, install some dependencies, compile the kernel and finally exit the chroot environment +```shell +cd mnt/ +./setup_build.sh +./build_kernel.sh ; ./build_pb_submitter.sh +exit +``` +6. Now execute the image in QEMU to boot the test system (to exit use <kbd>ctrl</kbd> + <kbd>c</kbd>) +```shell +./run_qemu.sh +``` + +# Running/testing + +Once booted into the test system, the following command can be used to create a prediction model using the test job, and directly test another instance of a test job against that model: +```shell +tmp=$(./measure ./test_prog) ; sleep 1 ; echo $tmp | ./pb_submitter ./test_prog +``` +The `measure` program returns the PID that is used to access the corresponding prediction model by the `pb_submitter` program. +The sleep is necessary to make sure that the exit of all processes is completed and that postprocessing of the model is finished (due to that happening in the kernel which the user has no information of). + +This can be trivially expanded to create one prediction model and then test it multiple times (e.g. 100) sequentially against the runtime behavior of a similar job instance, e.g. to evaluate deviation distributions: +```shell +tmp=$(./measure ./test_prog) ; sleep 1 ; for i in $(seq 1 100); do echo $i; echo $tmp | ./pb_submitter ./test_prog; sleep 1 ; done +``` + +## Generating deviation distribution plots + +This chapter describes the creation of the deviation data and the corresponding plots shown in the *Evaluation*-chapter of the thesis, so that the results can be reproduced. + +Instead of using the prediction models generated in the kernel (which would require the implementation to be expanded to write to a user-accessable log file), the logging messages can also be read directly from the STDOUT of the console. + +*Yes, i know this is rather ugly but it works and requires less additional work!* + +1. The STDOUT of the QEMU terminal running the test system image can be captured: +```shell +cd pb_utils +./run_qemu.sh | tee -a log.txt +``` +2. Now run the tests as previously described, which prints the deviations to STDOUT (which now gets written to `log.txt`): +```shell +tmp=$(./measure ./test_prog) ; sleep 1 ; for i in $(seq 1 100); do echo $i; echo $tmp | ./pb_submitter ./test_prog; sleep 1 ; done +``` +3. Now the output must be filtered for the relevant lines which include the instruction counts. `if ($10 ~ /^SHORT/)` checks if the tenth parameter begins with the string `SHORT` which is necessary since it has a suffix (due to the terminal coloring) (see [idea](https://unix.stackexchange.com/questions/72734/awk-print-line-only-if-the-first-field-start-with-string-as-linux1/72763#72763)) (show relevant columns for testing purposes: `grep 'PB TASK' log.txt | awk '{print $7, $10}'`): +```shell +grep 'PB TASK' log.txt | awk '{if ($12 ~ /^SHORT/) {print -$7, $9} else if ($12 ~ /^LONG/) {print $7, $9} else {print "ERROR"}}' > counts.txt +``` +4. Plot the graphs using pyplot: + +Absolute deviations: +```shell +python3 -c "import matplotlib.pyplot as plt +f = open('counts.txt') +x = [int(line.split()[0]) for line in f] +f.close() +plt.hist(x, bins=100) +plt.xlim(xmin=-8000, xmax=8000) +plt.xlabel('absolute deviation [number of instructions]') +plt.ylabel('number of occurances') +plt.grid(axis='y') +plt.gcf().set_size_inches(3.5, 2.5) +plt.tight_layout() +plt.savefig('deviation_absolute.svg', format='svg')" +``` + +Relative deviations (for the runs with 1M instructions, the bins are set to 10 instead of 100 to enhance bar visibility): +```shell +python3 -c "import matplotlib.pyplot as plt +f = open('counts.txt') +x = [(int(line.split()[0]) / int(line.split()[1])) * 100.0 for line in f] +f.close() +plt.hist(x, bins=100) +plt.xlim(xmin=-2, xmax=2) +plt.xlabel('relative deviation [%]') +plt.ylabel('number of occurances') +plt.grid(axis='y') +plt.gcf().set_size_inches(3.5, 2.5) +plt.tight_layout() +plt.savefig('deviation_relative.svg', format='svg')" +``` + +# Linux kernel README + +This file was moved to Documentation/admin-guide/README.rst + +Please notice that there are several guides for kernel developers and users. +These guides can be rendered in a number of formats, like HTML and PDF. + +In order to build the documentation, use ``make htmldocs`` or +``make pdfdocs``. + +There are various text files in the Documentation/ subdirectory, +several of them using the Restructured Text markup notation. +See Documentation/00-INDEX for a list of what is contained in each file. + +Please read the Documentation/process/changes.rst file, as it contains the +requirements for building and running the kernel, and information about +the problems which may result by upgrading your kernel