Skip to content
Snippets Groups Projects
Commit 2a9e5cf5 authored by Andi Gerken's avatar Andi Gerken
Browse files

Added docs to be used with pdocs.

parent a5257972
No related branches found
No related tags found
No related merge requests found
Pipeline #37227 passed
......@@ -8,7 +8,7 @@ dist
.coverage
report.xml
htmlcov
docs
html
env
!tests/resources/*.hdf5
......
......@@ -9,59 +9,61 @@
[![pipeline status](https://git.imp.fu-berlin.de/bioroboticslab/robofish/io/badges/master/pipeline.svg)](https://git.imp.fu-berlin.de/bioroboticslab/robofish/io/commits/master)
# Robofish IO
This repository implements an easy to use interface, to create, save, load, and work [specification-compliant](https://git.imp.fu-berlin.de/bioroboticslab/robofish/track_format) hdf5 files, containing 2D swarm data. This repository should be used by the different swarm projects to generate comparable standardized files.
This repository implements an easy to use interface, to create, save, load, and work with [specification-compliant](https://git.imp.fu-berlin.de/bioroboticslab/robofish/track_format) hdf5 files, containing 2D swarm data. This repository should be used by the different swarm projects to generate comparable standardized files.
## Installation
Quick variant:
```
pip3 install robofish-trackviewer robofish-io --extra-index-url https://git.imp.fu-berlin.de/api/v4/projects/6392/packages/pypi/simple
Add our [Artifacts repository](https://git.imp.fu-berlin.de/bioroboticslab/robofish/artifacts) to your pip config and install the packagage.
```bash
python3 -m pip config set global.extra-index-url https://git.imp.fu-berlin.de/api/v4/projects/6392/packages/pypi/simple
python3 -m pip install robofish-io
```
Better variant:
- Follow instructions at [Artifacts repository](https://git.imp.fu-berlin.de/bioroboticslab/robofish/artifacts)
- ```pip3 install robofish-io```
## Usage
We show a simple example below. More examples can be found in ```examples/```
```python
import robofish.io
import numpy as np
# Create a new robofish io file
f = robofish.io.File(world_size_cm=[100, 100], frequency_hz=25.0)
f.attrs["experiment_setup"] = "This is a simple example with made up data."
# Create a new robot entity. Positions and orientations are passed
# separately in this example. Since the orientations have two columns,
# unit vectors are assumed (orientation_x, orientation_y)
# Create a new robot entity with 10 timesteps.
# Positions and orientations are passed separately in this example.
# Since the orientations have two columns, unit vectors are assumed
# (orientation_x, orientation_y)
f.create_entity(
category="robot",
name="robot",
positions=np.zeros((100, 2)),
orientations=np.ones((100, 2)) * [0, 1],
positions=np.zeros((10, 2)),
orientations=np.ones((10, 2)) * [0, 1],
)
# Create a new fish entity.
# Create a new fish entity with 10 timesteps.
# In this case, we pass positions and orientations together (x, y, rad).
# Since it is a 3 column array, orientations in radiants are assumed.
poses = np.zeros((100, 3))
poses[:, 0] = np.arange(-50, 50)
poses[:, 1] = np.arange(-50, 50)
poses[:, 2] = np.arange(0, 2 * np.pi, step=2 * np.pi / 100)
poses = np.zeros((10, 3))
poses[:, 0] = np.arange(-5, 5)
poses[:, 1] = np.arange(-5, 5)
poses[:, 2] = np.arange(0, 2 * np.pi, step=2 * np.pi / 10)
fish = f.create_entity("fish", poses=poses)
fish.attrs["species"] = "My rotating spaghetti fish"
fish.attrs["fish_standard_length_cm"] = 10
# Show and save the file
print(f)
print("Poses Shape: ", f.entity_poses.shape)
# Some possibilities to access the data
print(f"The file:\n{f}")
print(
f"Poses Shape:\t{f.entity_poses_rad.shape}.\t"
+ "Representing(entities, timesteps, pose dimensions (x, y, ori)"
)
print(f"The actions of one Fish, (timesteps, (speed, turn)):\n{fish.speed_turn}")
print(f"Fish poses with calculated orientations:\n{fish.poses_calc_ori_rad}")
f.save_as(path)
# Save the file
f.save_as("example.hdf5")
```
### Evaluation
......
This diff is collapsed.
`robofish.io.file.File` objects are the root of the project. The object contains all information about the environment, entities, and time.
In the simplest form we define a new File with a world size in cm and a frequency in hz. Afterwards, we can save it with a path.
```python
import robofish.io
# Create a new robofish io file
f = robofish.io.File(world_size_cm=[100, 100], frequency_hz=25.0)
f.save_as("test.hdf5")
```
---
The File object can also be generated, with a given path. In this case, we work on the file directly. The `with` block ensures, that the file is validated after the block.
```python
with robofish.io.File(
"test.hdf5", mode="x", world_size_cm=[100, 100], frequency_hz=25.0
) as f:
# Use file f here
```
When opening a file with a path, a mode should be specified to describe how the file should be opened.
| Mode | Description |
|------ |---------------------------------------- |
| r | Readonly, file must exist (default) |
| r+ | Read/write, file must exist |
| w | Create file, truncate if exists |
| x | Create file, fail if exists |
| a | Read/write if exists, create otherwise |
---
Attributes of the file can be added, to describe the contents.
The attributes can be set like this:
```python
f.attrs["experiment_setup"] = "This file comes from the tutorial."
f.attrs["experiment_issues"] = "All data in this file is made up."
```
Any attribute is allowed, but some cannonical attributes are prepared:<br>
`publication_url, video_url, tracking_software_name, tracking_software_version, tracking_software_url, experiment_setup, experiment_issues`
---
All file functions and their documentation can be found at `robofish.io.file.File`.
\ No newline at end of file
testtesttest
\ No newline at end of file
docs/img/calc_ori_speed_turn.png

22.8 KiB

This package provides you:
- Creation, storing, loading, modifying, inspecting of io-files.
- Preprocessing (orientation calculation, action calculation, raycasting, ...)
- Quick and easy evaluation of behavior
- Data, which is interchangable between labs and tools. No conversions required, since units and coordinate systems are standardized.
- No custom data import, but just `include robofish.io`
Features coming up:
- Interface for unified behavior models
- Pytorch Datasets directly from `robofish.io` files.
## Installation
Add our [Artifacts repository](https://git.imp.fu-berlin.de/bioroboticslab/robofish/artifacts) to your pip config and install the packagage.
```bash
python3 -m pip config set global.extra-index-url https://git.imp.fu-berlin.de/api/v4/projects/6392/packages/pypi/simple
python3 -m pip install robofish-io robofish-trackviewer
```
## Usage
This documentation is structured to increase in complexity.
First we'll execute the example from the README. More examples can be found in ```examples/```.
```python
# Create a new robofish io file
f = robofish.io.File(world_size_cm=[100, 100], frequency_hz=25.0)
f.attrs["experiment_setup"] = "This is a simple example with made up data."
# Create a new robot entity with 10 timesteps.
# Positions and orientations are passed separately in this example.
# Since the orientations have two columns, unit vectors are assumed
# (orientation_x, orientation_y)
f.create_entity(
category="robot",
name="robot",
positions=np.zeros((10, 2)),
orientations=np.ones((10, 2)) * [0, 1],
)
# Create a new fish entity with 10 timesteps.
# In this case, we pass positions and orientations together (x, y, rad).
# Since it is a 3 column array, orientations in radiants are assumed.
poses = np.zeros((10, 3))
poses[:, 0] = np.arange(-5, 5)
poses[:, 1] = np.arange(-5, 5)
poses[:, 2] = np.arange(0, 2 * np.pi, step=2 * np.pi / 10)
fish = f.create_entity("fish", poses=poses)
fish.attrs["species"] = "My rotating spaghetti fish"
fish.attrs["fish_standard_length_cm"] = 10
# Some possibilities to access the data
print(f"The file:\n{f}")
print(
f"Poses Shape:\t{f.entity_poses_rad.shape}.\t"
+ "Representing(entities, timesteps, pose dimensions (x, y, ori)"
)
print(f"The actions of one Fish, (timesteps, (speed, turn)):\n{fish.speed_turn}")
print(f"Fish poses with calculated orientations:\n{fish.poses_calc_ori_rad}")
# Save the file
f.save_as("example.hdf5")
```
⚠️ Please try out the example on your computer and read the output.
We created an `robofish.io.file.File` object, then we added two `robofish.io.entity.Entity` objects.
Afterwards we read some properties of the file and printed them *(more info in [Reading Properties](#reading-properties))*.
Lastly, we saved the file to `example.hdf5`.
Congrats, you created your first io file. We'll continue working with it.
---
We can examine the file now by using commandline tools. These are some examples, more details in [Commandline Tools](## Commandline Tools)
```bash
robofish-io-print example.hdf5
```
Checking out the file content.
```bash
robofish-io-evaluate speed example.hdf5
```
Show a histogram of speeds in the file. For more evaluation options check `robofish-io-evaluate --help`
```bash
robofish-trackviewer example.hdf5
```
View a video of the track in an interactive window.
Further details about the commandline tools can be found in `robofish.io.app`.
## Accessing real data
Until now, we only worked with dummy data. Data from different sources is available in the Trackdb. It is currently stored at the FU Box.
⚠️ If you don't have access to the Trackdb yet, please text Andi by Mail or Mattermost (andi.gerken@gmail.com)
## Reading properties
Files and entities have usefull properties to access their content. In this way, positions, orientations, speeds, and turns can be accessed easily.
All shown property functions can be called from a file or on one entity.
The function names are identical but have a `entity_` prefix.
```python
f = robofish.io.File(world_size_cm=[100, 100], frequency_hz=25.0)
nemo = f.create_entity(category='fish', name='nemo', poses=np.zeros((10,3)))
dori = f.create_entity(category='fish', name='dori', poses=np.zeros((10,3)))
# Get the poses of nemo. Resulting in a (10,3) array
print(nemo.poses_rad)
# Get the poses of all entities. Resulting in a (2,10,3) array.
print(f.entity_poses_rad)
```
In the same scheme the following properties are available:
| File/ Entity function | Description |
|--------------------------------- |------------------------------------------------------------------------------------------------------- |
| *entity_*positions | The positions as a (*entities*, timesteps, 2 (x, y)) arary. |
| *entity_*orientations | The orientations as a (*entities*, timesteps, 2 (ori_x, ori_y)) arary. |
| *entity_*orientations_rad | The orientations as a (*entities*, timesteps, 1 (ori_rad)) arary. |
| *entity_*poses | The poses as a (*entities*, timesteps, 4 (x, y, x_ori, y_ori)) array. |
| *entity_*poses_rad | The poses as a (*entities*, timesteps, 3(x, y, ori_rad)) array. |
| *entity_*poses_calc_ori_rad | The poses with calculated orientations as a<br>(*entities*, timesteps - 1, 3 (x, y, calc_ori_rad)) array. |
| *entity_*speed_turn | The speed and turn as a (*entities*, timesteps - 2, 2 (speed_cm/s, turn_rad/s)) array. |
The functions `robofish.io.entity.Entity.poses_calc_ori_rad` and `robofish.io.entity.Entity.speed_turn` are described in detail in `robofish.io.entity`
## Where to continue?
We recommend continuing to read advanced options for `robofish.io.file`s and `robofish.io.entity`s.
Create some files, validate them, look at them in the trackviewer, evaluate them.
If you find bugs or get stuck somewhere, please text `Andi` on Mattermost or by mail (andi.gerken@gmail.com)
\ No newline at end of file
......@@ -7,32 +7,38 @@ def create_example_file(path):
f = robofish.io.File(world_size_cm=[100, 100], frequency_hz=25.0)
f.attrs["experiment_setup"] = "This is a simple example with made up data."
# Create a new robot entity. Positions and orientations are passed
# separately in this example. Since the orientations have two columns,
# unit vectors are assumed (orientation_x, orientation_y)
circle_rad = np.linspace(0, 2 * np.pi, num=100)
# Create a new robot entity with 10 timesteps.
# Positions and orientations are passed separately in this example.
# Since the orientations have two columns, unit vectors are assumed
# (orientation_x, orientation_y)
f.create_entity(
category="robot",
name="robot",
positions=np.stack((np.cos(circle_rad), np.sin(circle_rad))).T * 40,
orientations=np.stack((-np.sin(circle_rad), np.cos(circle_rad))).T,
positions=np.zeros((10, 2)),
orientations=np.ones((10, 2)) * [0, 1],
)
# Create a new fish entity.
# Create a new fish entity with 10 timesteps.
# In this case, we pass positions and orientations together (x, y, rad).
# Since it is a 3 column array, orientations in radiants are assumed.
poses = np.zeros((100, 3))
poses[:, 0] = np.arange(-50, 50)
poses[:, 1] = np.arange(-50, 50)
poses[:, 2] = np.arange(0, 2 * np.pi, step=2 * np.pi / 100)
poses = np.zeros((10, 3))
poses[:, 0] = np.arange(-5, 5)
poses[:, 1] = np.arange(-5, 5)
poses[:, 2] = np.arange(0, 2 * np.pi, step=2 * np.pi / 10)
fish = f.create_entity("fish", poses=poses)
fish.attrs["species"] = "My rotating spaghetti fish"
fish.attrs["fish_standard_length_cm"] = 10
# Show and save the file
print(f)
print("Poses Shape: ", f.entity_poses.shape)
# Some possibilities to access the data
print(f"The file:\n{f}")
print(
f"Poses Shape:\t{f.entity_poses_rad.shape}.\t"
+ "Representing(entities, timesteps, pose dimensions (x, y, ori)"
)
print(f"The actions of one Fish, (timesteps, (speed, turn)):\n{fish.speed_turn}")
print(f"Fish poses with calculated orientations:\n{fish.poses_calc_ori_rad}")
# Save the file
f.save_as(path)
......
......@@ -11,6 +11,7 @@ Functions available to be used in the commandline to evaluate robofish.io files.
import robofish.evaluate
import argparse
import string
def function_dict():
......@@ -41,24 +42,26 @@ def evaluate(args=None):
fdict = function_dict()
longest_name = max([len(k) for k in fdict.keys()])
parser = argparse.ArgumentParser(
description="This function can be called from the commandline to evaluate files.\
Different evaluation methods can be called, which generate graphs from the given files. \
\
With the first argument 'analysis_type', the type of analysis is chosen."
description="This function can be called from the commandline to evaluate files.\n"
+ "Different evaluation methods can be called, which generate graphs from the given files.\n"
+ "With the first argument 'analysis_type', the type of analysis is chosen.",
formatter_class=argparse.RawTextHelpFormatter,
)
parser.add_argument(
"analysis_type",
type=str,
choices=fdict.keys(),
help="The type of analysis.\
speed - A histogram of speeds\
turn - A histogram of angular velocities\
tank_positions - A heatmap of the positions in the tank\
trajectories - A plot of all the trajectories\
follow_iid - A plot of the follow metric in relation to iid (inter individual distance)\
",
help="The type of analysis.\n"
+ "\n".join(
[
f"{key}{' ' * (longest_name - len(key))} - {func.__doc__.splitlines()[0]}"
for key, func in fdict.items()
]
),
)
parser.add_argument(
"paths",
......
# SPDX-License-Identifier: LGPL-3.0-or-later
"""
The Python package `robofish.io` provides a simple interface to create, load, modify, and inspect files containing tracks of swarms.
The files are saved in the `.hdf5` format and following the [track_format specification](https://git.imp.fu-berlin.de/bioroboticslab/robofish/track_format/uploads/f76d86e7a629ca38f472b8f23234dbb4/RoboFish_Track_Format_-_1.0.pdf).
.. include:: ../../../docs/index.md
"""
import sys
import logging
......
"""
.. include:: ../../../docs/entity.md
"""
import robofish.io
import robofish.io.utils as utils
......@@ -137,6 +141,14 @@ class Entity(h5py.Group):
return np.tile([1, 0], (self.positions.shape[0], 1))
return self["orientations"]
@property
def orientations_rad(self):
ori_rad = utils.limit_angle_range(
np.arctan2(self.orientations[:, 1], self.orientations[:, 0]),
_range=(0, 2 * np.pi),
)
return ori_rad[:, np.newaxis]
@property
def poses_calc_ori_rad(self):
# Diff between positions [t - 1, 2]
......@@ -160,12 +172,7 @@ class Entity(h5py.Group):
@property
def poses_rad(self):
poses = self.poses
# calculate the angles from the orientation vectors, write them to the third row and delete the fourth row
ori_rad = utils.limit_angle_range(
np.arctan2(poses[:, 3], poses[:, 2]), _range=(0, 2 * np.pi)
)
return np.concatenate([poses[:, :2], ori_rad[:, np.newaxis]], axis=1)
return np.concatenate([self.positions, self.orientations_rad], axis=1)
@property
def speed_turn(self):
......
# -*- coding: utf-8 -*-
"""
.. include:: ../../../docs/file.md
"""
# -----------------------------------------------------------
# Utils functions for reading, validating and writing hdf5 files according to
# Robofish track format (1.0 Draft 7). The standard is available at
# https://git.imp.fu-berlin.de/bioroboticslab/robofish/track_format
#
# The term track is used to describe a dictionary, describing the track in a dict.
# To distinguish between attributes, dictionaries and groups, a prefix is used
# (a_ for attribute, d_ for dictionary, and g_ for groups).
#
# Dec 2020 Andreas Gerken, Berlin, Germany
# Released under GNU 3.0 License
......@@ -325,7 +326,6 @@ class File(h5py.File):
assert poses.ndim == 3
assert poses.shape[2] in [3, 4]
agents = poses.shape[0]
timesteps = poses.shape[1]
entity_names = []
for i in range(agents):
......@@ -361,9 +361,23 @@ class File(h5py.File):
for name in self.entity_names
]
@property
def entity_positions(self):
return self.select_entity_property(None, entity_property=Entity.positions)
@property
def entity_orientations(self):
return self.select_entity_property(None, entity_property=Entity.orientations)
@property
def entity_orientations_rad(self):
return self.select_entity_property(
None, entity_property=Entity.orientations_rad
)
@property
def entity_poses(self):
return self.select_entity_property(None)
return self.select_entity_property(None, entity_property=Entity.poses)
@property
def entity_poses_rad(self):
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment