Loaded path playback is out of sync

I forwarded the video to maybe 1/8 of its duration, started tracking and exporting, closed the program.

Opened it again, opened video, re-set the setting (to 5 entities, etc), loaded the tracks.

when I forward the video the approximate location where I started the tracking in the previous run there are no tracks displayed. Only when I rewind to frame 0 paths are displayed (not in sync with whats happening in the video). When I go back and forth between frame 0 and somewhere around the actual start of the sequence paths are drawn but every time a bit different (see screenshot).