Well that is quite annoying to add these each time you need it.
As I searched for the problem (whether I'm the only one facing it), I've found following bug reports:
- ubuntu bug report
- upstream bug report
- arch linux discussion (with possibly working patch)
However, as I'm not willing to build some xfce components myself, I rather searched for a workaround.
Workaround found
After briefly reading the notes in the ubuntu bug report I've found following idea: Adding keyboard layout via Menu -> System -> Settings -> Keyboard (hope the names are exact, as for me it's Menu -> Einstellungen -> Tastatur) solves the problem.
As it seems the settings section saves preferences to permanent store, however keyboard plugin uses some other place (that possibly gets overwritten by the settings one).
