qt: make some popups non-modal and fix popups do not close on press outside with `CompositorX11`
I don't see a reason why tracks or tool button popups should be modal. It seems that they were made modal because of a Qt 5 bug, this is no longer relevant to us.
Request review @chub.