qt: fix no aa for rounded images on wayland

On Wayland, the samples number is 0 by default instead of -1 and custom geometry demands MSAA for anti-aliasing.

Enable custom geometry only when the samples are greater than or equal to 4.

Request review @chub.