Reading an MKV/MKA file with Opus skips the beginning

When I read an MKV or MKA file that contains Opus audio, the first ~2 seconds are silent. Then the rest of the file is read normally.

I'm attaching 2 files, containing the same Opus audio track. The difference is the container (Ogg for one, Matroska for the other). The Ogg one works fine, the Matroska one skips a couple of seconds at the start (but it reads fine in MPC-HC or Audacity, for instance).

This issue has been occurring ever since I first tried to read an MKV file with Opus audio, on multiple hardware. I'm currently using VLC 3.0.11 on Windows 10 x64, but I remember seeing this on Win 7 too. I believe I saw it on Linux as well, but I don't have a machine where I could test this currently. I also gave a try to a recent VLC 4.0 nightly, the issue is still here.

Maybe this is something related to bug #17899?