Skip to content

d3d_dynamic_shader: Fix uses of out of bounds values in MultMat43

Martin Storsjö requested to merge mstorsjo/vlc:d3d-multmat into master

Since 6ab96386, the matrices in the function are only 4x3, but the function kept using the fourth row of the right hand matrix.

The out of bounds values were in practice multiplied with 0, which is why this wasn't initially noticed. However, 0 * can also evaluate to NaN, if the uninitialized data happens to make the bit pattern of a NaN.

In the upcoming Clang 14, the default options for float handling were changed slightly (in Clang commit https://github.com/llvm/llvm-project/commit/f04e387055e495e3e14570087d68e93593cf2918).

As Clang can see that these multiplications use undefined values out of bounds, it assumes that those values are NaNs, which let Clang essentially optimize out most of D3D_SetupQuad, just setting all of quad->shaderConstants->Colorspace to NaNs.

This fixes the colorspace matrix when built with Clang 14, by assuming the fourth, missing, row of the right hand matrix in MultMat43 to be [0, 0, 0, 1].

Merge request reports