libplacebo: display: fix variable type

pl-output-hint is an integer variable, and it was leading to warnings when compiling:

../../modules/video_output/libplacebo/display.c: In function 'UpdateColorspaceHint':
../../modules/video_output/libplacebo/display.c:499:5: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
  499 |     case OUTPUT_HDR10:
      |     ^~~~
../../modules/video_output/libplacebo/display.c:503:5: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
  503 |     case OUTPUT_HLG:
      |     ^~~~
../../modules/video_output/libplacebo/display.c:477:5: warning: switch condition has boolean value [-Wswitch-bool]
  477 |     switch (var_InheritBool(vd, "pl-output-hint")) {
      |     ^~~~~~