[rub.de/19] Assertion failure in `picture_Setup` (for certain raw video streams)

Ref:

• https://oss-fuzz.com/testcase-detail/5073534252023808

Issue from rub.de: bug may be publicly disclosed on Tuesday, April 29, 2025.

Raw video decoding can try to create over-sampled planes (e.g., w->den < w->num), which triggers the assertions in picture_Setup in src/misc/picture.c. This happens for the NV24 and NV42 codecs: In FourCC chroma list (src/misc/fourcc.c), they are described as SEMIPLANAR(1, 1, 8). The first 1 argument makes w->den = 1, which leads to the assertion failure because w->num = 2 is the default for SEMIPLANAR codecs.

Related OSS-Fuzz issue: https://issues.oss-fuzz.com/issues/42509262

xeon ~/work/git/vlc/build-asan $ gdb --args ./test/vlc-demux-dec-run /home/tom/Downloads/reproducer.bin 
GNU gdb (Debian 15.2-1) 15.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test/vlc-demux-dec-run...
(gdb) r
Starting program: /home/tom/work/git/vlc/build-asan/test/vlc-demux-dec-run /home/tom/Downloads/reproducer.bin
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
vlc-demux-dec-run: ../../src/misc/picture.c:184: picture_Setup: Assertion `w->den >= w->num' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, 
    no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
warning: 44	./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0  __pthread_kill_implementation
    (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0)
    at ./nptl/pthread_kill.c:44
#1  0x00007ffff709dcef in __pthread_kill_internal (threadid=<optimized out>, signo=6)
    at ./nptl/pthread_kill.c:78
#2  0x00007ffff7049c42 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff70324f0 in __GI_abort () at ./stdlib/abort.c:79
#4  0x00007ffff7032418 in __assert_fail_base
    (fmt=0x7ffff71b6ca0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7ffff74bf580 "w->den >= w->num", file=file@entry=0x7ffff74bf480 "../../src/misc/picture.c", line=line@entry=184, function=function@entry=0x7ffff74bf9c0 <__PRETTY_FUNCTION__.7> "picture_Setup") at ./assert/assert.c:94
#5  0x00007ffff7042552 in __assert_fail
    (assertion=assertion@entry=0x7ffff74bf580 "w->den >= w->num", file=file@entry=0x7ffff74bf480 "../../src/misc/picture.c", line=line@entry=184, function=function@entry=0x7ffff74bf9c0 <__PRETTY_FUNCTION__.7> "picture_Setup") at ./assert/assert.c:103
#6  0x00007ffff73fd118 in picture_Setup
    (p_picture=p_picture@entry=0x514000004440, fmt=fmt@entry=0x51600000aee8)
    at ../../src/misc/picture.c:184
#7  0x00007ffff73fd514 in picture_InitPrivate
    (p_fmt=p_fmt@entry=0x51600000aee8, priv=priv@entry=0x514000004440, p_resource=p_resource@entry=0x7ffff50500a0) at ../../src/misc/picture.c:216
#8  0x00007ffff73fda5e in picture_NewFromFormat (fmt=fmt@entry=0x51600000aee8)
    at ../../src/misc/picture.c:289
#9  0x00007ffff731868f in decoder_NewPicture (dec=dec@entry=0x51600000ae80)
    at ../../src/input/decoder_helpers.c:163
#10 0x00007ffff13f7405 in DecodeFrame (p_dec=0x51600000ae80, p_block=0x50800003a920)
    at ../../modules/codec/rawvideo.c:251
#11 0x0000555555559326 in test_decoder_process
    (decoder=0x51600000ae80, p_block=<optimized out>, p_block@entry=0x50800003a920)
    at ../../test/src/input/decoder.c:265
#12 0x0000555555557b71 in EsOutSend
    (out=<optimized out>, id=0x511000015e40, block=0x50800003a920)
    at ../../test/src/input/demux-run.c:112
#13 0x00007ffff13874b8 in es_out_Send
    (out=<optimized out>, id=<optimized out>, p_block=<optimized out>)
    at ../../include/vlc_es_out.h:155
#14 Demux (p_demux=<optimized out>) at ../../modules/demux/nsv.c:289
#15 0x00007ffff7319c46 in demux_Demux (demux=demux@entry=0x511000007480)
    at ../../src/input/demux.c:223
#16 0x00005555555583aa in demux_process_stream
    (args=args@entry=0x7ffff4d00020, s=s@entry=0x511000006bc0)
    at ../../test/src/input/demux-run.c:306
#17 0x0000555555558669 in vlc_demux_process_url
    (args=args@entry=0x7ffff4d00020, url=url@entry=0x504000000050 "file:///home/tom/Downloads/reproducer.bin") at ../../test/src/input/demux-run.c:348
#18 0x0000555555558708 in vlc_demux_process_path
    (args=args@entry=0x7ffff4d00020, path=0x7fffffffde73 "/home/tom/Downloads/reproducer.bin") at ../../test/src/input/demux-run.c:362
--Type <RET> for more, q to quit, c to continue without paging--
#19 0x000055555555755a in main (argc=2, argv=0x7fffffffda08)
    at ../../test/vlc-demux-run.c:50

Sample: reproducer.bin