Low Frame Rate on MacOS from version 3.0.18 when playing certain H264 videos

I spent several hours debugging the issue already reported in the forums Some H264 files play with very low frame rate (3-4pfs) on Mac OS, both Intel and arm64, starting from release 3.0.18.

3.0.17 and earlier are unaffected.

I managed to recreate a local build environment and used git bisect to hunt down the offending commit, which turns out to be:

commit ae37b995 Author: Francois Cartegnie fcvlcdev@free.fr Date: Thu Feb 4 21:09:21 2021 +0100

demux: mp4: only have unsigned pts offsets

since dts_shift is always computed, that should be
always positive.

(cherry picked from commit 5132db8ad6840d04fbca1a0ea23db5cd1e5ac0a6)
Signed-off-by: Steve Lhomme <robux4@ycbcr.xyz>

By adding debug statements, I can see what is happening:

the commit above assumes ck->p_sample_offset_pts[i] to never be negative, but while playing the sample file it happens to be negative in the last branch below:

                if ( i_current_index_samples_left )
                {
                    if ( i_current_index_samples_left > i_sample_count )
                    {
                        ck->p_sample_count_pts[i] = i_sample_count;
                        ck->p_sample_offset_pts[i] = i_ctsdelta;
                        i_current_index_samples_left -= i_sample_count;
                        i_sample_count = 0;
                        assert( i == ck->i_entries_pts - 1 );
                        break;
                    }
                    else
                    {
                        ck->p_sample_count_pts[i] = i_current_index_samples_left;
                        ck->p_sample_offset_pts[i] = i_ctsdelta;
                        i_sample_count -= i_current_index_samples_left;
                        i_current_index_samples_left = 0;
                        i_index++;
                    }
                }
                else
                {
                    if ( ctts->pi_sample_count[i_index] > i_sample_count )
                    {
                        ck->p_sample_count_pts[i] = i_sample_count;
                        ck->p_sample_offset_pts[i] = i_ctsdelta;
                        i_current_index_samples_left = ctts->pi_sample_count[i_index] - i_sample_count;
                        i_sample_count = 0;
                        assert( i == ck->i_entries_pts - 1 );
                        // keep building from same index
                    }
                    else
                    {
                        ck->p_sample_count_pts[i] = ctts->pi_sample_count[i_index];
                        ck->p_sample_offset_pts[i] = i_ctsdelta; // this can be negative: pi_sample_offset[i_index]: -512 i_cts_shift: 0
                        i_sample_count -= ctts->pi_sample_count[i_index];
                        i_index++;
                    }
                }

Applying the patch below on top of 3.0.20 solves the issue for me:

diff --git a/modules/demux/mp4/mp4.c b/modules/demux/mp4/mp4.c
index 8290f92c1e..f15b07ea9f 100644
--- a/modules/demux/mp4/mp4.c
+++ b/modules/demux/mp4/mp4.c
@@ -2730,8 +2730,6 @@ static int TrackCreateSamplesIndex( demux_t *p_demux,
             for( uint32_t i = 0; i < ck->i_entries_pts; i++ )
             {
                 int64_t i_ctsdelta = ctts->pi_sample_offset[i_index] + i_cts_shift;
-                if( i_ctsdelta < 0 ) /* should not */
-                    i_ctsdelta = 0;
                 if ( i_current_index_samples_left )
                 {
                     if ( i_current_index_samples_left > i_sample_count )
diff --git a/modules/demux/mp4/mp4.h b/modules/demux/mp4/mp4.h
index b7c6cdf3d7..d587b04a58 100644
--- a/modules/demux/mp4/mp4.h
+++ b/modules/demux/mp4/mp4.h
@@ -54,7 +54,7 @@ typedef struct

     uint32_t     i_entries_pts;
     uint32_t     *p_sample_count_pts;
-    uint32_t     *p_sample_offset_pts;  /* pts-dts */
+    int32_t      *p_sample_offset_pts;  /* pts-dts */
     uint32_t     small_pts_buf[4];

     /* TODO if needed add pts

Disabling HW Accel for Videotoolbox makes the video play smoothly, with high CPU usage.

VLC 4.0 Nightly build (Feb 5, 2024) seems unaffected (but I don't see videotoolbox there, maybe using another codec?)