# Sample Times in Fragmented MP4 This document explains how `MoofParser` reads a fragmented MP4 file and gives each sample a decode time and a composition range. It also gives the reason for each step. The examples use small numbers. ## Plain and fragmented MP4 A plain MP4 file keeps one sample table for each track in `moov/stbl`. `MP4Metadata` reads that table through mp4parse. The `MP4SampleIndex` constructor then copies the samples into `MP4SampleIndex::mIndex`. A fragmented MP4 file splits the samples into fragments. Each fragment is a `moof` box and an `mdat` box. The `moof` box has a `traf` box for each track. The `traf` box has a `tfdt` box and one or more `trun` boxes. When the `moov` box has no samples, `MP4SampleIndex` creates a `MoofParser`. The parser makes one `Moof` object for each `moof` box, and each `Moof` has its own `mIndex`. | File type | Where the samples are | Index | | ---------- | --------------------- | ----------------------------- | | Plain | `moov/stbl` | `MP4SampleIndex::mIndex` | | Fragmented | `trun` in each `moof` | `Moof::mIndex`, one per moof | MSE must use fragmented MP4. In the ISO BMFF byte stream format for MSE, a media segment is a `moof` box and its `mdat` box. ## Where the parser runs For file playback, the call chain is: ``` MP4TrackDemuxer::EnsureUpToDateIndex MP4SampleIndex::UpdateMoofIndex MoofParser::RebuildFragmentedIndex Moof::Moof one call for each moof box Moof::ParseTraf Moof::ParseTrun ``` `SampleIterator::GetNext` then makes a `MediaRawData` from each sample: | `MediaRawData` field | Value from the sample | | -------------------- | ------------------------------ | | `mTime` | start of `mCompositionRange` | | `mDuration` | length of `mCompositionRange` | | `mTimecode` | `mDecodeTime` | | `mKeyframe` | `mSync` | For MSE, `TrackBuffersManager::CreateDemuxerforMIMEType` creates an `MP4Demuxer` on the appended data. The demuxer uses the same path. Then `TrackBuffersManager::ProcessFrames` runs the MSE coded frame processing on the samples. ## Step 1: read each trun entry `Moof::ParseTraf` starts the decode time at the `tfdt` value. If the `traf` box has no valid `tfdt`, it continues from the decode end of the previous moof. `Moof::ParseTrun` then reads one entry for each sample. For each entry, it does these steps: 1. It sets the decode time (DTS) of the sample to the current decode time. 2. It sets the composition range to `[DTS + ctsOffset, DTS + ctsOffset + duration)`. 3. It adds the entry duration to the current decode time. The edit list moves both times by the same amount. The examples have no edit list. The `trun` duration is a decode delta: the time from this DTS to the next DTS. It is not the time that the frame stays on the screen. ## Step 2: decode order and display order The parser stores the samples in decode order. B-frames make the display order different from the decode order. The `ctsOffset` of each entry moves its composition time (PTS) away from its DTS. For example, a 30 fps H.264 stream has two B-frames between its I- and P-frames. Its timescale is 15360, so one frame is 512 ticks. The encoder puts three samples in each moof. The table shows the first two moofs in decode order. | moof | type | DTS | PTS | | ---- | ---- | ---- | ---- | | 1 | I | 0 | 512 | | 1 | P | 512 | 2048 | | 1 | B | 1024 | 1024 | | 2 | B | 1536 | 1536 | | 2 | P | 2048 | 3584 | | 2 | B | 2560 | 2560 | In display order, moof 1 has the frames at 512, 1024 and 2048. The frame at 1536 is in moof 2. So moof 1 has a gap in its composition times, and the next moof fills it. ``` PTS 512 1024 1536 2048 2560 |-----|-----|-----|-----| moof 1 [ I ] [ B ] [ P ] moof 2 [ B ] [ B ] ... ``` ## Step 3: close the composition gaps After `ParseTrun`, each sample has the composition range from its `trun` duration. But the `trun` duration is a decode delta, so it can be different from the display spacing. For example, a stream has a timescale of 90000, so one frame is 3000 ticks. In decode order, its first `trun` duration is 3000 ticks. Then the durations alternate between 1 and 5999 ticks, and one of them is 6149. But its composition times are about 3000 ticks apart. With the `trun` durations only, the sample at 6000 has a duration of 1 tick, and a gap of one frame follows it. So `Moof::Moof` sorts the samples of the moof by composition start. Then it sets the end of each sample to a new value. ### Why not end at the next start A simple rule sets each end to the start of the next sample in the moof. This closes every gap in the moof. But it also closes a gap that the next moof fills. In the B-frame example, the frame at 1024 gets the end 2048, and it covers the frame at 1536 in moof 2. | PTS | `trun` end | simple rule end | bounded rule end | | ---- | ---------- | --------------- | ---------------- | | 512 | 1024 | 1024 | 1024 | | 1024 | 1536 | 2048 | 1536 | | 2048 | 2560 | 2560 | 2560 | The video sink shows a frame until the clock passes its end. So the frame at 1024 stays on the screen too long. Then the sink drops the frames that start inside its span. ### The bounded rule `Moof::Moof` limits each end: ``` end = min(start of the next sample, start + longest duration in this moof) ``` The longest `trun` duration in the moof is the bound. In the B-frame example, every duration is 512. So the frame at 1024 ends at 1536. In the example with uneven durations, the longest duration is 6149. The sample at 6000 has a duration of 1 tick, and the next sample starts at 9001. The gap is smaller than the bound, so the sample ends at 9001. ### Fallbacks The rule uses the start of the next sample as the end in two cases: | Case | Why | | ------------------------------------- | -------------------------------- | | The longest duration is 0 | A zero bound gives empty samples | | `start + longest` is too large | The sum is not a valid time | ### Known limit One long sample raises the bound for all the samples in its moof. Suppose that in the B-frame example, the frame at 1024 in moof 1 has a duration of 7680 ticks. The bound becomes 7680, so the gap does not limit the end, and in moof 1 alone the frame at 1024 ends at 2048 again. The next section shows how the next moof ends it at 1536: | PTS | end | | ---- | ---- | | 512 | 1024 | | 2048 | 2560 | | 1024 | 2048 | ### Ends across moofs ISO BMFF stores no display duration. A sample duration is the step to the next decode time (ISO/IEC 14496-12, 8.6.1.2), and a sample has a composition time (8.6.1.3) but no composition duration; only the composition duration of the last sample is called unclear (8.6.1.4). So a sample is shown until the next composition time in the track, which can be in a later moof. For example, in variable frame rate content, the next frame can come after a long display gap, in the next moof. Suppose that every sample duration is 512 ticks, and that after the frame at 2048, the next frame comes at 17920, in moof 2. Moof 1 alone gives the frame at 2048 its own duration, so it ends at 2560: ``` PTS 512 1024 1536 2048 2560 17920 18432 moof 1 [ I ] [ B ] [ P ] moof 2 [ B ] [ P ] ``` When moof 2 is added, the next composition time is known, and the frame at 2048 is extended to it: ``` PTS 512 1024 1536 2048 17920 18432 moof 1 [ I ] [ B ] [ P ......................] moof 2 [ B ] [ P ] ``` A sample can also come in between later. For example, with one frame per moof and B-frames, the P-frame comes before the B-frames that are shown before it. Moof 2 extends the I-frame to the P-frame, and then the B-frames of moofs 3 and 4 start inside the I-frame: ``` PTS 512 1024 1536 2048 moof 1 [ I .............] moof 2 [ P ] moof 3 [ B ] moof 4 [ B ] ``` So when a new sample starts inside an earlier sample, and the earlier sample ends at the start of the next earlier sample, that end came from the next sample, and the earlier sample is ended at the new sample. The new samples are then extended up to that next earlier sample, so the time stays covered: ``` PTS 512 1024 1536 2048 moof 1 [ I ] moof 2 [ P ] moof 3 [ B ] moof 4 [ B ] ``` An earlier sample that ends at no sample start keeps its own duration. Only samples next to the moof boundary in composition order can change. The search covers up to 16 samples before the new moof, the largest decoded picture buffer of H.264 and HEVC. A longer run of non-reference frames can put the next frame further back, and then the search misses it. ## Step 4: rewrite the decode times MSE coded frame processing checks each DTS. `TrackBuffersManager::ProcessFrames` finds a discontinuity in two cases: - The DTS is smaller than the last DTS. - The DTS is more than 2 * `mLongestFrameDuration` after the last DTS. After a discontinuity, the track needs a random access point. So `ProcessFrames` drops each frame until the next keyframe. The DTS from `ParseTrun` steps by the `trun` durations. But Step 3 changed the durations, so the two can be different. So `Moof::Moof` rewrites the DTS of each sample in the moof. It keeps the decode span of the moof, from the first DTS to the decode end of the `trun` entries. It spreads that span over the samples in proportion to their durations: ``` adjust = decode span / sum of the sample durations DTS(n) = first DTS + (sum of the durations before sample n) * adjust ``` The last DTS is then never above the decode end of the `trun` entries. That end is the start of the next moof only when the next `tfdt` matches the `trun` durations. ### Normal case In moof 1 of the B-frame example, the decode span is 3 * 512 = 1536. The sum of the durations is also 1536. So `adjust` is 1.0, and the DTS values stay 0, 512 and 1024. The next moof starts at 1536. ### Uneven case In the example with uneven durations, a moof has a decode span of 69150 ticks. After Step 3, the sum of the durations is 72150 ticks. So `adjust` is 69150 / 72150, about 0.958. The first samples get these values: | PTS | end | DTS | | ---- | ---- | ---- | | 0 | 3001 | 0 | | 6000 | 9001 | 2876 | | 3001 | 6000 | 5752 | ### Why the sum and not the composition span The composition span of the moof could also be the denominator. But the span includes the gaps that the next moof fills. In moof 1 of the B-frame example, the span is 2560 - 512 = 2048. The sum of the durations is 1536. | Denominator | Value | `adjust` | DTS values | Step into moof 2 | | ---------------- | ----- | -------- | ------------ | ---------------- | | Composition span | 2048 | 0.75 | 0, 384, 768 | 768 | | Sum of durations | 1536 | 1.0 | 0, 512, 1024 | 512 | With the span, the DTS steps do not follow the sample durations. ## Step 5: the moof time range `Moof::mTimeRange` is the composition span of the moof: from the first composition start to the last composition end. It includes the gaps that the next moof fills. `Moof::FixRounding` moves its end to the start of the next moof when the gap between them is small. The limit is 35 ms plus one tick for each sample. These functions use `mTimeRange`: | Function | Use | | ----------------------------------------------- | ----------------------- | | `MP4SampleIndex::ConvertByteRangesToTimeRanges` | buffered ranges | | `MP4SampleIndex::GetEvictionOffset` | data that it can remove | | `MoofParser::GetCompositionRange` | time span of the data |