# Benchmarking PySceneDetect This page contains the results of benchmarking PySceneDetect's detection accuracy againts public shot-boundary-detection datasets. Scoring follows the [TRECVID-SBD][trecvid] convention (greedy 1-to-1 nearest-neighbor matching, with a configurable frame tolerance for hard cuts; point-in-interval matching for fade transitions; mean absolute frame offset on matched events) so numbers are comparable to published SBD results. [trecvid]: https://www-nlpir.nist.gov/projects/tv2007/pastdata/shot_boundary.07.html Supported datasets: - [BBC Planet Earth](https://zenodo.org/records/14865504): 11 long-form broadcast clips; hard cuts only - [AutoShot](https://drive.google.com/file/d/17diRkLlNUUjHDooXdqFUTXYje2-x4Yt6/view?usp=sharing): Short-form web clips; hard cuts only - [ClipShots](https://github.com/Tangshitao/ClipShots): Short-form web clips; hard cuts and typed gradual transitions (fades/dissolves) ## Usage ```bash # Single detector x single dataset: python -m benchmark --detector detect-content --dataset BBC ``` Pass `--help` for `--dataset-root`, `--backend`, `--tolerance`, and `--out` options. ### Parameter sweeps `python -m benchmark.sweep` runs a grid over detector parameters and reports the top cells by F1 plus the Pareto front across tolerances. One decode is shared by up to `--workers` parallel detectors via an internal fan-out wrapper, so the cost scales with `ceil(cells / workers)` decodes per video rather than `cells` decodes. ```bash python -m benchmark.sweep \ --detector detect-content --dataset BBC \ --params "threshold=15:35:1;min_scene_len=0.0:1.0:0.1" \ --tolerance 0,1 --workers 16 \ --out sweep-content-bbc.json ``` `--params`: assignments joined by `;`. Each is either `key=v1,v2,v3` (enumerated values) or `key=start:stop:step` (numeric range, inclusive when `stop` lands on a step). Omitted keys use the detector's default. Time-valued kwargs (`min_scene_len`, etc.) accept `TimecodeLike` - integers are frames, floats are seconds, and strings like `"0.1s"` / `"00:00:00.500"` also work. Prefer floats so the same sweep is comparable across datasets with different framerates. Use `--quick N` to limit to the first N samples for iteration; published numbers should always come from the full corpus. ## Dataset Download ### BBC ```bash # annotations wget -O BBC/fixed.zip https://zenodo.org/records/14873790/files/fixed.zip unzip BBC/fixed.zip -d BBC rm -rf BBC/fixed.zip # videos wget -O BBC/videos.zip https://zenodo.org/records/14873790/files/videos.zip unzip BBC/videos.zip -d BBC rm -rf BBC/videos.zip ``` ### AutoShot Download `AutoShot_test.tar.gz` from [Google Drive](https://drive.google.com/file/d/17diRkLlNUUjHDooXdqFUTXYje2-x4Yt6/view?usp=sharing). ```bash tar -zxvf AutoShot_test.tar.gz rm AutoShot_test.tar.gz ``` ### ClipShots ClipShots is gated behind a dataset request form; direct `wget`-style download links are not published. See [the download instructions](https://github.com/Tangshitao/ClipShots#downloads) to obtain the annotations and videos. The expected on-disk layout is: ``` ClipShots/ annotations/{train,test,only_gradual}.json video_lists/{train,test,only_gradual}.txt videos/*.mp4 ``` The loader defaults to the test split (500 videos). The full corpus is ~46 GB. Set `--dataset-root /path/to/datasets` to override. The default dataset location assumes they are all placed in the benchmark folder (e.g. `benchmark/BBC`, `benchmark/AutoShot`, `benchmark/ClipShots`). ## Results (defaults) Generated by `scripts/benchmark_defaults.sh` at `tolerance=0` (frame-exact matching). Elapsed is mean wall-clock seconds per video. #### BBC | Detector | Recall | Precision | F1 | Mean s/video | |:-----------------:|:------:|:---------:|:-----:|:------------:| | AdaptiveDetector | 87.12 | 96.55 | 91.59 | 36.12 | | ContentDetector | 84.70 | 88.77 | 86.69 | 37.02 | | HashDetector | 92.30 | 75.56 | 83.10 | 25.51 | | HistogramDetector | 89.84 | 72.03 | 79.96 | 22.29 | | ThresholdDetector | 0.06 | 0.70 | 0.11 | 16.05 | #### AutoShot | Detector | Recall | Precision | F1 | Mean s/video | |:-----------------:|:------:|:---------:|:-----:|:------------:| | AdaptiveDetector | 70.59 | 77.46 | 73.86 | 3.52 | | ContentDetector | 63.49 | 76.19 | 69.26 | 4.80 | | HashDetector | 56.48 | 76.11 | 64.84 | 4.14 | | HistogramDetector | 63.27 | 53.23 | 57.82 | 3.76 | | ThresholdDetector | 0.75 | 38.64 | 1.47 | 3.28 | #### ClipShots (hard cuts) | Detector | Recall | Precision | F1 | Mean s/video | |:-----------------:|:------:|:---------:|:-----:|:------------:| | AdaptiveDetector | 85.97 | 41.25 | 55.75 | 1.81 | | ContentDetector | 81.93 | 42.36 | 55.84 | 2.52 | | HashDetector | 81.34 | 30.14 | 43.98 | 1.04 | | HistogramDetector | 72.20 | 11.47 | 19.80 | 0.71 | | ThresholdDetector | 0.08 | 0.58 | 0.14 | 0.64 | #### ClipShots (fades) | Detector | Recall | Precision | F1 | |:-----------------:|:------:|:---------:|:-----:| | AdaptiveDetector | 13.65 | 98.12 | 23.96 | | ContentDetector | 26.03 | 98.04 | 41.14 | | HashDetector | 18.77 | 94.53 | 31.33 | | HistogramDetector | 69.67 | 81.99 | 75.33 | | ThresholdDetector | 5.69 | 99.24 | 10.77 | ## Parameter sweep results The tables above use each detector's v0.7 defaults. A grid sweep over the key parameters scored by hard-cut F1 at 1-frame tolerance, averaged across BBC / AutoShot / ClipShots gives the best single parameter set for this corpus mix: | Detector | Best mean F1 | Best params | v0.7 default | |:-----------------:|:------------:|:-----------------------------------------------------------|:---------------------------------------| | ContentDetector | 73.4 | threshold=31, min_scene_len=0.6s | threshold=27 | | AdaptiveDetector | 76.3 | adaptive_threshold=3.5, window_width=3, min_scene_len=0.6s | adaptive_threshold=3.0, window_width=2 | | HashDetector | 69.8 | threshold=0.35, size=8 | threshold=0.395, size=16 | | HistogramDetector | 66.3 | threshold=0.20, bins=128 | threshold=0.05, bins=256 | | ThresholdDetector | -- | detects fades, not hard cuts (validation only) | threshold=12 | Full per-dataset breakdowns are in [`SWEEP_REPORT.md`](SWEEP_REPORT.md), and can be generated with `python -m benchmark.report_sweep`. The full grids (all detectors and datasets) are driven by `scripts/benchmark_sweep.sh`. ## Citations ### BBC ``` @InProceedings{bbc_dataset, author = {Lorenzo Baraldi and Costantino Grana and Rita Cucchiara}, title = {A Deep Siamese Network for Scene Detection in Broadcast Videos}, booktitle = {Proceedings of the 23rd ACM International Conference on Multimedia}, year = {2015}, } ``` ### AutoShot ``` @InProceedings{autoshot_dataset, author = {Wentao Zhu and Yufang Huang and Xiufeng Xie and Wenxian Liu and Jincan Deng and Debing Zhang and Zhangyang Wang and Ji Liu}, title = {AutoShot: A Short Video Dataset and State-of-the-Art Shot Boundary Detection}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops}, year = {2023}, } ``` ### ClipShots ``` @InProceedings{clipshots_dataset, author = {Shitao Tang and Litong Feng and Zhanghui Kuang and Yimin Chen and Wei Zhang}, title = {Fast Video Shot Transition Localization with Deep Structured Models}, booktitle = {Asian Conference on Computer Vision (ACCV)}, year = {2018}, } ```