# dettato
KobiMusic's **dettato** is an audio to MIDI transcriber. You give it a recording and it writes a MIDI file, with one track for each instrument and a drum track. On nine public benchmark sets it beats MuScriptor 1.3B, which is the next best open model. It scores 59.2 composite F1 and MuScriptor scores 45.1. On the three sets that Mirelo publishes, it also scores a bit higher than Mirelo A2M v1.1 Pro, which is a closed model (60.8 to 56.1). The part I benchmarked is 20M parameters, and MuScriptor is about 65 times bigger. dettato is four models in a chain. The transcriber (15.3M parameters) finds the notes. The evidence repairer (4.9M) fixes them using the transcriber's evidence. The grouper (2.9M) puts the notes into instrument tracks, and the drum model (5.0M) writes the drums. The code and the weights are under Apache-2.0, so commercial use is fine. **No copyrighted recordings.** dettato was trained on a mix of synthetic instrument recordings and CC solo instruments from the URMP dataset. The rest of the instruments morph together around 20 soundfonts, using the [soundfont babies](https://kobi.music/research/making-soundfont-babies) technique I published. The vocals and the other augmentation are CC data too, and the grouper and drum model cards list the CC research datasets they use. No benchmark recording was used for training. **What to expect.** It is hard to know how well it will work for your music from benchmarks alone, so I would try it first. The weakest step is putting the notes into the instrument tracks they belong in. For instance, on jazz recordings a quarter of the saxophone notes can land in the piano track. It also writes performance MIDI. This means the notes keep the timing of the recording, and they are not quantized to a beat. If you want something you can play from, like sheet music, I would recommend [Songscription](https://www.songscription.ai) or [Klangio](https://klang.io). They are both great tools.  The benchmark is the eight sets from Table 2 of the MuScriptor paper, plus the Slakh2100 test set. The eight sets are Bach10, Dagstuhl ChoirSet, PHENICX-Anechoic, RWC Popular, RWC Classical, RWC Genre, RWC Jazz and RWC Royalty-Free. I scored them with the rules from that paper. The composite is the mean of the note onset, note frame, note offset and drum F1. | system | parameters | composite | onset | frame | offset | drums | |---|---|---|---|---|---|---| | **dettato transcriber + drums** | 20.3M | **59.2** (95% 57.9-60.6) | **67.2** | **77.9** | **44.8** | **47.1** | | MuScriptor | 1.3B | 45.1 | 48.6 | 70.2 | 27.8 | 34.0 | Mirelo A2M v1.1 Pro reports three of the sets: PHENICX-Anechoic, RWC Classical and RWC Royalty-Free. On those three, dettato scores 60.8. Mirelo A2M scores 56.1 (Mirelo published that number), MuScriptor scores 47.4 and YourMT3+ scores 45.7.  On the eight MuScriptor sets alone, dettato scores 58.4. MuScriptor scores 44.6 and YourMT3+ scores 41.1, which are the numbers from the paper. The tables for each set are on the [dettato](https://huggingface.co/kobimusic/dettato) and [dettato-drums](https://huggingface.co/kobimusic/dettato-drums) cards. The table scores the notes from the transcriber and the hits from the drum model. I measured the evidence repairer alone (+0.30 composite with the base network of the transcriber in front). The grouper was measured on my own track benchmark, and their cards have the details. I have not scored the four models as one chain yet, the way this package runs them. ## Install ```bash pip install git+https://github.com/kobimusic/dettato dettato song.mp3 -o out/ # -> out/song.mid + out/song.json ``` The weights download from the Hugging Face Hub on first use ([the dettato collection](https://huggingface.co/collections/kobimusic/dettato)).  ## Models Four models, 28.1M parameters in all, collected in the [dettato collection](https://huggingface.co/collections/kobimusic/dettato). Each has its own model card: | stage | model | parameters | what it does | |---|---|---|---| | 1 | [kobimusic/dettato](https://huggingface.co/kobimusic/dettato) | 15.3M | notes from audio: onset, end and pitch on 88 keys, two voices per key | | 2 | [kobimusic/dettato-evidence](https://huggingface.co/kobimusic/dettato-evidence) | 4.9M | repairs the notes using the transcriber's own evidence: drops wrong notes, fixes onsets and ends, adds missed notes | | 3 | [kobimusic/dettato-grouper](https://huggingface.co/kobimusic/dettato-grouper) | 2.9M | puts every note in an instrument track and names the instrument | | 4 | [kobimusic/dettato-drums](https://huggingface.co/kobimusic/dettato-drums) | 5.0M | drum hits on General MIDI drum keys, drum kits and orchestral percussion | ## Use ### Command line ```bash dettato song.mp3 other.wav folder/ -o out/ ``` - Takes any number of files and folders. Folders are searched recursively, and the folder structure is kept under `out/`. - Writes `out/