# Audio and Speech

Speech, music and environmental sounds vary in timing, speaker, source and recording conditions. This review covers learning to align, recognize, represent, generate and compress audio. Generic network architectures belong in [Neural Networks](https://mlpapers.org/neural-nets/), and joint audio-text objectives in [Multi-View Learning](https://mlpapers.org/multiview-learning/#cross-modal-alignment-and-fusion).

## Alignment and speech recognition

- **Dynamic time warping** - Aligns sequences by optimizing a nonlinear time-warping path with dynamic programming. Sakoe-Chiba studies path constraints for spoken-word matching; this is an alignment method, not a forecasting model. [Sakoe and Chiba 1978](<https://doi.org/10.1109/TASSP.1978.1163055> "Dynamic programming algorithm optimization for spoken word recognition; IEEE Trans. Acoust., Speech, Signal Process.").
- **CTC** - Trains sequence recognition without frame-level alignments by summing paths through labels and a blank symbol. Frame output probabilities factor independently given the input sequence. [Graves et al. 2006](<https://www.cs.toronto.edu/~graves/icml_2006.pdf> "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks; ICML").
    - **RNN Transducer** - Adds a prediction network conditioned on previous output labels to an acoustic transcriber. Summing over monotonic alignments allows joint acoustic and label-sequence modelling without a supplied frame-level alignment. [Graves 2012](<https://arxiv.org/abs/1211.3711v1> "Sequence Transduction with Recurrent Neural Networks; arXiv").
- **Whisper** - Trains on 680,000 hours of multilingual audio and web transcripts for recognition and translation. Zero-shot tests across speech datasets show broader transfer than models tuned to a single benchmark, despite similar LibriSpeech accuracy. [Radford et al. 2023](<https://proceedings.mlr.press/v202/radford23a.html> "Robust Speech Recognition via Large-Scale Weak Supervision; ICML").
- **Hidden-state recognition** - See [Modeling](https://mlpapers.org/models/#state-space-and-sequence-models) for hidden Markov models and their inference problems.

## Self-supervised acoustic representations

- **wav2vec 2.0** - Masks latent speech representations and predicts jointly learned quantized targets with a contrastive loss. Unlabeled audio pretraining precedes supervised speech-recognition fine-tuning. [Baevski et al. 2020](<https://arxiv.org/abs/2006.11477> "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations; NeurIPS").
- **HuBERT** - Predicts offline cluster labels for masked speech segments, then reclusters learned features. Consistent targets matter more than accurate phonetic labels. Pretraining matches or improves wav2vec 2.0 across tested LibriSpeech label budgets. [Hsu et al. 2021](<https://arxiv.org/abs/2106.07447v1> "HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units; IEEE/ACM TASLP").
- **BEATs** - Iterates acoustic tokenization and masked audio-token prediction to learn general audio representations. [Chen et al. 2023](<https://proceedings.mlr.press/v202/chen23ag.html> "BEATs: Audio Pre-Training with Acoustic Tokenizers; ICML"); [code](<https://github.com/microsoft/unilm/tree/master/beats>).

## Sound classification and cross-modal matching

- **Audio Spectrogram Transformer** - Treats overlapping spectrogram patches as Transformer tokens. ImageNet initialization substantially improves audio classification over training from scratch, showing that image pretraining can transfer to time-frequency representations. [Gong et al. 2021](<https://www.isca-archive.org/interspeech_2021/gong21b_interspeech.html> "AST: Audio Spectrogram Transformer; Interspeech").
- **Audio-text alignment** - See [Multi-View Learning](https://mlpapers.org/multiview-learning/#cross-modal-alignment-and-fusion) for CLAP and text-conditioned audio retrieval.

## Speech and music synthesis

- **Waveform generation** - See [Generative Models](https://mlpapers.org/generative-models/#autoregressive-and-masked-generation) for WaveNet and autoregressive waveform modelling.
- **Tacotron 2** - Predicts mel spectrograms from characters and uses a WaveNet vocoder to synthesize speech. In a single-speaker English test, listeners rate it 4.53 out of 5 versus 4.58 for recordings. [Shen et al. 2018](<https://arxiv.org/abs/1712.05884v2> "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions; ICASSP").
- **MusicGen** - A single autoregressive Transformer generates music as EnCodec tokens conditioned on text or melody. Delays between codebook streams let it predict several streams in parallel without separate coarse and fine generative models. [Copet et al. 2023](<https://arxiv.org/abs/2306.05284v3> "Simple and Controllable Music Generation; NeurIPS").

## Neural audio compression

- **SoundStream** - A convolutional encoder-decoder and residual vector quantizer compress speech, music and other audio. Dropping quantizer layers during training lets one model operate at 3-18 kbps with little quality loss against separate fixed-rate models. [Zeghidour et al. 2022](<https://arxiv.org/abs/2107.03312v1> "SoundStream: An End-to-End Neural Audio Codec; IEEE/ACM TASLP").
    - **EnCodec** - Extends residual-quantizer audio coding with a multiscale spectrogram discriminator and loss balancing by gradient contribution. Listening tests cover speech and music in mono and stereo. A token language model further reduces bitrate. [Défossez et al. 2023](<https://arxiv.org/abs/2210.13438v1> "High Fidelity Neural Audio Compression; TMLR").

## Related topics

- [Neural Networks](https://mlpapers.org/neural-nets/)
- [Feature Extraction](https://mlpapers.org/feature-extraction/)
- [Multi-View Learning](https://mlpapers.org/multiview-learning/)
- [Natural Language Processing](https://mlpapers.org/nlp/)
- [Generative Models](https://mlpapers.org/generative-models/)