{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-253", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/253", "wordCount": 1879, "segments": [{"start": 6.0, "text": "Hi. I'm Conner from the Computational Music Team."}, {"start": 11.0, "text": "And I'm excited to introduce you to a framework called Music Understanding."}, {"start": 16.0, "text": "It gives you access to on-device musical intelligence across all Apple platforms."}, {"start": 23.0, "text": "It handles all the signal processing and model inference for you"}, {"start": 27.0, "text": "so you don't need any expertise in signal processing or machine learning to use it."}, {"start": 34.0, "text": "And because it runs entirely on-device,"}, {"start": 37.0, "text": "the audio you analyze stays private and works offline."}, {"start": 42.0, "text": "At Apple, The Final Cut Pro team used the Music Understanding framework"}, {"start": 47.0, "text": "to power two features of their app."}, {"start": 51.0, "text": "In the beat detection feature,"}, {"start": 53.0, "text": "Final Cut Pro analyzes a song for its rhythm"}, {"start": 57.0, "text": "and structure to reveal its beat grid."}, {"start": 61.0, "text": "This helps editors visualize"}, {"start": 63.0, "text": "and align their edits to song parts, bars, and beats."}, {"start": 69.0, "text": "And in Final Cut Pro for iPad"}, {"start": 72.0, "text": "the montage feature analyzes for rhythm, pace, and structure"}, {"start": 77.0, "text": "to automatically synchronize clips to the music."}, {"start": 82.0, "text": "I'll start by going over what the framework can do."}, {"start": 86.0, "text": "Then, I'll follow that up by explaining how you can use the framework."}, {"start": 91.0, "text": "Finally, I'll go through the API"}, {"start": 94.0, "text": "and show how it was used to build a sample app"}, {"start": 96.0, "text": "for understanding music."}, {"start": 99.0, "text": "The framework provides analysis around six main areas:"}, {"start": 104.0, "text": "key,"}, {"start": 106.0, "text": "rhythm,"}, {"start": 107.0, "text": "structure,"}, {"start": 108.0, "text": "pace,"}, {"start": 109.0, "text": "instrument activity,"}, {"start": 111.0, "text": "and loudness."}, {"start": 113.0, "text": "Rhythm is the pulse of a song, driven by individual beats."}, {"start": 119.0, "text": "These beats build into bars."}, {"start": 122.0, "text": "The number of beats in one minute is called beats per minute"}, {"start": 126.0, "text": "or bpm."}, {"start": 129.0, "text": "Bars form phrases,"}, {"start": 131.0, "text": "which you can think of as musical sentences."}, {"start": 135.0, "text": "Phrases combine into segments,"}, {"start": 137.0, "text": "creating a more complete musical statement..."}, {"start": 142.0, "text": "and those segments ultimately build the sections."}, {"start": 146.0, "text": "You can think of a section"}, {"start": 147.0, "text": "as a chorus, verse, intro or bridge."}, {"start": 152.0, "text": "During a song,"}, {"start": 153.0, "text": "instruments such as a drum,"}, {"start": 156.0, "text": "bass,"}, {"start": 157.0, "text": "or vocals"}, {"start": 159.0, "text": "may be playing at different times"}, {"start": 161.0, "text": "and at different intensities."}, {"start": 164.0, "text": "These instruments play around a common set of notes"}, {"start": 167.0, "text": "called the key."}, {"start": 170.0, "text": "While the song may have a consistent pulse or bpm,"}, {"start": 174.0, "text": "different parts of the song may feel slower or faster."}, {"start": 179.0, "text": "This is called pace."}, {"start": 182.0, "text": "Over time the song may sound louder at some points than others."}, {"start": 188.0, "text": "These are the building blocks of the Music Understanding framework,"}, {"start": 193.0, "text": "and by integrating it in your app,"}, {"start": 195.0, "text": "you unlock a whole new level of possibilities."}, {"start": 199.0, "text": "Next, I'll talk about how to use the framework."}, {"start": 203.0, "text": "At a high level,"}, {"start": 204.0, "text": "apps interact with a MusicUnderstandingSession,"}, {"start": 208.0, "text": "initializing with either an AVAsset"}, {"start": 211.0, "text": "or a custom audio provider."}, {"start": 214.0, "text": "To start analysis, clients call analyze and await results."}, {"start": 220.0, "text": "By default, the framework analyzes for all analysis types."}, {"start": 225.0, "text": "For the highest performance,"}, {"start": 227.0, "text": "you can specify which analysis types you are interested in"}, {"start": 231.0, "text": "to avoid unnecessary computations."}, {"start": 235.0, "text": "To explore the framework more deeply,"}, {"start": 237.0, "text": "I'll review a sample app called Music Understanding Lab,"}, {"start": 241.0, "text": "available on developer.apple.com."}, {"start": 246.0, "text": "Let me show you how Music Understanding Lab works."}, {"start": 250.0, "text": "First, I'll select a song on the device."}, {"start": 255.0, "text": "The app uses the Music Understanding framework to analyze the audio,"}, {"start": 260.0, "text": "turning it into a visual experience"}, {"start": 263.0, "text": "with a dedicated tile for each result."}, {"start": 266.0, "text": "When I hit play, notice,"}, {"start": 268.0, "text": "the Rhythm and Structure tiles update as the song plays."}, {"start": 273.0, "text": "The playhead ties the experience together,"}, {"start": 276.0, "text": "letting you follow along with the music."}, {"start": 281.0, "text": "I'll start by talking about how the Select Song... button is implemented."}, {"start": 287.0, "text": "Using the SwiftUI fileImporter,"}, {"start": 290.0, "text": "I'll select a file to get its URL."}, {"start": 293.0, "text": "Then I'll use that URL to create an AVURLAsset."}, {"start": 299.0, "text": "Be sure to set PreferPreciseDurationAndTimingKey to true"}, {"start": 304.0, "text": "to ensure the most accurate results."}, {"start": 307.0, "text": "Next, I'll create the session from the asset"}, {"start": 311.0, "text": "and call analyze and await the return of the session results."}, {"start": 316.0, "text": "Inside the SessionResult struct,"}, {"start": 318.0, "text": "every feature Music Understanding analyzes gets its own results field."}, {"start": 324.0, "text": "These are all optionals."}, {"start": 327.0, "text": "When you use the general analyze() API,"}, {"start": 330.0, "text": "all results will be available."}, {"start": 333.0, "text": "However, if you use the targeted analyze(for:) API,"}, {"start": 338.0, "text": "the framework will only return the results you asked for,"}, {"start": 341.0, "text": "and the rest will be nil."}, {"start": 344.0, "text": "Throughout the Music Understanding framework,"}, {"start": 347.0, "text": "there are two standard types used to associate time with a value."}, {"start": 352.0, "text": "A TimedValue associates a value with a CMTime."}, {"start": 358.0, "text": "Similar to TimedValue,"}, {"start": 360.0, "text": "a RangedValue associates a CMTimeRange with a value."}, {"start": 366.0, "text": "With these time-based types in mind,"}, {"start": 368.0, "text": "I'll discuss the features Music Understanding analyzes"}, {"start": 372.0, "text": "by showing how they are used in the Music Understanding Lab UI."}, {"start": 377.0, "text": "First I'm going to start with the Key tile."}, {"start": 382.0, "text": "In this song the musical key is D flat major."}, {"start": 387.0, "text": "For key analysis,"}, {"start": 388.0, "text": "the Music Understanding framework returns a KeyResult struct."}, {"start": 393.0, "text": "The result contains an array of ranges,"}, {"start": 396.0, "text": "mapping a KeySignature to a specific time range"}, {"start": 400.0, "text": "using a RangedValue."}, {"start": 403.0, "text": "A KeySignature contains a tonic"}, {"start": 406.0, "text": "and a mode."}, {"start": 408.0, "text": "A tonic can be any of the standard chromatic pitches."}, {"start": 413.0, "text": "It represents the root note, like C or G,"}, {"start": 416.0, "text": "around which the song is built..."}, {"start": 419.0, "text": "and the mode, which is either major or minor."}, {"start": 424.0, "text": "Next to Key is the Rhythm tile."}, {"start": 428.0, "text": "The tile displays bpm on the left"}, {"start": 430.0, "text": "and indicators on the right that light up as each beat plays."}, {"start": 436.0, "text": "When you analyze for rhythm,"}, {"start": 438.0, "text": "you get back a RhythmResult."}, {"start": 441.0, "text": "In this struct, Music Understanding gives you the timestamps"}, {"start": 445.0, "text": "for every beat and bar as arrays of CMTime."}, {"start": 450.0, "text": "The framework also provides the overall global tempo"}, {"start": 453.0, "text": "with beatsPerMinute."}, {"start": 456.0, "text": "Notice bpm is optional."}, {"start": 459.0, "text": "That's because if the framework hasn't processed enough audio"}, {"start": 463.0, "text": "to find at least two beats,"}, {"start": 465.0, "text": "the bpm will be set to nil."}, {"start": 468.0, "text": "Now, I'm going to talk about the Structure tile."}, {"start": 473.0, "text": "In the tile there are 3 rows of rectangles"}, {"start": 476.0, "text": "that indicate a song's structural hierarchy."}, {"start": 480.0, "text": "Each rectangle is a time range in the song."}, {"start": 484.0, "text": "Music Understanding supports three levels of structure:"}, {"start": 488.0, "text": "sections, segments and phrases."}, {"start": 493.0, "text": "The top row represents sections of a song."}, {"start": 497.0, "text": "Each block shows the time range of a section."}, {"start": 502.0, "text": "Each section is made up of one or more segments,"}, {"start": 505.0, "text": "which appear below the section rectangles,"}, {"start": 509.0, "text": "and each segment is made up of phrases."}, {"start": 513.0, "text": "During playback the current section, segment and phrase appear highlighted."}, {"start": 520.0, "text": "When you request structure analysis,"}, {"start": 523.0, "text": "the framework returns a StructureResult."}, {"start": 526.0, "text": "It has three properties"}, {"start": 528.0, "text": "for sections, segments and phrases."}, {"start": 532.0, "text": "For each of these, you get an array of CMTimeRanges."}, {"start": 537.0, "text": "The next tile is Pace."}, {"start": 540.0, "text": "It tells you how fast the music feels to the listener."}, {"start": 545.0, "text": "Parts of a song that feel faster or more energetic"}, {"start": 549.0, "text": "will have a higher value compared to slower or less energetic parts."}, {"start": 555.0, "text": "In this UI, taller bars represent higher energy,"}, {"start": 560.0, "text": "while shorter bars represent lower energy."}, {"start": 565.0, "text": "When you request pace analysis,"}, {"start": 567.0, "text": "you get back a PaceResult."}, {"start": 570.0, "text": "This struct has a single property"}, {"start": 572.0, "text": "containing an array of ranged values."}, {"start": 576.0, "text": "Next, I'll talk about the instrument activity tiles."}, {"start": 582.0, "text": "Music Understanding Lab displays several tiles"}, {"start": 585.0, "text": "that visualize instrument activity."}, {"start": 589.0, "text": "Either as time ranges,"}, {"start": 591.0, "text": "where color-coded bars indicate the active instruments present"}, {"start": 595.0, "text": "or as a detailed activity graph."}, {"start": 599.0, "text": "The graph plots values between 0 and 1"}, {"start": 602.0, "text": "representing the strength of each instrument."}, {"start": 606.0, "text": "The closer the value is to 1,"}, {"start": 609.0, "text": "the louder the instrument is in the mix."}, {"start": 613.0, "text": "When you request instrument activity,"}, {"start": 615.0, "text": "the framework returns an InstrumentActivityResult."}, {"start": 619.0, "text": "It has two properties,"}, {"start": 622.0, "text": "one for ranges"}, {"start": 623.0, "text": "and one for activity."}, {"start": 626.0, "text": "The Ranges API provides a dictionary,"}, {"start": 629.0, "text": "mapping each Instrument to an array of CMTimeRanges."}, {"start": 634.0, "text": "This is great for situations where you just want to know"}, {"start": 638.0, "text": "if an instrument is present or not."}, {"start": 641.0, "text": "But sometimes you need more detail"}, {"start": 643.0, "text": "and activity provides that."}, {"start": 646.0, "text": "Activity maps an instrument to a TimedValue of Floats."}, {"start": 652.0, "text": "The activity results express how intensely an instrument is playing over time,"}, {"start": 658.0, "text": "and is a great source to drive audio-reactive animations."}, {"start": 663.0, "text": "The Loudness tile appears below instrument activity."}, {"start": 668.0, "text": "The framework provides measurements"}, {"start": 670.0, "text": "in Loudness Units Full Scale,"}, {"start": 672.0, "text": "or LUFS,"}, {"start": 673.0, "text": "which is the industry standard"}, {"start": 675.0, "text": "for modeling how the human ear perceives volume."}, {"start": 680.0, "text": "At the top of the tile"}, {"start": 681.0, "text": "is a single Integrated Loudness value,"}, {"start": 684.0, "text": "which gives the average loudness for the whole song."}, {"start": 688.0, "text": "Below that is a graph of momentary loudness,"}, {"start": 692.0, "text": "showing how loudness varies over time."}, {"start": 696.0, "text": "The framework also provides a peak value,"}, {"start": 699.0, "text": "which describes the absolute highest audio volume in decibels."}, {"start": 705.0, "text": "When you request loudness analysis,"}, {"start": 707.0, "text": "the framework returns a LoudnessResult struct."}, {"start": 711.0, "text": "Music Understanding supports integrated, momentary and shortTerm loudness."}, {"start": 717.0, "text": "Integrated provides a single value"}, {"start": 720.0, "text": "that represents the overall loudness of the audio."}, {"start": 724.0, "text": "Momentary and shortTerm"}, {"start": 726.0, "text": "provide time-stamped values every 100 milliseconds."}, {"start": 731.0, "text": "Momentary values are calculated over a window of 400 milliseconds."}, {"start": 737.0, "text": "They are useful to detect short, sudden spikes in loudness."}, {"start": 742.0, "text": "ShortTerm values are calculated over a window of 3 seconds,"}, {"start": 747.0, "text": "which provides a smoother view of the loudness trend over time."}, {"start": 752.0, "text": "The peak value tells you exactly where the track hits its maximum level"}, {"start": 757.0, "text": "and is measured in decibels."}, {"start": 760.0, "text": "MusicUnderstandingSession also provides a streaming API for loudness."}, {"start": 766.0, "text": "Values are delivered via an AsyncSequence"}, {"start": 769.0, "text": "for every 100ms of audio analyzed by the framework."}, {"start": 775.0, "text": "Here's an example of how it can be used."}, {"start": 778.0, "text": "I'll initialize the session as I did previously"}, {"start": 782.0, "text": "but this time I'll set up two tasks:"}, {"start": 785.0, "text": "one to consume the loudness results as they are delivered,"}, {"start": 790.0, "text": "and another to begin the analysis."}, {"start": 794.0, "text": "Now I'll talk about the AudioProvider in this example."}, {"start": 799.0, "text": "An AudioProvider conforms to AsyncSequence"}, {"start": 803.0, "text": "and yields AVReadOnlyAudioPCMBuffer objects."}, {"start": 808.0, "text": "When the AudioProvider has sent all audio buffers,"}, {"start": 812.0, "text": "it must send a final nil to signal completion."}, {"start": 818.0, "text": "I'd like to focus now on the two icons at the top of Music Understanding Lab."}, {"start": 824.0, "text": "At the far right there is a Share button."}, {"start": 828.0, "text": "When you tap it, a JSON file is exported with all analysis data."}, {"start": 835.0, "text": "All MusicUnderstanding results are codable."}, {"start": 839.0, "text": "To encode to JSON, just create a JSONEncoder"}, {"start": 843.0, "text": "and encode the session results."}, {"start": 846.0, "text": "A button with a filmstrip icon appears next to the export button."}, {"start": 852.0, "text": "When tapped, it opens the Video tile."}, {"start": 856.0, "text": "The Video tile uses structure and pace"}, {"start": 859.0, "text": "to create a video synced to the music."}, {"start": 862.0, "text": "I'll talk about how it works."}, {"start": 865.0, "text": "The Video tile displays a series of video clips that match the feeling of the music."}, {"start": 872.0, "text": "The algorithm starts by identifying the song section time ranges."}, {"start": 878.0, "text": "Then it uses the pace of each section"}, {"start": 882.0, "text": "to determine how many clips to display in that time range."}, {"start": 887.0, "text": "Since pace is an event per minute rate,"}, {"start": 889.0, "text": "it can be divided by 60 seconds to determine the time per clip."}, {"start": 895.0, "text": "This produces clips that always begin at the start of each section,"}, {"start": 900.0, "text": "and the clips within a section match the energy of the song."}, {"start": 906.0, "text": "I then used this timing information"}, {"start": 909.0, "text": "to construct a video synced to the music."}, {"start": 912.0, "text": "The video clips are retimed to match the target clip duration"}, {"start": 917.0, "text": "with longer, slower clips during the less energetic parts,"}, {"start": 922.0, "text": "and shorter, faster clips during the more energetic parts."}, {"start": 928.0, "text": "This overview gives you a sense of how these APIs work together in practice."}, {"start": 934.0, "text": "With these basics covered, you are ready to dive in!"}, {"start": 939.0, "text": "With Music Understanding,"}, {"start": 941.0, "text": "you can sync visuals to the beat, loudness, or pace of a song"}, {"start": 945.0, "text": "to build powerful video editing features,"}, {"start": 948.0, "text": "organize music catalogs by tempo or key to drive a dj app"}, {"start": 954.0, "text": "or pre-compute and bundle analysis data"}, {"start": 957.0, "text": "to animate your game to music."}, {"start": 960.0, "text": "Check out the Music Understanding Framework documentation"}, {"start": 964.0, "text": "on developer.apple.com,"}, {"start": 966.0, "text": "and download \"Music Understanding Lab\""}, {"start": 969.0, "text": "to help kickstart development of your own applications."}, {"start": 974.0, "text": "This technology was built to solve real challenges here at Apple."}, {"start": 979.0, "text": "The ideas were there"}, {"start": 981.0, "text": "but the tools to build them were not."}, {"start": 984.0, "text": "These tools are now in your hands."}, {"start": 988.0, "text": "Make something awesome."}, {"start": 990.0, "text": "Thanks for watching!"}]}