{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-203", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/203", "wordCount": 1641, "segments": [{"start": 7.0, "text": "Welcome!"}, {"start": 8.0, "text": "I am Yichen, an engineer who works on drawing features."}, {"start": 12.0, "text": "In this session, I'll show you some exciting new APIs in PencilKit"}, {"start": 17.0, "text": "that bring handwriting recognition to your apps,"}, {"start": 21.0, "text": "along with improved access to the drawing model"}, {"start": 24.0, "text": "that open up interesting use cases."}, {"start": 27.0, "text": "Writing on iPad with Apple Pencil feels as natural as pen on paper."}, {"start": 32.0, "text": "But unlike paper, your handwriting becomes searchable, recognizable,"}, {"start": 37.0, "text": "and interactive."}, {"start": 39.0, "text": "PencilKit is the core framework for freehand drawing support"}, {"start": 43.0, "text": "on Apple's platforms."}, {"start": 45.0, "text": "It gives you a low-latency canvas with expressive inks,"}, {"start": 50.0, "text": "full Apple Pencil support, and access to the underlying data model."}, {"start": 55.0, "text": "iOS 26 introduced PaperKit, which builds on top of PencilKit"}, {"start": 61.0, "text": "to power Apple's drawing experience system-wide."}, {"start": 64.0, "text": "All of the new PencilKit APIs covered in this session are also available"}, {"start": 69.0, "text": "when working with PaperKit."}, {"start": 71.0, "text": "To learn more about it, check out the \"Meet PaperKit\" video"}, {"start": 75.0, "text": "from WWDC25"}, {"start": 78.0, "text": "and the \"Unwrap PaperKit\" video from WWDC26."}, {"start": 82.0, "text": "Apple's platforms have world-class handwriting recognition."}, {"start": 86.0, "text": "You've seen it power features"}, {"start": 88.0, "text": "like searching handwriting in Freeform and Notes."}, {"start": 92.0, "text": "One of the most requested capabilities for PencilKit is handwriting recognition."}, {"start": 97.0, "text": "Now it's available to you in iOS,"}, {"start": 101.0, "text": "iPadOS, macOS, and visionOS 27."}, {"start": 106.0, "text": "I'll start with PencilKit's handwriting recognition APIs,"}, {"start": 109.0, "text": "and show you how to turn handwriting into recognized words."}, {"start": 114.0, "text": "Next, I'll go over new conversion APIs that let you translate"}, {"start": 120.0, "text": "between PencilKit stroke paths and standard Bézier paths."}, {"start": 124.0, "text": "Then, I'll cover improvements to the drawing model,"}, {"start": 129.0, "text": "including stroke identity and selection access."}, {"start": 134.0, "text": "And finally, I'll show new stroke slicing APIs"}, {"start": 138.0, "text": "that let you split or extract segments from strokes."}, {"start": 143.0, "text": "I've been working with PencilKit in iOS 27 to build my son an app for learning"}, {"start": 149.0, "text": "to write in both Chinese and English."}, {"start": 152.0, "text": "In my app, he can practice writing words in both languages"}, {"start": 156.0, "text": "and get feedback on his work."}, {"start": 159.0, "text": "Here, the app has a flashcard showing a word in English."}, {"start": 164.0, "text": "To practice, I will write the Chinese translation."}, {"start": 167.0, "text": "Right now, it's showing \"Heart\" in English."}, {"start": 170.0, "text": "So I need to write \"心\" in Chinese."}, {"start": 174.0, "text": "Now, the app can use handwriting recognition"}, {"start": 177.0, "text": "to check my current work before I finish the whole word."}, {"start": 182.0, "text": "The Check button shakes to tell me the answer wasn't right."}, {"start": 189.0, "text": "This time the app recognized my handwriting as the Chinese word \"心\""}, {"start": 194.0, "text": "which matched the prompt, and the app confirmed it's correct."}, {"start": 198.0, "text": "Great!!"}, {"start": 200.0, "text": "That's what handwriting recognition APIs can do in PencilKit."}, {"start": 205.0, "text": "I'll get started with the core of handwriting recognition."}, {"start": 209.0, "text": "PKStrokeRecognizer is a Swift actor, so it's thread-safe by design."}, {"start": 216.0, "text": "All methods are asynchronous, because recognition takes time."}, {"start": 221.0, "text": "There are three main capabilities"}, {"start": 223.0, "text": "that together provide a handwriting recognition experience"}, {"start": 227.0, "text": "that is broadly useful for your apps."}, {"start": 231.0, "text": "I'll walk through each one."}, {"start": 233.0, "text": "The first capability is Recognized text."}, {"start": 236.0, "text": "It returns the single most likely result for what was written."}, {"start": 242.0, "text": "By default, PKStrokeRecognizer uses the device's languages to determine"}, {"start": 247.0, "text": "how to interpret the handwriting."}, {"start": 250.0, "text": "You can also configure preferredLanguages explicitly"}, {"start": 254.0, "text": "to match your app's language context."}, {"start": 258.0, "text": "You can recognize an entire drawing at once,"}, {"start": 261.0, "text": "or pass in a specific subset of strokeIDs."}, {"start": 265.0, "text": "I'll swipe over to the open-ended practice view, to show Recognized text in action."}, {"start": 272.0, "text": "I can write anything I want here, and PencilKit will recognize it."}, {"start": 278.0, "text": "Awesome!"}, {"start": 280.0, "text": "Hello WWDC."}, {"start": 282.0, "text": "As of iOS 27, PKStrokeRecognizer supports 29 languages."}, {"start": 288.0, "text": "The list of languages supported can be accessed"}, {"start": 291.0, "text": "using the supportedLanguages property on PKStrokeRecognizer."}, {"start": 297.0, "text": "Note that handwriting recognition in Simulator only supports languages"}, {"start": 301.0, "text": "that use Latin characters."}, {"start": 304.0, "text": "Handwriting recognition in PencilKit runs entirely on device."}, {"start": 309.0, "text": "The recognition model is offline and included with the operating system."}, {"start": 314.0, "text": "Handwriting recognition is fast,"}, {"start": 316.0, "text": "and works on all devices supported by iOS 27."}, {"start": 322.0, "text": "The next API is Indexable content."}, {"start": 325.0, "text": "It provides a single string representing the contents of the entire drawing."}, {"start": 333.0, "text": "This is particularly useful for features like Spotlight search."}, {"start": 338.0, "text": "When multiple languages are active, Indexable content may contain results"}, {"start": 344.0, "text": "in more than one of those languages."}, {"start": 347.0, "text": "When handwriting is ambiguous, you want all possible interpretations"}, {"start": 352.0, "text": "in your index;"}, {"start": 353.0, "text": "so users can find their content regardless of what they search for."}, {"start": 358.0, "text": "Is that a \"1\" or a lowercase \"L\"?"}, {"start": 362.0, "text": "Is that \"101\" or \"lol\"?"}, {"start": 365.0, "text": "Where recognized text gives you one best answer,"}, {"start": 369.0, "text": "Indexable content gives you all the candidates concatenated together."}, {"start": 375.0, "text": "If you're persisting Indexable content to disk,"}, {"start": 378.0, "text": "keep in mind that recognition results"}, {"start": 381.0, "text": "can improve over time as the underlying models are updated."}, {"start": 386.0, "text": "PKStrokeRecognizer exposes a recognizerVersion property."}, {"start": 391.0, "text": "Store this alongside your indexedContent, and when loading it back,"}, {"start": 397.0, "text": "compare against the current version to decide whether to re-index."}, {"start": 404.0, "text": "Additionally, consider throttling your calls to PKStrokeRecognizer."}, {"start": 409.0, "text": "Updating the results on every stroke may use more power"}, {"start": 414.0, "text": "than you need to provide your indexing feature."}, {"start": 417.0, "text": "The third API is for searching text."}, {"start": 422.0, "text": "Given a target string, it returns an array of search results"}, {"start": 427.0, "text": "indicating where that word likely appears in the drawing,"}, {"start": 432.0, "text": "considering all the candidates."}, {"start": 435.0, "text": "This is what powers interactive search,"}, {"start": 439.0, "text": "where a highlight appears around matching strokes."}, {"start": 443.0, "text": "search() also pairs naturally with UIFindInteraction,"}, {"start": 448.0, "text": "the system find and replace experience."}, {"start": 452.0, "text": "By implementing UIFindInteractionDelegate"}, {"start": 455.0, "text": "and driving it with search() under the hood,"}, {"start": 458.0, "text": "you get the system search UI, complete with result navigation"}, {"start": 463.0, "text": "and highlighting, directly in your drawing canvas."}, {"start": 468.0, "text": "Back in the flashcards,"}, {"start": 470.0, "text": "I've made a change to show how the search function"}, {"start": 473.0, "text": "powers flashcard matching."}, {"start": 477.0, "text": "I've added a visualization of the bounds returned in SearchResults."}, {"start": 482.0, "text": "Now there's a box showing where the search matched."}, {"start": 488.0, "text": "These capabilities also make handwritten content more accessible."}, {"start": 494.0, "text": "You can connect VoiceOver to speak handwriting aloud,"}, {"start": 498.0, "text": "making handwriting accessible to people who rely on screen readers."}, {"start": 503.0, "text": "And search lets assistive features locate and navigate to specific words"}, {"start": 509.0, "text": "within a drawing."}, {"start": 510.0, "text": "Together, they help bridge the gap in accessibility"}, {"start": 515.0, "text": "between handwritten and typed text."}, {"start": 518.0, "text": "Next: Path conversion."}, {"start": 521.0, "text": "Path conversion is a powerful set of APIs"}, {"start": 525.0, "text": "that help you to bring PencilKit to your app."}, {"start": 530.0, "text": "PencilKit represents stroke paths as cubic uniform B-splines."}, {"start": 535.0, "text": "For more information on how PencilKit stores paths,"}, {"start": 539.0, "text": "check out the \"Inspect, modify, and construct"}, {"start": 542.0, "text": "PencilKit drawings\" video from WWDC20."}, {"start": 547.0, "text": "B-Splines are a great representation for drawing,"}, {"start": 551.0, "text": "but are less common than Bézier paths."}, {"start": 555.0, "text": "In iOS 27, PKStrokePath supports conversion between the two."}, {"start": 562.0, "text": "When converting a path, PencilKit handles the geometry."}, {"start": 567.0, "text": "Bézier paths don't carry PencilKit properties like size, opacity, or force."}, {"start": 573.0, "text": "So you need to provide those for each control point."}, {"start": 578.0, "text": "When starting from a PKStrokePath, converting to and back from a Bézier path"}, {"start": 584.0, "text": "will result in the same control point locations,"}, {"start": 587.0, "text": "allowing for storing PencilKit strokes in a Bézier-based format,"}, {"start": 593.0, "text": "and reconstructing without any loss of fidelity."}, {"start": 597.0, "text": "If your app has its own canvas with strokes stored as Bézier paths,"}, {"start": 603.0, "text": "you can now convert those to PKStrokePaths,"}, {"start": 607.0, "text": "build a PKDrawing,"}, {"start": 609.0, "text": "and feed it into PKStrokeRecognizer."}, {"start": 611.0, "text": "This makes handwriting recognition compatible with any canvas,"}, {"start": 617.0, "text": "not just PKCanvasView."}, {"start": 621.0, "text": "Beyond expanding where PencilKit can be used,"}, {"start": 625.0, "text": "iOS 27 introduced key additions"}, {"start": 628.0, "text": "that enable deeper access to the model."}, {"start": 632.0, "text": "This flexibility supports more custom use cases in your apps."}, {"start": 638.0, "text": "PKStroke and PKStrokePath both gain conformance to the Identifiable protocol."}, {"start": 645.0, "text": "It is a stable UUID, so you can track a stroke across transforms, edits,"}, {"start": 650.0, "text": "and undo operations."}, {"start": 654.0, "text": "With a stable identity in place, you can now control the selection state"}, {"start": 658.0, "text": "on PKCanvasView."}, {"start": 661.0, "text": "There's also a new delegate method, canvasViewSelectionDidChange,"}, {"start": 666.0, "text": "that fires whenever the user's selection changes."}, {"start": 670.0, "text": "When certain inks are drawn together quickly,"}, {"start": 672.0, "text": "PencilKit composites those strokes together"}, {"start": 676.0, "text": "as if the inks were still wet, by using an equal renderGroupID."}, {"start": 681.0, "text": "In iOS 27, it is now controllable."}, {"start": 685.0, "text": "The last set of new APIs gives you the ability to slice through strokes"}, {"start": 690.0, "text": "in two different ways:"}, {"start": 692.0, "text": "Programmatic erasing and Substroke extraction."}, {"start": 696.0, "text": "If you have worked with PencilKit's data model,"}, {"start": 700.0, "text": "you may already be familiar with the stroke mask."}, {"start": 703.0, "text": "PencilKit represents partial erasure using a mask."}, {"start": 708.0, "text": "When the pixel eraser removes part of a stroke,"}, {"start": 711.0, "text": "the remaining visible portions are defined by a mask."}, {"start": 715.0, "text": "Starting in iOS 27, you can apply that same operation programmatically."}, {"start": 721.0, "text": "You provide a PKStrokePath as the eraser."}, {"start": 725.0, "text": "It cuts through the drawing, slicing a single stroke"}, {"start": 730.0, "text": "into multiple independent strokes"}, {"start": 732.0, "text": "with their own mask,"}, {"start": 735.0, "text": "just as if the user had used the eraser tool on the canvas."}, {"start": 741.0, "text": "One thing to keep in mind: slicing can be expensive on complex drawings."}, {"start": 747.0, "text": "If your drawing has a large number of strokes, be mindful of performance"}, {"start": 753.0, "text": "and consider erasing on a background thread"}, {"start": 757.0, "text": "rather than blocking your UI."}, {"start": 762.0, "text": "When erasing visually cuts strokes,"}, {"start": 765.0, "text": "Substroke extraction lets you efficiently obtain a section"}, {"start": 769.0, "text": "as a new stroke or path."}, {"start": 773.0, "text": "Starting in iOS 27, both PKStroke and PKStrokePath support subscript access"}, {"start": 781.0, "text": "with parametric ranges anywhere along the path."}, {"start": 785.0, "text": "This gives you precise control over exactly where a slice begins and ends."}, {"start": 794.0, "text": "PencilKit inks have features like pencil texture particles"}, {"start": 797.0, "text": "that are positioned relative to the full stroke."}, {"start": 802.0, "text": "When taking substrokes,"}, {"start": 803.0, "text": "PencilKit maintains the consistency of those particles."}, {"start": 808.0, "text": "For Chinese characters, the order that you write the strokes is important."}, {"start": 812.0, "text": "To check that the order is correct, I built a feature using substrokes"}, {"start": 816.0, "text": "to replay how I wrote the word."}, {"start": 820.0, "text": "PencilKit implements rendering in Metal"}, {"start": 822.0, "text": "with optimizations that make my animation smooth."}, {"start": 828.0, "text": "To take full advantage of these powerful APIs:"}, {"start": 832.0, "text": "Adopt PKStrokeRecognizer and bring handwriting recognition to your app."}, {"start": 838.0, "text": "Try PencilKit in new places to convert your existing Bézier paths"}, {"start": 843.0, "text": "into PKStrokePaths and take advantage of handwriting recognition,"}, {"start": 848.0, "text": "even without PKCanvasView."}, {"start": 850.0, "text": "Dive deep into PencilKit's model to track strokes with stable identity,"}, {"start": 856.0, "text": "respond to selection changes,"}, {"start": 858.0, "text": "and build a new level of custom experiences."}, {"start": 863.0, "text": "Finally, take advantage of stroke slicing."}, {"start": 867.0, "text": "Cut through drawings with programmatic erasing,"}, {"start": 869.0, "text": "and build smooth animations with substrokes."}, {"start": 874.0, "text": "Each of these APIs are simple to integrate."}, {"start": 877.0, "text": "Together, they unlock features in your apps that were not possible before."}, {"start": 884.0, "text": "The sample code for the demo is available in the resources for this video."}, {"start": 890.0, "text": "Handwriting becomes recognized, searchable, and interactive —"}, {"start": 895.0, "text": "everywhere in your app."}, {"start": 897.0, "text": "That's PencilKit in iOS 27."}, {"start": 900.0, "text": "Thanks for watching!"}]}