{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-347", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/347", "wordCount": 4048, "segments": [{"start": 7.0, "text": "Howdy, I'm Willy."}, {"start": 9.0, "text": "Today, I'll be telling you how you can identify"}, {"start": 12.0, "text": "and mitigate new risks to agentic features within your app."}, {"start": 16.0, "text": "Later, my colleague Akshay will provide the concrete actionable steps"}, {"start": 21.0, "text": "you can take to secure your app using APIs available on our platform."}, {"start": 26.0, "text": "With large language models, or LLMs, becoming commonplace,"}, {"start": 30.0, "text": "many apps are looking at ways to use them to enable new, intelligent features,"}, {"start": 35.0, "text": "making the LLM a key system component."}, {"start": 38.0, "text": "Within your application, you can send instructions and a prompt"}, {"start": 42.0, "text": "containing the user's request and extra context"}, {"start": 45.0, "text": "to an LLM to have it execute one or multiple actions"}, {"start": 48.0, "text": "to get intermediate results,"}, {"start": 50.0, "text": "until finally providing a response to the user."}, {"start": 53.0, "text": "Our platform lets you create agentic experiences using either"}, {"start": 58.0, "text": "the Foundation Models framework to design your own agent,"}, {"start": 61.0, "text": "or the App Intents framework to let your app work with Siri."}, {"start": 66.0, "text": "Now, with new capabilities comes new security risks."}, {"start": 70.0, "text": "LLMs introduce a new probabilistic engine within your application"}, {"start": 75.0, "text": "that is both powerful, but risks being tricked."}, {"start": 79.0, "text": "The purpose of this talk is to highlight the new security risks"}, {"start": 82.0, "text": "with agentic features,"}, {"start": 84.0, "text": "and provide techniques and APIs you can use to protect your users."}, {"start": 88.0, "text": "The key thing is, we want to make sure that your app works as you intend"}, {"start": 93.0, "text": "and with the user's security in mind."}, {"start": 96.0, "text": "Before going forward, we want to make clear what this talk is not covering."}, {"start": 100.0, "text": "We won't be talking about model safety,"}, {"start": 102.0, "text": "which refers to ensuring that what the model outputs is safe."}, {"start": 106.0, "text": "And we also won't be discussing model guardrails"}, {"start": 109.0, "text": "and protecting against circumvention."}, {"start": 111.0, "text": "While some of the principles we will discuss"}, {"start": 113.0, "text": "could be used to handle such cases,"}, {"start": 115.0, "text": "we'll be focusing on an external attacker trying to compromise your application."}, {"start": 121.0, "text": "If you want to review model safety, check out the great talk linked below."}, {"start": 126.0, "text": "Let's start by talking about the new risks that come with agentic systems."}, {"start": 131.0, "text": "To begin, we'll start by considering why an attacker may try to target your app."}, {"start": 137.0, "text": "Your application may do things an attacker is interested in, such as:"}, {"start": 141.0, "text": "host sensitive user data, perform financial transactions,"}, {"start": 145.0, "text": "access system resources like the microphone or camera,"}, {"start": 148.0, "text": "or even control physical devices."}, {"start": 151.0, "text": "An attacker may want to exploit your application to achieve their goals."}, {"start": 156.0, "text": "To help illustrate our attacks and mitigations,"}, {"start": 159.0, "text": "we'll be working with the Loose Leaf app,"}, {"start": 161.0, "text": "an example app that could be the next generation social network"}, {"start": 165.0, "text": "for all things tea, from hot to cold to boba,"}, {"start": 168.0, "text": "we're certain Loose Leaf is the future of social networking."}, {"start": 173.0, "text": "Loose Leaf already ships with some exciting features"}, {"start": 176.0, "text": "such as being able to message tea recipes to friends,"}, {"start": 179.0, "text": "or the ability to share the incredible tea-party photos you took."}, {"start": 184.0, "text": "We previously talked about how you can threat model"}, {"start": 186.0, "text": "and mitigate these traditional features."}, {"start": 189.0, "text": "It's worth reviewing that video to ensure we don't forget our essentials."}, {"start": 192.0, "text": "Now, the Loose Leaf developers have been brewing"}, {"start": 195.0, "text": "and they're excited to announce a spicy new feature called\""}, {"start": 198.0, "text": "\"Organize a tea party,\""}, {"start": 200.0, "text": "which uses the Foundation Models and App Intents framework to:"}, {"start": 204.0, "text": "look at your calendar to find the best time to host a tea party,"}, {"start": 207.0, "text": "determine what friends should be invited"}, {"start": 209.0, "text": "along what teas to serve based on their profiles,"}, {"start": 212.0, "text": "and also order the teas that everyone likes!"}, {"start": 215.0, "text": "Wow!"}, {"start": 217.0, "text": "This new feature relies on Loose Leaf's agentic loop"}, {"start": 221.0, "text": "and has two notable properties."}, {"start": 224.0, "text": "First, this feature takes in context from multiple locations"}, {"start": 227.0, "text": "to help the agent make decisions."}, {"start": 230.0, "text": "Second, the agent can call one or more actions on the user's behalf"}, {"start": 234.0, "text": "that can have different kinds of side effects."}, {"start": 237.0, "text": "Starting with the first property, we introduce a new risk:"}, {"start": 241.0, "text": "indirect prompt injection."}, {"start": 244.0, "text": "Indirect prompt injection refers to instructions embedded in extra context"}, {"start": 249.0, "text": "provided to the model with the intent to redirect control flow."}, {"start": 254.0, "text": "In our agentic loop, we see that this refers to instructions"}, {"start": 258.0, "text": "that may be embedded in the initial extra context in the prompt,"}, {"start": 261.0, "text": "or within a tool result."}, {"start": 264.0, "text": "What this may look like in practice"}, {"start": 266.0, "text": "is a user requesting to start a tea party with their calendar appended,"}, {"start": 270.0, "text": "but the calendar containing an event with instructions to the model"}, {"start": 273.0, "text": "to perform another action, such as deleting sensitive user data!"}, {"start": 277.0, "text": "Yikes!"}, {"start": 279.0, "text": "Part of the threat modeling exercise we'll go through"}, {"start": 281.0, "text": "is identifying all the sources of untrusted context."}, {"start": 284.0, "text": "The second property of our agentic system is the action calling capability,"}, {"start": 289.0, "text": "which could potentially have side effects,"}, {"start": 291.0, "text": "or unintended consequences of executing the action."}, {"start": 295.0, "text": "When combined with indirect prompt injections,"}, {"start": 298.0, "text": "an attacker could cause an action with side effects to be executed"}, {"start": 302.0, "text": "that achieves their goal,"}, {"start": 304.0, "text": "such as exfiltrating your user's data, stealing money,"}, {"start": 307.0, "text": "controlling physical devices, or deleting data."}, {"start": 312.0, "text": "When an indirect prompt injection leads to an unintended action,"}, {"start": 316.0, "text": "we can consider the injection having two different effects."}, {"start": 319.0, "text": "First is data poisoning,"}, {"start": 322.0, "text": "which we refer to an attacker influencing the parameters of an executed action."}, {"start": 327.0, "text": "For example, a user may want to send a message to their mom,"}, {"start": 330.0, "text": "but an attacker injects an instruction to send a message to themselves instead."}, {"start": 336.0, "text": "Second is action poisoning,"}, {"start": 338.0, "text": "where the attacker influences what action to execute."}, {"start": 342.0, "text": "A user may simply ask to summarize an email,"}, {"start": 345.0, "text": "but an attacker could steer the LLM to open a malicious web page"}, {"start": 349.0, "text": "with the email appended to an attacker-chosen URL instead."}, {"start": 355.0, "text": "Conceptualizing these risks, we can look to Simon Willison's Lethal Trifecta,"}, {"start": 360.0, "text": "which describes that a user is in most danger"}, {"start": 362.0, "text": "whenever an agentic system has:"}, {"start": 365.0, "text": "access to private data,"}, {"start": 366.0, "text": "exposure to untrusted content,"}, {"start": 369.0, "text": "and the ability to externally communicate."}, {"start": 371.0, "text": "This last bullet we can further generalize"}, {"start": 374.0, "text": "to consider the risk of actions with any side effect."}, {"start": 378.0, "text": "Summarizing this section,"}, {"start": 380.0, "text": "we want to emphasize that solving indirect prompt injection"}, {"start": 383.0, "text": "is an active research area,"}, {"start": 385.0, "text": "meaning that our best approach at the moment"}, {"start": 387.0, "text": "is to understand how much your app is at risk,"}, {"start": 389.0, "text": "and aim to mitigate that risk."}, {"start": 391.0, "text": "Now that we've discussed the risks that come from agentic systems,"}, {"start": 395.0, "text": "we will go through a threat modeling exercise"}, {"start": 397.0, "text": "that you can do on your app to identify untrusted sources of data"}, {"start": 401.0, "text": "and identify potentially risky actions."}, {"start": 404.0, "text": "We start by performing a data flow analysis for our agentic loop input,"}, {"start": 409.0, "text": "aka the prompt."}, {"start": 410.0, "text": "We want to identify the data sources you'll use to construct your prompt."}, {"start": 415.0, "text": "For this exercise, we want to pinpoint the sources of untrusted context"}, {"start": 420.0, "text": "that may contain prompt injections."}, {"start": 422.0, "text": "Going back to our Loose Leaf app,"}, {"start": 424.0, "text": "there are a few data sources that feed into prompt construction."}, {"start": 427.0, "text": "First, is the instructions that provides the LLM some guidance"}, {"start": 431.0, "text": "about its purpose and role."}, {"start": 433.0, "text": "Next, is the user's prompt,"}, {"start": 435.0, "text": "the task the LLM will work on and the goal it's trying to achieve."}, {"start": 440.0, "text": "The prompt can also include extra context to help the LLM achieve the goal."}, {"start": 444.0, "text": "Such as including past tea orders, tea recipes that user has stored,"}, {"start": 449.0, "text": "upcoming calendar events to help determine the best time for a tea party"}, {"start": 453.0, "text": "and a friend feed, to incorporate content our friends are sharing."}, {"start": 457.0, "text": "Once we have an understanding of data sources that feed into our prompt,"}, {"start": 461.0, "text": "we need to identify what is considered untrusted."}, {"start": 464.0, "text": "As a general rule of thumb, we can consider any inputs"}, {"start": 468.0, "text": "coming from an external entity as the attack surface."}, {"start": 471.0, "text": "In our case, we identify the calendar content"}, {"start": 475.0, "text": "and the friend feed as untrusted because"}, {"start": 478.0, "text": "anyone could send the user a calendar invite"}, {"start": 480.0, "text": "that may get fed into the model"}, {"start": 482.0, "text": "and a user's \"friend\" could post anything on their feed"}, {"start": 485.0, "text": "that gets fed into the prompt,"}, {"start": 487.0, "text": "which could all contain prompt injections to influence the action to execute."}, {"start": 493.0, "text": "After identifying the sources of untrusted context,"}, {"start": 496.0, "text": "we want to examine the actions available to the agent"}, {"start": 499.0, "text": "and what side effects they may have."}, {"start": 501.0, "text": "First, we have the OrderTeaTool(),"}, {"start": 504.0, "text": "which is an essential action for getting tea for your tea party."}, {"start": 508.0, "text": "The PostAndFetchPublicFeedTool() will post on the user's feed"}, {"start": 511.0, "text": "with a message generated by the model, helpful for getting the word out to friends."}, {"start": 516.0, "text": "The BrewingTimerIntent() will help you during your tea party"}, {"start": 520.0, "text": "to ensure your tea is brewed for the right amount of time."}, {"start": 523.0, "text": "Finally, Delete Photo will remove a photo from the user's feed,"}, {"start": 527.0, "text": "in case the tea leaves didn't look just right."}, {"start": 530.0, "text": "As we consider all these actions,"}, {"start": 532.0, "text": "we need to identify what side effects each action may have."}, {"start": 537.0, "text": "The OrderTeaTool() has a financial risk associated with it,"}, {"start": 541.0, "text": "meaning the user could lose money if unintentionally called."}, {"start": 546.0, "text": "The PostAndFetchPublicFeedTool() on the other hand"}, {"start": 549.0, "text": "has a data exfiltration risk"}, {"start": 550.0, "text": "as the model could leak sensitive information via a public post."}, {"start": 555.0, "text": "BrewingTimerIntent() may not have side effects on its own,"}, {"start": 559.0, "text": "but if it takes a label, it could allow a prompt injection"}, {"start": 562.0, "text": "to write more instructions for later attacks."}, {"start": 565.0, "text": "Delete Photo has a data loss risk, especially if there is no undo capability."}, {"start": 571.0, "text": "Now that we've identified where malicious inputs could go into the LLM"}, {"start": 575.0, "text": "and the side effects actions may have,"}, {"start": 578.0, "text": "we can begin to design and implement mitigations"}, {"start": 581.0, "text": "that will protect the user."}, {"start": 582.0, "text": "We want to highlight that we should try to focus on"}, {"start": 585.0, "text": "deterministic mitigations as a baseline"}, {"start": 588.0, "text": "because their security guarantees are easier to audit and reason about."}, {"start": 593.0, "text": "Given the rapid development of model capabilities,"}, {"start": 595.0, "text": "we can also consider other mitigations that have more probabilistic guarantees."}, {"start": 600.0, "text": "Here, we present a few different mitigations"}, {"start": 602.0, "text": "that can be used to protect your application"}, {"start": 605.0, "text": "by either adding checks at the prompt level,"}, {"start": 607.0, "text": "or at the action execution stage."}, {"start": 609.0, "text": "We've used some of these as we've designed Siri AI."}, {"start": 612.0, "text": "Let's walk through these now."}, {"start": 614.0, "text": "First, we can look back at our prompt and begin to add prompt mitigations."}, {"start": 619.0, "text": "We can redact sensitive data,"}, {"start": 621.0, "text": "such as personally identifiable information, or PII,"}, {"start": 625.0, "text": "that may be stored in past orders."}, {"start": 627.0, "text": "That way sensitive data never reaches the LLM and thus cannot be exfiltrated."}, {"start": 633.0, "text": "Next, we can incorporate spotlighting to the model"}, {"start": 636.0, "text": "to indicate that this content is considered untrusted."}, {"start": 640.0, "text": "This is a probabilistic mitigation"}, {"start": 642.0, "text": "because the prompt injection could be constructed in a way"}, {"start": 644.0, "text": "that negates the spotlighting."}, {"start": 646.0, "text": "We suggest incorporating it, though,"}, {"start": 648.0, "text": "as different models could more effectively enforce these restrictions."}, {"start": 653.0, "text": "Now, let's look at the action mitigations you can implement."}, {"start": 658.0, "text": "First, consider which actions should have a user confirmation."}, {"start": 662.0, "text": "These are actions that are worth having a human check before continuing"}, {"start": 666.0, "text": "due to the side effects they contain."}, {"start": 668.0, "text": "Next, consider which tools should only work"}, {"start": 671.0, "text": "whenever the device is authenticated, or unlocked."}, {"start": 675.0, "text": "Because the agent may be reachable from the lock screen,"}, {"start": 678.0, "text": "actions with significant risk to the user should not be accessible."}, {"start": 683.0, "text": "We've walked through different kinds of mitigations"}, {"start": 686.0, "text": "and how they can apply to your system,"}, {"start": 688.0, "text": "but there are many more types that exist"}, {"start": 690.0, "text": "and we welcome you to go and explore them"}, {"start": 693.0, "text": "to mitigate the risks to your app."}, {"start": 696.0, "text": "The key thing to remember when threat modeling"}, {"start": 698.0, "text": "is that you want to identify what an attacker may want from your application"}, {"start": 703.0, "text": "and from there apply mitigations to address risks at the prompt level,"}, {"start": 707.0, "text": "or at the action execution stage."}, {"start": 709.0, "text": "And now Akshay will show you concrete tools"}, {"start": 711.0, "text": "that you can use to protect your app."}, {"start": 713.0, "text": "Take it away, Akshay!"}, {"start": 715.0, "text": "Thanks Willy."}, {"start": 716.0, "text": "Hi, I am Akshay, and I'll show you how to secure your agentic app"}, {"start": 720.0, "text": "with some of the guardrails that Willy just discussed."}, {"start": 723.0, "text": "If you are building your app using Foundation Models framework,"}, {"start": 726.0, "text": "I will show you how to inject security checkpoints"}, {"start": 728.0, "text": "into your agent execution."}, {"start": 731.0, "text": "If you are integrating with Apple Intelligence using App Intents,"}, {"start": 734.0, "text": "I will cover the security mitigations available there."}, {"start": 737.0, "text": "Let's start with Foundation Models."}, {"start": 739.0, "text": "The Foundation Models framework provides a powerful API for building agents."}, {"start": 744.0, "text": "I am going to highlight the lifecycle event modifier API,"}, {"start": 747.0, "text": "and use it for injecting security guardrails."}, {"start": 749.0, "text": "I will assume basic familiarity with the framework."}, {"start": 752.0, "text": "To learn more, do checkout the excellent talk linked below."}, {"start": 755.0, "text": "Let's first build a simple agent for our Loose Leaf app using Foundation Models."}, {"start": 760.0, "text": "Now I don't know about you, but I can't build any agents"}, {"start": 763.0, "text": "without a cup of my Darjeeling black tea."}, {"start": 765.0, "text": "So before anything else, we will build a tool to order teas."}, {"start": 769.0, "text": "To define a tool, we conform to the Tool protocol."}, {"start": 771.0, "text": "We specify the name, description, and Arguments to the tool."}, {"start": 775.0, "text": "The model uses this metadata to understand our tool's purpose, and how to call it."}, {"start": 781.0, "text": "Then, we provide the actual Implementation that is run when this tool is called."}, {"start": 785.0, "text": "Let's define one more tool."}, {"start": 787.0, "text": "The PostAndFetchPublicFeedTool posts your message to the public feed,"}, {"start": 791.0, "text": "and retrieves newly posted messages."}, {"start": 793.0, "text": "The next step in building our agent, is to create a Profile."}, {"start": 797.0, "text": "In the Profile, we first add model Instructions"}, {"start": 800.0, "text": "and the tools we just defined."}, {"start": 801.0, "text": "Then we attach session properties, like which model to use."}, {"start": 805.0, "text": "Here, we are using the on-device model."}, {"start": 808.0, "text": "This Profile is then used to instantiate a LanguageModelSession,"}, {"start": 812.0, "text": "which can then be used in an agentic loop."}, {"start": 815.0, "text": "Now that we have a basic agent, we will inject our security policy."}, {"start": 819.0, "text": "To do this, we will use lifecycle event modifiers."}, {"start": 822.0, "text": "These modifiers are callbacks that deterministically trigger"}, {"start": 825.0, "text": "at certain lifecycle points in a session execution."}, {"start": 829.0, "text": "Thus we can use these lifecycle events as checkpoints"}, {"start": 832.0, "text": "to implement security policy."}, {"start": 834.0, "text": "We will look at two of these modifiers now."}, {"start": 837.0, "text": "Let's go back to our simplified agentic loop."}, {"start": 839.0, "text": "Like Sisyphus, the LLM outputs an action at each iteration;"}, {"start": 843.0, "text": "this action is run by the Executor,"}, {"start": 845.0, "text": "and its output rendered back to the LLM for the next iteration."}, {"start": 848.0, "text": "The first modifier lets us intercept tool calls before they run."}, {"start": 852.0, "text": "This is the .onToolCall modifier."}, {"start": 855.0, "text": "It is guaranteed to trigger when the LLM outputs a tool call,"}, {"start": 858.0, "text": "before the executor runs the tool."}, {"start": 861.0, "text": "The important point here is if this callback throws an error,"}, {"start": 864.0, "text": "then the tool is never executed."}, {"start": 866.0, "text": "Control returns to the loop immediately."}, {"start": 869.0, "text": "This makes this the perfect place to enforce confirmations."}, {"start": 873.0, "text": "Going back to our Loose Leaf agent,"}, {"start": 875.0, "text": "we notice that the OrderTeaTool has financial impact,"}, {"start": 878.0, "text": "and that makes me very nervous."}, {"start": 880.0, "text": "So I want to always ask for user confirmation"}, {"start": 882.0, "text": "before running this tool and transferring money."}, {"start": 886.0, "text": "To do this, we add an .onToolCall callback to our profile."}, {"start": 890.0, "text": "As this callback runs before every tool call,"}, {"start": 892.0, "text": "we first check if the current tool is the OrderTeaTool."}, {"start": 896.0, "text": "If not, we return immediately, and the tool is run."}, {"start": 899.0, "text": "But if it is, we ask the user for confirmation."}, {"start": 902.0, "text": "If the user does not confirm, we throw an error,"}, {"start": 905.0, "text": "which stops the tool from running."}, {"start": 907.0, "text": "You will replace confirmWithUser() function with your own implementation."}, {"start": 911.0, "text": "The point is that by adding confirmation logic"}, {"start": 913.0, "text": "to just this one point in our code,"}, {"start": 916.0, "text": "we get full coverage for all our tool calls."}, {"start": 919.0, "text": "So in summary, remember that this modifier runs before every tool execution,"}, {"start": 923.0, "text": "and the tool itself is not run until this callback returns."}, {"start": 927.0, "text": "You can block tool execution by throwing an error."}, {"start": 930.0, "text": "So conceptually, the .onToolCall modifier runs on the output of the model."}, {"start": 935.0, "text": "Let's now look at a modifier that helps us check the input to the model."}, {"start": 939.0, "text": "The .historyTransform fires"}, {"start": 941.0, "text": "before the transcript is rendered to the model for inference."}, {"start": 944.0, "text": "This happens both when a new user request arrives,"}, {"start": 947.0, "text": "and at each iteration of the loop."}, {"start": 950.0, "text": "The transformation modifies the tail of the transcript,"}, {"start": 952.0, "text": "and we will use that for spotlighting and redacting PII."}, {"start": 955.0, "text": "Returning to our example, note that PostAndFetchPublicFeedTool()"}, {"start": 959.0, "text": "returns posts from a public feed."}, {"start": 961.0, "text": "An attacker can easily post a prompt injection to that feed."}, {"start": 964.0, "text": "We must treat this feed's output with suspicion."}, {"start": 967.0, "text": "So we want to demarcate this output with special tags"}, {"start": 970.0, "text": "to tell the model that this is untrusted data."}, {"start": 974.0, "text": "We do this by adding Spotlighting delimiters"}, {"start": 976.0, "text": "inside the .historyTransform."}, {"start": 978.0, "text": "In the callback, we first iterate over the entries"}, {"start": 981.0, "text": "and focus only on toolOutput entries from our tool."}, {"start": 984.0, "text": "All other entries are copied to the output transcript unmodified."}, {"start": 988.0, "text": "We then modify the toolOutput entries."}, {"start": 991.0, "text": "We iterate over the segments, and for each relevant segment,"}, {"start": 994.0, "text": "add delimiter tags."}, {"start": 996.0, "text": "We are using angled brackets \"<>\" in this example."}, {"start": 999.0, "text": "You will use tags that are relevant for your model."}, {"start": 1001.0, "text": "The delimit() function, whose implementation we will skip,"}, {"start": 1004.0, "text": "transforms a text segment to one with delimited content."}, {"start": 1008.0, "text": "And now let's look at redaction."}, {"start": 1009.0, "text": "Actually, we can use exactly the same idea for redacting sensitive data too."}, {"start": 1014.0, "text": "We just need to replace the delimit() function"}, {"start": 1016.0, "text": "with a redaction function,"}, {"start": 1018.0, "text": "that replaces sensitive data"}, {"start": 1020.0, "text": "with the placeHolder string literal \"\"."}, {"start": 1023.0, "text": "Here's an important thing to remember."}, {"start": 1026.0, "text": "The transformed entries are scoped to the current inference iteration only."}, {"start": 1030.0, "text": "This means that these modifications will not be visible to the next inference call."}, {"start": 1034.0, "text": "You must apply them again."}, {"start": 1036.0, "text": "For expensive transformations that you want to persist,"}, {"start": 1039.0, "text": "use the @SessionProperty annotation."}, {"start": 1041.0, "text": "This lets you apply stateful transformations to your session history."}, {"start": 1045.0, "text": "See the documentation for details."}, {"start": 1048.0, "text": "Ok, so we saw how lifecycle event modifiers"}, {"start": 1051.0, "text": "provide deterministic hooks for injecting security policy."}, {"start": 1055.0, "text": "But I did not cover all the modifiers."}, {"start": 1057.0, "text": "The framework provides many more,"}, {"start": 1059.0, "text": "that trigger at other critical points in the agentic loop."}, {"start": 1062.0, "text": "The framework also allows you to build your own profile modifiers,"}, {"start": 1066.0, "text": "and package them in reusable components."}, {"start": 1069.0, "text": "Do see the Foundation Models documentation to learn more about these"}, {"start": 1073.0, "text": "and many other powerful features."}, {"start": 1075.0, "text": "Alright, now let's change context to App Intents."}, {"start": 1079.0, "text": "App Intents let you integrate your app with Apple Intelligence"}, {"start": 1082.0, "text": "and rich system experiences like Siri, Spotlight, Shortcuts, and many more."}, {"start": 1086.0, "text": "For the rest of this talk,"}, {"start": 1087.0, "text": "I will assume you are familiar with the basics of App Intents and App Schemas."}, {"start": 1091.0, "text": "To learn more, check out these great sessions linked below."}, {"start": 1094.0, "text": "As a quick recap, when an App Intent adopts an intent schema,"}, {"start": 1098.0, "text": "it becomes available as a tool to the Siri model."}, {"start": 1101.0, "text": "For example, here our DeletePhotoIntent"}, {"start": 1103.0, "text": "adopts the deleteAssets schema from the photos domain."}, {"start": 1106.0, "text": "Conceptually, this adds our Delete Photo action into the Siri Toolbox."}, {"start": 1111.0, "text": "And this allows Siri to reason over our tool definition"}, {"start": 1114.0, "text": "and invoke it to service user queries."}, {"start": 1117.0, "text": "However, as it is the model which decides which intent to call,"}, {"start": 1121.0, "text": "a prompt injection attack can let an attacker misuse your app"}, {"start": 1125.0, "text": "for data exfiltration or other malicious goals."}, {"start": 1128.0, "text": "For example, here we are running with external context,"}, {"start": 1131.0, "text": "which may try to run our Delete Photo function without user intent."}, {"start": 1135.0, "text": "If such an attack succeeds,"}, {"start": 1137.0, "text": "and we don't have any other deterministic guardrails"}, {"start": 1139.0, "text": "in spite of Willy's vigorous warnings,"}, {"start": 1142.0, "text": "then there's a real possibility of data loss."}, {"start": 1145.0, "text": "Actions which have externally visible side effects or are destructive"}, {"start": 1149.0, "text": "are tempting targets for attackers."}, {"start": 1151.0, "text": "The App Intents system has a number of guardrails in place"}, {"start": 1154.0, "text": "to help developers mitigate such attacks."}, {"start": 1157.0, "text": "We will look at two of these: confirmations and lock-screen authentication."}, {"start": 1160.0, "text": "Let's start with confirmations."}, {"start": 1163.0, "text": "The system uses a risk-based, contextual confirmation mechanism."}, {"start": 1167.0, "text": "This automatically triggers confirmations on high-risk actions from your app."}, {"start": 1171.0, "text": "The risk of an action is determined by considering static action metadata"}, {"start": 1175.0, "text": "and the dynamic system state."}, {"start": 1178.0, "text": "When an intent is chosen, before execution,"}, {"start": 1180.0, "text": "the system invokes a Risk Evaluation system with intent's risk metadata."}, {"start": 1185.0, "text": "We will come back to this metadata later."}, {"start": 1188.0, "text": "The Risk Evaluation component also takes as input the dynamic state of the system."}, {"start": 1193.0, "text": "It combines both to determine the overall risk of this intent."}, {"start": 1197.0, "text": "If the risk is considered high, the user is asked for a confirmation."}, {"start": 1201.0, "text": "If the user confirms this action, normal control-flow continues"}, {"start": 1204.0, "text": "and the intent is executed."}, {"start": 1207.0, "text": "On the other hand, if the user declines,"}, {"start": 1209.0, "text": "execution is blocked, and the intent is never invoked."}, {"start": 1213.0, "text": "Now let's go back to risk metadata."}, {"start": 1216.0, "text": "Risk metadata is internal risk data that is assigned to all intents."}, {"start": 1221.0, "text": "It is based on the intent's side effects."}, {"start": 1223.0, "text": "Certain side effects are considered riskier than others."}, {"start": 1227.0, "text": "For example, intents that delete device state,"}, {"start": 1229.0, "text": "like our DeletePhotoIntent, can be considered high-risk."}, {"start": 1233.0, "text": "Intents which exfiltrate data may also cause damage"}, {"start": 1236.0, "text": "if executed in a poisoned context."}, {"start": 1239.0, "text": "And update intents that operate on shared content can also be risky."}, {"start": 1243.0, "text": "The system is more likely to trigger confirmations for high-risk tools."}, {"start": 1248.0, "text": "So, how is risk metadata associated with your App Intent?"}, {"start": 1252.0, "text": "The risk metadata is automatically assigned to an intent when it adopts a schema."}, {"start": 1257.0, "text": "You don't have to do anything extra."}, {"start": 1259.0, "text": "Technically, it is the schemas which have risk metadata associated with them."}, {"start": 1263.0, "text": "For example, the deleteAssets schema is used to delete photos,"}, {"start": 1267.0, "text": "and thus has a destructive side effect."}, {"start": 1269.0, "text": "And so our DeletePhotoIntent is assigned this destructive side effect too."}, {"start": 1274.0, "text": "But risk is subtle."}, {"start": 1276.0, "text": "Let's define a new intent that sets a brewing timer."}, {"start": 1279.0, "text": "This intent adopts the createTimer schema."}, {"start": 1282.0, "text": "What do we think about the risk of this schema?"}, {"start": 1285.0, "text": "On the face of it, it seems an attacker can not cause too much damage"}, {"start": 1288.0, "text": "by creating timers, so perhaps we don't need to confirm this action."}, {"start": 1293.0, "text": "But if we look further, the schema defines an optional String property"}, {"start": 1296.0, "text": "for a label for the timer."}, {"start": 1298.0, "text": "Now remember that it is the model"}, {"start": 1300.0, "text": "which determines the arguments for your intent."}, {"start": 1303.0, "text": "So a prompt injection can cause this label being set to an attacker controlled value."}, {"start": 1308.0, "text": "And a subsequent query to list timers, can then pull this attacker controlled data"}, {"start": 1312.0, "text": "into that context, thus poisoning the new context too."}, {"start": 1317.0, "text": "So it is not safe to entirely skip confirmation"}, {"start": 1320.0, "text": "in cases like createTimer."}, {"start": 1322.0, "text": "The system understands these in-between situations."}, {"start": 1325.0, "text": "This is where the dynamic system state we discussed earlier plays a role."}, {"start": 1329.0, "text": "This information is used to figure out if a confirmation is needed"}, {"start": 1333.0, "text": "in the current system context,"}, {"start": 1335.0, "text": "thus capturing the dynamic risk of this action."}, {"start": 1338.0, "text": "To recap, remember that the confirmation system"}, {"start": 1340.0, "text": "is contextual and risk-based."}, {"start": 1343.0, "text": "Your intents will inherit side effects from the intent schemas they adopt."}, {"start": 1347.0, "text": "And actions with risky side effects are more likely to be confirmed."}, {"start": 1352.0, "text": "Now let's look at lock screen authentication."}, {"start": 1355.0, "text": "As you know, you can interact with Siri on the lock screen,"}, {"start": 1358.0, "text": "without having to first unlock your device."}, {"start": 1360.0, "text": "This is great for accomplishing quick tasks,"}, {"start": 1362.0, "text": "or when your hands are occupied."}, {"start": 1364.0, "text": "But this also means that an attacker in physical possession of a locked device"}, {"start": 1368.0, "text": "can potentially invoke your intent via Siri."}, {"start": 1371.0, "text": "So if you are not careful, your app can be used by such an attacker"}, {"start": 1374.0, "text": "for data exfiltration or running malicious actions."}, {"start": 1378.0, "text": "The main mitigation against such a threat"}, {"start": 1380.0, "text": "is to request the user to unlock their device"}, {"start": 1382.0, "text": "before running risky actions."}, {"start": 1384.0, "text": "So let's see how authentication policy is defined on an App Intent."}, {"start": 1388.0, "text": "For custom App Intents, you can explicitly set authentication behaviour"}, {"start": 1392.0, "text": "by setting the authenticationPolicy property."}, {"start": 1395.0, "text": "For example, here, as our DeletePhotoIntent is destructive,"}, {"start": 1398.0, "text": "we want to ensure it doesn't run on a locked device."}, {"start": 1401.0, "text": "So we explicitly set authenticationPolicy property"}, {"start": 1405.0, "text": "to .requiresAuthentication."}, {"start": 1407.0, "text": "The situation is slightly different when your @AppIntent adopts an intent schema."}, {"start": 1412.0, "text": "Schemas have their own default authenticationPolicy."}, {"start": 1415.0, "text": "This policy is set internally, based on the sensitivity of each schema"}, {"start": 1419.0, "text": "and the data it handles."}, {"start": 1421.0, "text": "And similar to side effects,"}, {"start": 1422.0, "text": "your intent is automatically assigned the schema's default policy."}, {"start": 1426.0, "text": "But if you want, you can still explicitly override the default."}, {"start": 1429.0, "text": "The only constraint is that your policy has to be stricter."}, {"start": 1433.0, "text": "For example, let's assume the default policy"}, {"start": 1435.0, "text": "of the deleteAssets schema is .requiresAuthentication."}, {"start": 1439.0, "text": "Then as we don't explicitly set policy here,"}, {"start": 1441.0, "text": "our @AppIntent is assigned the same,"}, {"start": 1443.0, "text": "and will require authentication before running."}, {"start": 1447.0, "text": "But if we try to set a weaker policy,"}, {"start": 1449.0, "text": "we get a build error which helpfully tells us"}, {"start": 1452.0, "text": "the minimum allowed policy."}, {"start": 1454.0, "text": "So to recap, authentication is an important mitigation for lock screen attacks."}, {"start": 1459.0, "text": "Schemas have their own default authentication policies,"}, {"start": 1462.0, "text": "which get assigned to your App Intent."}, {"start": 1464.0, "text": "And you can override the schema policy, but only to make it stricter."}, {"start": 1468.0, "text": "So please go and review your intents with their lock screen behaviour in mind."}, {"start": 1472.0, "text": "Now back to Willy!"}, {"start": 1473.0, "text": "Quality stuff, Akshay!"}, {"start": 1475.0, "text": "To summarize, the next steps for your agentic application"}, {"start": 1479.0, "text": "consists of coming up with a threat model,"}, {"start": 1481.0, "text": "which requires finding sources of untrusted context in your prompt,"}, {"start": 1485.0, "text": "and then determining the risk level for each action"}, {"start": 1488.0, "text": "based on its side effects."}, {"start": 1490.0, "text": "With Akshay's guidance, you have some stepping stones"}, {"start": 1493.0, "text": "on how you can implement the best mitigations for your app"}, {"start": 1496.0, "text": "using the Foundation Models framework and the App Intents framework."}, {"start": 1500.0, "text": "Now let's raise the bar, and promptly inject your defenses!"}]}