2-691.pptx

Cortana Extensibility in Universal Windows Apps

1.0x

2-691.pptx

Created 2 years ago

Duration 0:00:00
lesson view count 173
Cortana Extensibility in Universal Windows Apps
Select the file type you wish to download
Slide Content
  1. Ed Doran, Ali Emami

    Slide 1 - Ed Doran, Ali Emami

    • Microsoft
    • Cortana Extensibility in Universal Windows Apps
    • 2-691
    • //build/ content is being presented by Microsoft Office Mix The video for this session will be available shortly
  2. Submit an evaluation for each Breakout Session you attend.

    Slide 2 - Submit an evaluation for each Breakout Session you attend.

    • BE ENTERED INTO THE DAILY PRIZE DRAWING!
  3. Slide 3

    • Cortana & what’s new
    • How Cortana helps Apps
    • Engage users
    • Get things done
    • Get discovered
    • Q&A
    • Agenda
  4. Cortana

    Slide 4 - Cortana

    • World’s most personal digital assistant helps you get things done throughout the day
    • A whole new natural, way of interacting with your PC
    • Available to millions of users acrossWindows 10 PCs, tablets, and phones
    • Integrated search across device, the Web and personal and professional clouds
    • A new opportunity for partners
  5. DESKTOP: TAP TO OPEN CORTANA HOME

    Slide 6 - DESKTOP: TAP TO OPEN CORTANA HOME

    • Loading animation
  6. DESKTOP: CORTANA HOME

    Slide 7 - DESKTOP: CORTANA HOME

    • Celebrate the Day examples
  7. 8

    Slide 8 - 8

  8. Engage usersGet things doneGet discovered

    Slide 11 - Engage usersGet things doneGet discovered

    • Voice Commands for Windows
    • Natural Language Interactions
    • Task Completion
    • App Discovery in Cortana
  9. Slide 12

    • Engage usersGet things doneGet discovered
    • Voice Commands for Windows
    • Natural Language Interactions
    • Task Completion
    • App Discovery in Cortana
  10. Universal Apps can register to handle Cortana voice commands directed at their app

    Slide 13 - Universal Apps can register to handle Cortana voice commands directed at their app

    • When users speak these commands, Cortana activates the app with the spoken context
    • Apps trigger scenarios based on this context
    • Available for Universal Windows Apps built with XAML and JavaScript
    • Voice Commands for Universal Windows Apps
    • Foreground App Activation with Voice Commands
  11. Foreground Voice Commands Architecture

    Slide 14 - Foreground Voice Commands Architecture

    • 2
    • Windows Speech
    • Platform API (on-device)
    • Call
    • Text
    • Remind
    • Email
    • Voice Command
    • 3
    • 6
    • 4
    • 5
    • X
    • Adventure Works
    • Voice Command Definition (VCD)
    • 7
    • 1
    • Microsoft Speech
    • Platform Service (cloud)
    • Adventure Works, show my trip to London
    • Application Code
  12. Voice Commands: Foreground App Activation

    Slide 15 - Voice Commands: Foreground App Activation

  13. Slide 16

    • <?xml version="1.0" encoding="utf-8"?>
    • <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">
    • <CommandSet xml:lang="en-us" Name=“en-us-CommandSet">
    • <AppName> Adventure Works </AppName>
    • <Example> Show my trip to San Francisco </Example>
    • <Command Name=“showTripToDestination">
    • <Example>Show my trip to San Francisco</Example>
    • <ListenFor>Show [my] trip to {destination}</ListenFor>
    • <Feedback>Looking for trip to {destination}</Feedback>
    • <Navigate/>
    • </Command>
    • Voice Command Definition
  14. Slide 17

    • <?xml version="1.0" encoding="utf-8"?>
    • <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">
    • <CommandSet xml:lang="en-us" Name=“en-us-CommandSet">
    • <AppName> Quick Start </AppName>
    • <Example> Ask me something </Example>
    • <Command Name=“showTripToDestination">
    • <Example>Show my trip to San Francisco</Example>
    • <ListenFor>show [my] trip to {destination}</ListenFor>
    • <Feedback>Looking for trip to {destination}</Feedback>
    • <Navigate/>
    • </Command>
    • Voice Command Definition
    • <CommandSet xml:lang="en-us" Name=“en-us-CommandSet">
    • Command Set
    • Set of commands in the definition for a given language
  15. Slide 18

    • <?xml version="1.0" encoding="utf-8"?>
    • <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">
    • <CommandSet xml:lang="en-us" Name=“en-us-CommandSet">
    • <AppName> Adventure Works</AppName>
    • <Example> Ask me something </Example>
    • <Command Name=“showTripToDestination">
    • <Example>Show my trip to San Francisco</Example>
    • <ListenFor>show [my] trip to {destination}</ListenFor>
    • <Feedback>Looking for trip to {destination}</Feedback>
    • <Navigate/>
    • </Command>
    • Voice Command Definition
    • AppName
    • Your Application’s name, or short form of same
    • Users can use either to address your application
    • <AppName> Adventure Works </AppName>
  16. Slide 19

    • <?xml version="1.0" encoding="utf-8"?>
    • <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">
    • <CommandSet xml:lang="en-us" Name=“en-us-CommandSet">
    • <AppName> Adventure Works </AppName>
    • <Example> Show my trips </Example>
    • <Command Name=“showTripToDestination">
    • <Example>Show my trip to San Francisco</Example>
    • <ListenFor>show [my] trip to {destination}</ListenFor>
    • <Feedback>Looking for trip to {destination}</Feedback>
    • <Navigate/>
    • </Command>
    • Voice Command Definition
    • <Example> Show my trip to San Francisco </Example>
    • Example
    • Top level example of what a user can say
    • “Advertised” in Cortana’s “What can I do?” UX
  17. Slide 20

    • <Command Name=“showTripToDestination">
    • <Example>Show my trip to San Francisco</Example>
    • <ListenFor>show [my] trip to {destination}</ListenFor>
    • <Feedback>Looking for trip to {destination}</Feedback>
    • <Navigate/>
    • </Command>
    • Voice Command Definition
    • Next Steps:
    • Register VCD with Windows.ApplicationModel.VoiceCommands.VoiceCommandDefinitionManager
    • Override App.OnActivated() and launch your scenario based on context passed as name/value pairs
    • <Command Name=“showTripToDestination">
    • <Example>Show my trip to San Francisco</Example>
    • <ListenFor>show [my] trip to {destination}</ListenFor>
    • <Feedback>Looking for trip to {destination}</Feedback>
    • <Navigate/>
    • </Command>
    • Command
    • Logical unit of user “intent”. Contains …
    • … What the user says to Cortana
    • … What Cortana says in response
    • … What action Cortana will perform, i.e.
    • <Navigate/> element indicates it is a foreground voice command
  18. Windows Web Apps can take advantage of VCD-based App activation without writing ANY native platform code.

    Slide 21 - Windows Web Apps can take advantage of VCD-based App activation without writing ANY native platform code.

    • Enable it by dropping a VCD file on your website, add a meta tag to the HTML pointing to the VCD
    • Voice Commands for Windows Web Apps
    • Foreground App Activation with Voice Commands for Windows Web Apps.
  19. Cortana + Windows Web Apps

    Slide 22 - Cortana + Windows Web Apps

  20. Slide 23

    • Engage usersGet things doneGet discovered
    • Voice Commands for Windows
    • Natural Language Interactions
    • Task Completion
    • App Discovery in Cortana
  21. Authoring Natural Language Voice  Commands in Visual Studio

    Slide 24 - Authoring Natural Language Voice Commands in Visual Studio

  22. Slide 25

    • Engage usersGet things doneGet discovered
    • Voice Commands for Windows
    • Natural Language Interactions
    • Task Completion
    • App Discovery in Cortana
  23. Task Completion with Voice Commands

    Slide 26 - Task Completion with Voice Commands

    • Apps are empowered to integrate high-value tasks directly into the Cortana experience through new Cortana APIs supporting:
    • Task workflow events
    • UI templates
    • Speech interactivity model
    • Here are your upcoming trips.
    • May 14th 2015
    • Las Vegas
    • May 28th 2015
    • San Diego
    • Go to Adventure Works
    • August 24th 2015
    • Phoenix
    • Content Area
    • App GUI Text
    • App Icon
    • Link to your app
    • App Screen
    • Text-To-Speech (TTS)
    • Content Tile
    • A
    • Here are your upcoming trips.
  24. Task Completion with Voice Commands

    Slide 27 - Task Completion with Voice Commands

  25. Slide 28

    • Background Voice Commands Architecture
    • 2
    • Windows Speech
    • Platform API (on-device)
    • Call
    • Text
    • Remind
    • Email
    • Voice Command
    • 3
    • 6
    • 4
    • 5
    • VCD
    • 9
    • Isolated Storage
    • Voice Command
    • Background Service
    • Implements
    • Windows.
    • AppModel.Background.
    • IBackgroundTask
    • Render XAML
    • 7
    • 1
    • App Web Service
    • Microsoft Speech
    • Platform Service (cloud)
    • A
    • Go to Adventure Works
    • Here is your trip to
    • London
    • Adventure Works, show my trip to London
    • Application Code
    • 8
    • 8
  26. Cancelled this trip.

    Slide 29 - Cancelled this trip.

    • May 14th 2015
    • Vegas Tech Conference
    • Go to Adventure Works
    • Cancelling this trip…
    • Cancel
    • May 14th 2015
    • Vegas Tech Conference
    • Go to Adventure Works
    • Cancel this trip?
    • Listening…
    • May 14th 2015
    • Vegas Tech Conference
    • Yes
    • No
    • Which one do you want to cancel?
    • May 14th 2015
    • Vegas Tech Conference
    • May 14th 2015
    • Party in Phoenix
    • Listening…
    • Confirmation
    • Progress
    • Completion
    • Disambiguation
    • A
    • A
    • A
    • A
    • Which ‘Vegas’ trip do you wanna cancel?
    • Vegas Tech Conference
    • Yes
    • Do you wanna cancel ‘Vegas Tech Conference’?
    • Cancelling this trip…
    • I’ve cancelled it. [success-earcon]
    • Task Completion Workflow with Background Voice Commands
  27. VCD Event APIs

    Slide 30 - VCD Event APIs

    • Core Class: Windows.ApplicationModel.VoiceCommands.VoiceCommandServiceConnection
    • API
    • Used To…
    • RequestDisambiguationAsync(VoiceCommandResponse)
    • Request a disambiguation screen in Cortana
    • RequestConfirmationAsync(VoiceCommandResponse)
    • Request a confirmation screen in Cortana
    • RequestAppLaunchAsync(VoiceCommandResponse)
    • Request an app launch to complete the task
    • ReportProgressAsync(VoiceCommandResponse)
    • Inform Cortana on status of an in-progress task.
    • ReportSuccessAsync(VoiceCommandResponse)
    • Inform Cortana the task completed successfully and to render the completion screen
    • ReportFailureAsync(VoiceCommandResponse)
    • Inform Cortana the task failed to complete and to render the task failure screen
    • VoiceCommandCompleted
    • Event that Cortana notifies to end the interaction
  28. VoiceCommandResponse

    Slide 31 - VoiceCommandResponse

    • VoiceCommandResponse
    • {
    • VoiceCommandUserMessage Message;
    • VoiceCommandUserMessage RepeatMessage;
    • List<VoiceCommandContentTile> VoiceCommandContentTiles;
    • string AppLaunchArgument;
    • }
    • Icon, Title, Line1, Line2
    • Which one do you want to add to your trip?
    • Listening…
    • Title Name One
    • Long text lorem ipsum
    • Title Name Two
    • Long text lorem ipsum
    • Title Name Three
    • Long text lorem ipsum
    • Title Name Four
    • Long text lorem ipsum
    • Title Name Five
    • Long text lorem ipsum
    • Go to Adventure Works
    • A
  29. VoiceCommandContentTile

    Slide 32 - VoiceCommandContentTile

    • VoiceCommandContentTile
    • {
    • string Title;
    • string TextLine1;
    • string TextLine2;
    • string TextLine3;
    • Windows.Storage.StorageFile Icon;
    • VoiceCommandContentTileType ContentTileType;
    • string AppLaunchArgument;
    • string AppContext;
    • }
    • Which one do you want to add to your trip?
    • Title Name
    • Text Line One
    • Text Line Two
    • Text Line Three
    • A
  30. Check in at “CoffeShop on 5th Ave?”

    Slide 33 - Check in at “CoffeShop on 5th Ave?”

    • Listening…
    • Cancel this trip?
    • Listening…
    • May 14th 2015
    • Vegas Tech Conference
    • Yes
    • No
    • A
    • I’ll get MyPlace on that.
    • Search “Check in on MyPlace”
    • MyPlace
    • Looking for your profile…
    • Cancel
    • You’re checked in.
    • 2044 5th Ave
    • Seattle, WA 98001
    • CoffeeShop
    • Go to MyPlace
    • Looking for your profile…
    • Check in on MyPlace
    • Yes
    • Do you want to check in at CoffeeShop on 5th Ave?
    • You’re checked in. (success-earcon)
    • Check in
    • Yes
    • No
  31. Change budget to $200?

    Slide 34 - Change budget to $200?

    • Listening…
    • Cancel this trip?
    • Listening…
    • May 14th 2015
    • Vegas Tech Conference
    • Yes
    • No
    • A
    • I’ll get MyBudget on that.
    • Search “Change my groceries budget to $200 on MyBudget”
    • MyBudget
    • Looking for Groceries Budget…
    • Cancel
    • Your budget was updated.
    • Go to MyBudget
    • Looking for groceries budget…
    • Change my groceries budget to$200 on MyBudget
    • Yes
    • Change groceries budget to $200?
    • Your budget was updated. (success-earcon)
    • Change budget
    • Yes
    • No
    • $200
    • Groceries Budget
    • $200
    • Groceries Budget
  32. Which Happy do you want to add?

    Slide 35 - Which Happy do you want to add?

    • By Artist Name
    • Happy One
    • Listening…
    • By Artist Name
    • Happy Two
    • By Artist Name
    • Happy Three
    • Cancel this trip?
    • Listening…
    • May 14th 2015
    • Vegas Tech Conference
    • Yes
    • No
    • A
    • I’ll get MyMusic on that.
    • Search “Add a song Happy to my MyMusic playlist”
    • MyMusic
    • Looking for Happy…
    • Cancel
    • This is added to your playlist.
    • By Artist Name
    • Happy one
    • Go to MyMusic
    • Looking for Happy…
    • Add a song Happy to my MyMusic playlist
    • Happy one
    • Which Happy do you want to add to your playlist?
    • Happy one was added to your playlist(success-earcon)
    • Add song to playlist
  33. Searching large datasets is a common Voice Command pattern

    Slide 36 - Searching large datasets is a common Voice Command pattern

    • How should apps handle free-text search terms?
    • Searching over large, multi-faceted datasets at scale is hard.
    • Consider integrating your Cortana Voice Commands with a search backend to:
    • Search quickly across large data stores
    • Return ranked results based on relevance governed by your own business rules
    • Handle linguistics appropriately (e.g. inflections, accents)
    • Handling free-text searches
  34. Search-as-a-service in the Azure platform

    Slide 37 - Search-as-a-service in the Azure platform

    • Four steps to a working search backend
    • Provision a service, create an index, load data into it, search
    • Start at http://aka.ms/azsearchflow, there’s a free option that’s great for experimentation
    • Provides building blocks for great search experiences
    • Linguistics for 50 languages
    • Cortana + Azure Search as a backend
    • Azure Search
  35. Demo: Cortana + Azure Search

    Slide 38 - Demo: Cortana + Azure Search

  36. Slide 39

    • Engage usersGet things doneGet discovered
    • Voice Commands for Windows
    • Natural Language Interactions
    • Task Completion
    • App Discovery in Cortana
  37. App Search in Cortana

    Slide 40 - App Search in Cortana

  38. Deepen app engagement even further by marking up entities and actions on your web site.

    Slide 41 - Deepen app engagement even further by marking up entities and actions on your web site.

    • App Linking
    • Improve app discoverability by linking pages in your app to corresponding pages in your web site.
    • Users can discover your app and launch it to a specific location directly from search results.
    • Users can take action using your app directly from the search results or from Cortana.
    • For more information on app linking, see session 727
    • Coming soon to Windows 10
  39. App Linking Demo

    Slide 42 - App Linking Demo

  40. Give users quick access to your app functionality through app activation with Voice Commands.

    Slide 43 - Give users quick access to your app functionality through app activation with Voice Commands.

    • Add natural language elements to your app using Voice Commands.
    • For WWAs, take advantage of Voice Commands to integrate easily with Cortana.
    • Help users complete your app’s high-value tasks more quickly by integrating them with Cortana using background voice commands.
    • Leverage AppLinks.org and Schema.org website markup now to increase discoverability on the web now, and coming soon to Cortana.
    • Call to Action
  41. Must see talks

    Slide 44 - Must see talks

    • Session Title
    • Speakers
    • Session Code
    • Cortana and Speech Platform In-depth
    • Adina Trufinescu
    • Khuram Shahid
    • 716
    • Hosted Web Apps and Web Platform Innovations
    • Jeff Burtoft
    • Kiril Seksenov
    • 665
    • AllJoyn: Building Universal Windows Apps that Discover, Connect, and Interact with Other Devices and Cloud Services Using AllJoyn
    • Gavin Gear
    • Brian Rockwell
    • 623
    • Using Azure Search to Build Great Search Experiences in Mobile and Web Apps
    • Pablo Castro
    • 745
    • Build with Bing: Use Bing APIs and Tools to Enrich Your Apps and Make Them More Discoverable
    • Andre Furtado
    • Vincent Wehren
    • 727
  42. Appendix

    Slide 46 - Appendix