Preview 3-716.pptx

Cortana Extensibility and Speech Platform In-Depth

1.0x

Preview 3-716.pptx

Created 2 years ago

Duration 0:00:00
lesson view count 163
Cortana Extensibility and Speech Platform In-Depth
Select the file type you wish to download
Slide Content
  1. Adina Trufinescu - Principal Program Manager

    Slide 1 - Adina Trufinescu - Principal Program Manager

    • Khuram Shahid - Senior Developer
    • Cortana Extensibility and Speech Platform In-Depth
    • 3-716
    • //build/ content is being presented by Microsoft Office Mix The video for this session will be available shortly
  2. Adina Trufinescu - Principal Program Manager

    Slide 2 - Adina Trufinescu - Principal Program Manager

    • Khuram Shahid - Senior Developer
    • Cortana Extensibility and Speech Platform In-Depth
    • 3-716
  3. Slide 3

    • Hey Cortana, set a reminder to ….
    • Hey Cortana, send email to Jonas Brandell ….
    • Hey Cortana, call Jonas Brandell
    • Hey Cortana, text Jonas Brandell …
  4. Slide 4

    • Hey Cortana, what’s up with Jonas Brandell on Facebook
    • Hey Cortana, create a station for Hozier on Pandora
    • Hey Cortana, send a message to Jonas Brandell on Viber
    • Hey Cortana, Tweet flying into San Francisco tonight
  5. Differentiate your app with voice commands

    Slide 5 - Differentiate your app with voice commands

    • Delight your users with natural interactions
    • Increase user engagement
    • Increase productivity
    • Increase discovery
    • Developer Opportunity
  6. Voice Commands in the Foreground

    Slide 6 - Voice Commands in the Foreground

    • Speech Recognition in your app
    • Voice Commands in the Background
    • Agenda
  7. Voice Commands in the Foreground

    Slide 7 - Voice Commands in the Foreground

  8. Windows Universal Apps can register voice commands for users to say to Cortana

    Slide 8 - Windows Universal Apps can register voice commands for users to say to Cortana

    • Cortana launches the app to complete the task
    • Available for Windows Universal Apps built with XAML and JavaScript
    • Voice Commands for Universal Windows Apps
    • Foreground
    • Voice Commands
  9. Demo:Voice Commands for Adventure Works

    Slide 9 - Demo:Voice Commands for Adventure Works

  10. Foreground Voice Commands Architecture

    Slide 10 - Foreground Voice Commands Architecture

    • 2
    • Windows Speech
    • Platform
    • Call
    • Text
    • Remind
    • Email
    • Voice Command
    • 3
    • 6
    • 4
    • 5
    • X
    • Application
    • Voice Command Definition
    • 7
    • 1
    • Microsoft Speech
    • Recognition Service
    • Adventure Works, show my trip to London
  11. Foreground Voice Commands - How To

    Slide 11 - Foreground Voice Commands - How To

    • Create the Voice Command Definition (VCD)
    • Register the VCD XML on App Startup
    • Handle Voice Command Activation
  12. Determine the task to enable and what should the user say

    Slide 12 - Determine the task to enable and what should the user say

    • Start a chat with {contact}
    • Send a message to {contact}
    • Determine what Cortana should say as the app is launched
    • Starting a chat with {contact}
    • Provide an example of what the user can say for Cortana to show on “What can I do?”
    • Start a chat with Jonas
    • Voice Command Intent
  13. Users need to say the app name as part of the voice command

    Slide 13 - Users need to say the app name as part of the voice command

    • Viber, start a chat with Jonas
    • Facebook, what’s up with Jonas
    • Adventure Works, show my trip to London
    • By default uses the app’s display name
    • Provide a alternate such as short form when the app name is long or hard to pronounce
    • Voice Command & App Name
  14. Explicitly specify the app name where feels most natural

    Slide 14 - Explicitly specify the app name where feels most natural

    • Start a chat with Jonas on/with/using Viber
    • What’s up with Jonas on Facebook
    • Show my Adventure Works trip to London
    • Increase the recognition accuracy by providing a phonetically-rich phrase
    • Don’t: Get Jonas on Viber
    • Do: Start a chat with Jonas on Viber
    • Avoid using a Cortana reserved command, e.g. Call, Text, Start, Open, Find. If you have to, use the app name as a prefix
    • Viber, text Jonas I’m on my way
    • Voice Commands Guidance
  15. Voice Command Registration

    Slide 15 - Voice Command Registration

    • Register the Voice Command Definition on app startup
    • Use VoiceCommandDefinitionManager API in Windows.ApplicationModel.VoiceCommands namespace
    • Update the VCD with application data
  16. Handle the Voice Command Activation

    Slide 16 - Handle the Voice Command Activation

    • Override App.OnActivated method
    • Check for ActivationKind.VoiceCommand
    • Use VoiceCommandActivatedEventArgs.Result to determine what the user said
    • Navigate to page and continue the interaction
  17. Demo:Code walkthrough for“Show my trip to London for AdventureWorks”

    Slide 17 - Demo:Code walkthrough for“Show my trip to London for AdventureWorks”

  18. Websites can be submitted to Windows Store

    Slide 18 - Websites can be submitted to Windows Store

    • Cortana can launch Web-Hosted Apps with Voice Commands
    • Same Voice Command Definition schema and activation model as for JavaScript apps
    • Voice Commands for Web-Hosted Apps
    • Voice Commands for Web-Hosted Applications
  19. Voice Activation for Web-Hosted Apps

    Slide 19 - Voice Activation for Web-Hosted Apps

    • 2
    • Windows Speech
    • Platform
    • Call
    • Text
    • Remind
    • Email
    • Voice Command
    • 3
    • 6
    • 4
    • 5
    • 1
    • 7
    • X
    • Application
    • Package.appxmanifest
    • default.html
    • <meta name="msapplication-cortanavcd" content="https://contoso.com/vcd.xml"/>
    • <script src="voice.js"></script>
    • Vcd.xml
    • Microsoft Speech
    • Recognition Service
    • Contoso.com
    • /default.html
    • if (typeof Windows != 'undefined')
    • {
    • var activation = Windows.ApplicationModel.Activation;
    • Windows.UI.WebUI.WebUIApplication.addEventListener("activated", function (args)
    • {
    • if (args.kind === activation.ActivationKind.voiceCommand)
    • {
    • var speechRecognitionResult = args.result;
    • var properties = speechRecognitionResult.SemanticInterpretation.Properties;
    • //...
    • }
    • });
    • <Application
    • Id="App"
    • StartPage="https://flightarcade.azurewebsites.net">
    • <uap:VisualElements
    • ...
    • </uap:VisualElements>
    • <uap:ApplicationContentUriRules>
    • <uap:Rule Type="include" WindowsRuntimeAccess="all“ Match="https://flightarcade.azurewebsites.net"/>
    • </uap:ApplicationContentUriRules>
    • </Application>
    • Package.appxmanifest
    • Voice.js
  20. Voice Commands for Web-Hosted Apps

    Slide 20 - Voice Commands for Web-Hosted Apps

    • Create the Voice Command Definition and host on web service
    • Register the VCD on the default HTML page
    • Handle Voice Command Activation in .js
  21. Demo:Voice Activation for Web-Hosted App

    Slide 21 - Demo:Voice Activation for Web-Hosted App

  22. Speech Recognition in your app

    Slide 22 - Speech Recognition in your app

  23. Check if the interaction was input through voice or text

    Slide 23 - Check if the interaction was input through voice or text

    • For voice, use Speech to continue the user interaction in the app
    • Use Windows.Media.SpeechSynthesis APIs to talk to the user
    • Use Windows.Media.SpeechRecognition for speech recognition
    • In-App Speech Recognition
  24. Use SpeechRecognizer.RecognizeWithUIAsync to present the user with System listening GUI that can be customized

    Slide 24 - Use SpeechRecognizer.RecognizeWithUIAsync to present the user with System listening GUI that can be customized

    • Use SpeechRecognizer.RecognizeAsync for on demand recognition and provide a listening UI experience
    • System predefined grammars: Dictation, Web Search, Form Filling
    • App grammars: a list of strings or SRGS file
    • On-Demand Speech Recognition
  25. SpeechRecognizer.ContinuousRecognitionSessionenables listening continuously

    Slide 25 - SpeechRecognizer.ContinuousRecognitionSessionenables listening continuously

    • Can be used with Dictation grammar for scenarios where users speak for long periods
    • Can be used with app-provided grammars for always-listening in-app commands
    • Continuous Speech Recognition
  26. Demo: Adventure Works walkthrough for Speech Recognition

    Slide 26 - Demo: Adventure Works walkthrough for Speech Recognition

  27. Voice Commands in the Background

    Slide 27 - Voice Commands in the Background

  28. Demo:Voice Commands in the Background

    Slide 28 - Demo:Voice Commands in the Background

  29. Universal App can register voice commands to run in an AppService

    Slide 29 - Universal App can register voice commands to run in an AppService

    • AppService takes action on the voice command
    • May use Cortana’s canvas and voice to present feedback to the user
    • Available for Store Apps built with XAML and JavaScript
    • Voice Commands in the Background
    • Background Voice Activation
  30. Slide 30

    • Background Voice Commands Architecture
    • 2
    • Windows Speech
    • Platform
    • 3
    • 6
    • 4
    • 5
    • Isolated Storage
    • Voice Command
    • App Service
    • Render XAML
    • 7
    • Web Service
    • Microsoft Speech
    • Recognition Service
    • Adventure Works, what trips do I have coming up?
    • VCD
    • Your App
    • 8
    • Go to Adventure Works
    • Here are your upcomingtrips
    • A
    • Yosemite
    • August 15th
    • San Francisco
    • May 29th
    • 9
    • 1
  31. Background Voice Commands - How To

    Slide 31 - Background Voice Commands - How To

    • Create the Voice Command Definition (VCD)
    • Register the VCD XML on App Startup
    • Handle the Voice Command Activation in an App Service
  32. <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">

    Slide 32 - <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">

    • <CommandSet xml:lang="en-us" Name=“en-us-CommandSet">
    • <AppName> Adventure Works </AppName>
    • <Example> Show my trip to San Francisco </Example>
    • <Command Name=“showTripToDestination">
    • <Example>Show my trip to San Francisco</Example>
    • <ListenFor >show [my] trip to {destination}</ListenFor>
    • <Feedback>Looking for trip to {destination}</Feedback>
    • <VoiceCommandService Target=“AdventureWorksService"/>
    • </Command>
    • Voice Command Definition
    • Register the Voice Command Definition on app startup
  33. <Extensions>

    Slide 33 - <Extensions>

    • <Extension
    • Category="windows.appService"
    • EntryPoint="AdventureWorks.AdventureWorksService">
    • <AppService Name=“AdventureWorksService"/>
    • </Extension>
    • </Extensions>
    • Create an AppService
    • Inherit from IBackgroundTask and implement Run()
    • Register the AppService in the package.appxmanifest
    • public async void Run(IBackgroundTaskInstance taskInstance)
    • See Session 765 For Details on AppServices
  34. using Windows.ApplicationModel.AppService;

    Slide 34 - using Windows.ApplicationModel.AppService;

    • using Windows.ApplicationModel.Background;
    • using Windows.ApplicationModel.VoiceCommands;
    • VoiceCommandDefinitionManager
    • VoiceCommandDefinition
    • VoiceCommandServiceConnection
    • VoiceCommand
    • VoiceCommandUserMessage
    • VoiceCommandContentTile
    • VoiceCommandResponse
    • WinRT Namespaces and APIs
  35. VoiceCommandServiceConnection

    Slide 35 - VoiceCommandServiceConnection

    • Request actions from Cortana:
    • RequestConfirmationAsync(…)
    • RequestDisambiguationAsync(…)
    • RequestAppLaunchAsync(…)
    • Send updates to Cortana:
    • ReportProgressAsync(…)
    • ReportFailureAsync(…)
    • ReportSuccessAsync(…)
    • Handle Events:
    • VoiceCommandCompleted
  36. Cortana Canvas Interactions from Background

    Slide 36 - Cortana Canvas Interactions from Background

    • Present UI on Cortana’s canvas
    • Speak a message using Cortana’s voice
    • Which trip would you like to cancel?
    • May 14th 2015
    • Las Vegas
    • May 28th 2015
    • San Diego
    • Go to Adventure Works
    • August 24th 2015
    • Phoenix
    • Content Area
    • App GUI Text
    • App Icon
    • Link to your app
    • App Screen
    • Text-To-Speech (TTS)
    • Content Tile
    • Which one do you wanna cancel?
    • A
  37. VoiceCommandResponse

    Slide 37 - VoiceCommandResponse

    • Which trip would you like to cancel?
    • May 14th 2015
    • Las Vegas
    • May 28th 2015
    • San Diego
    • Go to Adventure Works
    • August 24th 2015
    • Phoenix
    • Which one do you wanna cancel?
    • A
  38. Which trip would you like to cancel?

    Slide 38 - Which trip would you like to cancel?

    • May 14th 2015
    • Las Vegas
    • May 28th 2015
    • San Diego
    • Go to Adventure Works
    • August 24th 2015
    • Phoenix
    • Which one do you wanna cancel?
    • A
    • VoiceCommandResponse
    • VoiceCommandResponse
    • {
    • VoiceCommandUserMessage Message;
    • VoiceCommandUserMessage RepeatMessage;
    • List<VoiceCommandContentTile> VoiceCommandContentTiles;
    • string AppLaunchArgument;
    • }
  39. VoiceCommandContentTile

    Slide 39 - VoiceCommandContentTile

    • VoiceCommandContentTile
    • {
    • string Title;
    • string TextLine1;
    • string TextLine2;
    • string TextLine3;
    • Object AppContext;
    • string AppLaunchArgument;
    • Windows.Storage.StorageFile Image;
    • VoiceCommandContentTileType ContentTileType;
    • }
    • Which one do you want to add to your trip?
    • Title Name
    • Text Line One
    • Text Line Two
    • Text Line Three
    • A
  40. VoiceCommandContentTile – Text

    Slide 40 - VoiceCommandContentTile – Text

    • Listening…
    • Listening…
    • Icon, Title
    • Which one do you want to add to your trip?
    • Listening…
    • Icon, Title, Line1, Line2, Line3
    • Title Name One
    • Long text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod
    • Title Name One
    • Title Name Two
    • Title Name Three
    • Title Name Four
    • Title Name Two
    • Long text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod
    • Title Name Three
    • Long text lorem ipsum dolor sit amet, Consec tetuer amias Sed do eiusmod
    • Title Name Four
    • Go to Adventure Works
    • Which one do you want to add to your trip?
    • A
    • A
  41. VoiceCommandContentTile– Icons

    Slide 41 - VoiceCommandContentTile– Icons

    • 68x92xIcon, Title, Line1,
    • Line2, Line3
    • Which one do you want to add to your trip?
    • Listening…
    • Title Name One
    • Long text two lines lorem ipsum consec tetuer amias
    • Title Name Two
    • Long text two lines lorem ipsum consec tetuer amias
    • Title Name Three
    • Long text two lines lorem ipsum consec tetuer amias
    • Title Name Four
    • Long text two lines lorem ipsum consec tetuer amias
    • 240x140 Icon, Title, Line1, Line2, Line3
    • Which one do you want to add to your trip?
    • Listening…
    • Title Name One
    • One line text
    • Title Name Two
    • One line text
    • Long text two lines lorem ipsum consec tetuer amias odoran
    • 68x68 Icon, Title, Line1, Line2
    • Which one do you want to add to your trip?
    • Listening…
    • Title Name One
    • Long text lorem ipsum
    • Title Name Two
    • Long text lorem ipsum
    • Title Name Three
    • Long text lorem ipsum
    • Title Name Four
    • Long text lorem ipsum
    • Title Name Five
    • Long text lorem ipsum
    • Go to Adventure Works
    • A
    • A
    • A
  42. Demo: Code walkthrough for  “When’s my trip to Las Vegas with AdventureWorks”

    Slide 42 - Demo: Code walkthrough for “When’s my trip to Las Vegas with AdventureWorks”

  43. Which one do you want to cancel?

    Slide 43 - Which one do you want to cancel?

    • May 14th 2015
    • Vegas Tech Conference
    • August 18th 2015
    • Party in Phoenix
    • Listening…
    • Disambiguation
    • Vegas Tech Conference
    • A
    • Cancel this trip?
    • Listening…
    • May 14th 2015
    • Vegas Tech Conference
    • Yes
    • No
    • Confirmation
    • Yes
    • A
    • Cancelling this trip…
    • Cancel
    • May 14th 2015
    • Vegas Tech Conference
    • Go to Adventure Works
    • Progress
    • A
    • Cancelled this trip.
    • May 14th 2015
    • Vegas Tech Conference
    • Go to Adventure Works
    • Completion
    • A
    • Which ‘Vegas’ trip do you wanna cancel? Vegas Tech Conference or Party in Vegas.
    • Do you wanna cancel ‘Vegas Tech Conference’?
    • Cancelling this trip…
    • I’ve cancelled it. [success-earcon]
    • Cancel my Adventure Works trip
  44. Which one do you want to cancel?

    Slide 44 - Which one do you want to cancel?

    • May 14th
    • Vegas Tech Conference
    • August 18th
    • Party in Phoenix
    • Listening…
    • A
    • Which trip do you wanna cancel?
    • Disambiguation
    • Vegas Tech Conference
    • The first one
    • The one on August 18th
    • The user may select an option via speech or UI input.
    • Cortana automatically listens for utterances that uniquely identify a content tile.
    • RequestDisambiguationAsync() returns only after a selection has been made.
  45. Slide 45

    • Do you wanna cancel ‘Vegas Tech Conference’?
    • Confirmation
    • Yes
    • No
    • Always confirm before taking destructive or irreversible actions
    • Cancel this trip?
    • Listening…
    • May 14th 2015
    • Vegas Tech Conference
    • Yes
    • No
    • A
  46. Progress

    Slide 46 - Progress

    • Text To Speech is optional
    • You may send multiple progress updates. Updates are coalesced where possible.
    • Send progress during long-running operations. If there are no updates for 5 seconds, Cortana may cancel the flow.
    • Cancelling this trip…
    • Cancel
    • May 15th 2015
    • Vegas Tech Conference
    • Go to Adventure Works
    • Cancelling this trip…
    • A
  47. Cortana Supplied Progress Screens

    Slide 47 - Cortana Supplied Progress Screens

    • Listening…
    • Working on that…
    • Cancel
    • I’ll get Adventure Works on that.
    • Adventure Works
    • Looking for trip to San Francisco
    • Cancel
    • Search “Cancel my Adventure Works trip to Vegas”
    • Looking for your trip to San Francisco…
    • A
    • A
    • May 14th 2015
    • Vegas Tech Conference
    • Go to Adventure Works
    • Shown if launching the App service, or the initial response takes > 500ms
    • Feedback string from VCD, if any, is spoken out loud.
    • Shown after RequestConfirmation or RequestDisambiguation if the App does not respond quickly (500ms)
    • Content derived from the previous Confirmation/Disambiguation Response
  48. Ask the user to confirm before taking action on the voice command

    Slide 48 - Ask the user to confirm before taking action on the voice command

    • Ask the user to disambiguate if there are more than one action or target of the action
    • Show progress as the voice command is handled
    • Interaction Summary:
    • Canvas Interactions from Background
  49. Slide 49

    • Cortana
    • Your AppService
    • Adventure Works, cancel my trip!
    • I’ll get Adventure Works on that!
    • Inspect Voice Command
    • Las Vegas
    • yes
    • Invoke IBackgroundTask::Run()
    • RequestDisambiguationAsync
    • RequestConfirmationAsync
    • VoiceCommandConfirmationResult
    • VoiceCommandDisambiguationResult
    • Interaction Flow
    • Which one?
    • Las Vegas
    • Cancun
    • //build/
    • check SelectedItem
    • Are you sure?
    • yes no
    • Process request
    • (Talk to web service,
    • save data
    • etc.)
  50. Slide 50

    • Cortana
    • Your App
    • Adventure Works, cancel my trip!
    • I’ll get Adventure Works on that!
    • Inspect Voice Command
    • Which one?
    • Las Vegas
    • Cancun
    • //build/
    • Are you sure?
    • yes no
    • Las Vegas
    • yes
    • check SelectedItem
    • Invoke IBackgroundTask::Run()
    • RequestDisambiguationAsync
    • RequestConfirmationAsync
    • VoiceCommandConfirmationResult
    • VoiceCommandDisambiguationResult
    • Process request
    • (Talk to web service,
    • save data
    • etc.)
    • Canceling…
    • "Done!
    • Still canceling…
    • ReportSuccessAsync
    • ReportProgressAsync
    • ReportProgressAsync
    • foreground
    • Event: VoiceCommandCompleted
    • Teardown
    • App Launch
    • Event: VoiceCommandCompleted
    • UI Interaction
  51. Demo:Authoring & Debugging an App that supports background Voice Commands

    Slide 51 - Demo:Authoring & Debugging an App that supports background Voice Commands

  52. Be efficient. Less is more!

    Slide 52 - Be efficient. Less is more!

    • Be relevant. Provide information that’s relevant to the given task, content, and context.
    • Be clear. Avoid ambiguity. Use everyday language.
    • Be trustworthy. Present accurate information. Be transparent. Respect privacy.
    • Conversational Writing Guidance
    • Canvas Interactions from Background
  53. For each supported language, localize the CommandSet in the Voice Command Definition

    Slide 53 - For each supported language, localize the CommandSet in the Voice Command Definition

    • When launched, inspect the value of Language property of the VoiceCommandServiceConnection
    • Localize the dialog on the app provided screens to match Cortana’s language
    • Localization
    • Canvas Interactions from Background
  54. Create a Voice Command Definition

    Slide 54 - Create a Voice Command Definition

    • Register the VCD when the app starts
    • Handle the Voice Activation in the app or AppService
    • Voice Commands for Universal Apps - Recap
  55. Slide 55

    • for tasks that require complex interactions
    • for tasks that demand the user’s attention for a long time
    • Voice Command Handling Guidance
    • for simple tasks that don’t require additional input beyond Confirmation and Disambiguation
    • for tasks that complete within seconds
    • to initiate longer tasks that execute remotely
    • may use RequestAppLaunchAsync method to launch the app and complete remainder of the flow
    • Foreground
    • Background
  56. Launch and deep link into your app from Cortana

    Slide 56 - Launch and deep link into your app from Cortana

    • Extend Cortana with your app’s capabilities through voice commands in the background
    • Add speech recognition as an input modality to your app
    • References:
    • See AppServices Session 765
    • See Web-Hosted Apps Session 665
    • See Adventure Works Sample Code
    • See Design Guidelines for extending Cortana
    • Call to Action