Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile Apps




 http://www.bigbaldapps.com







 www.AgeofMobility.com
 @ActiveNick
2005-2014

Mobile Speech is a Responsibility

Accessibility
Medical
Conditions
Hands &
Eyes Free
Driving Apps
& Games
Learning &
Therapy
Travel Apps
with
Translation

Speech SDK
Features for Devs
Windows Phone Windows Store
iOS
(iPhone/iPad)
Android
(Phone/Tablet)
Built-in Personal
Assistant with
Speech
Cortana* N/A Siri* Google Now
Launcher*
Personal Assistant
Extensions in third-
party apps using
Voice Commands
Yes N/A No
(first-party apps
only)
No
(first-party apps
only)
Speech Synthesis
SDK for Devs
(Text-to-Speech)
Yes Yes Yes Yes
In-app Speech
Recognition SDK
for Devs
Yes Yes* No* Yes

<?xml version="1.0" encoding="utf-8"?>
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.1">
<CommandSet xml:lang="en-us" Name="englishCommands">
<CommandPrefix>MSDN</CommandPrefix>
<Example>How do I add Voice Commands to my application</Example>
<Command Name="FindText">
<Example>Find Install Voice Command Sets</Example>
<ListenFor>Search</ListenFor>
<ListenFor>Search for {dictatedSearchTerms}</ListenFor>
<ListenFor>Find</ListenFor>
<ListenFor>Find {dictatedSearchTerms}</ListenFor>
<Feedback>Search on MSDN</Feedback>
<Navigate Target="MainPage.xaml" />
</Command>
<Command Name="nlpCommand">
<ListenFor>{dictatedVoiceCommandText}</ListenFor>
<Feedback>Starting MSDN...</Feedback>
</Command>
<PhraseTopic Label="dictatedVoiceCommandText" Scenario="Dictation">
<Subject>MSDN</Subject>
</PhraseTopic>
<PhraseTopic Label="dictatedSearchTerms" Scenario="Search">
</PhraseTopic>
</CommandSet>
</VoiceCommands>

</Command>
</Command>
</PhraseTopic>
</PhraseTopic>
</CommandSet>
</VoiceCommands>

</Command>
</Command>
</PhraseTopic>
</PhraseTopic>
</CommandSet>
</VoiceCommands>
</Command>
/Command>

</Command>
</Command>
</PhraseTopic>
</PhraseTopic>
</CommandSet>
</VoiceCommands>

</Command>
</Command>
</PhraseTopic>
</PhraseTopic>
</CommandSet>
</VoiceCommands>
</PhraseTopic>
</PhraseTopic>

<Example>Find Voice Commands</Example>
<Example>Find Windows Phone</Example>
<ListenFor>Search {*}</ListenFor>
<ListenFor>Search for {listSearchTerms}</ListenFor>
<ListenFor>Find {*}</ListenFor>
<ListenFor>Find {listSearchTerms}</ListenFor>
</Command>
<PhraseList Label="listSearchTerms" Disambiguate="false">
<Item>Voice Commands</Item>
<Item>Windows Phone</Item>
</PhraseList>
</CommandSet>
</VoiceCommands>
</Command>
</PhraseTopic>
</CommandSet>
</VoiceCommands>
Windows Phone 8.0 Windows Phone 8.1
Simple Scenarios, Limited Accuracy More Powerful, Easier, More Accurate

<Example>Find Voice Commands</Example>
<Example>Find Windows Phone</Example>
<ListenFor>Search {*}</ListenFor>
<ListenFor>Search for {listSearchTerms}</ListenFor>
<ListenFor>Find {*}</ListenFor>
<ListenFor>Find {listSearchTerms}</ListenFor>
</Command>
<PhraseList Label="listSearchTerms" Disambiguate="false">
<Item>Voice Commands</Item>
<Item>Windows Phone</Item>
</PhraseList>
</CommandSet>
</VoiceCommands>
</Command>
</PhraseTopic>
</CommandSet>
</VoiceCommands>
Windows Phone 8.0 Windows Phone 8.1
Simple Scenarios, Limited Accuracy … and enables Natural Language

</Command>
</Command>
</PhraseTopic>
</PhraseTopic>
</CommandSet>
</VoiceCommands>

private async void
// SHOULD BE PERFORMED UNDER TRY/CATCH
Uri new ms-appx:///vcd.xml UriKind.Absolute
await
Windows Phone Silverlight Apps on Windows Phone 8.1
Windows Runtime Apps on Windows Phone 8.1
private async void
// SHOULD BE PERFORMED UNDER TRY/CATCH
Uri uriVoiceCommands = new Uri("ms-appx:///vcd.xml", UriKind.Absolute);
StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(uriVoiceCommands);
await VoiceCommandManager.InstallCommandSetsFromStorageFileAsync(file);

// Windows Phone Silverlight App, in MainPage.xaml.cs
protected override void OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)
{
base.OnNavigatedTo(e);
if (e.NavigationMode == System.Windows.Navigation.NavigationMode.New)
{
string recoText = null; // What did the user say? e.g. MSDN, "Find Windows Phone Voice Commands"
NavigationContext.QueryString.TryGetValue("reco", out recoText);
string voiceCommandName = null; // Which command was recognized in the VCD.XML file? e.g. "FindText"
NavigationContext.QueryString.TryGetValue("voiceCommandName", out voiceCommandName);
string searchTerms = null; // What did the user say, for named phrase topic or list "slots"? e.g. "Windows Phone Voice Commands"
NavigationContext.QueryString.TryGetValue("dictatedSearchTerms", out searchTerms);
switch (voiceCommandName) // What command launched the app?
{
case "FindText":
HandleFindText(searchTerms);
break;
case "nlpCommand":
HandleNlpCommand(recoText);
break;
}
}
}

{
{
{
case "FindText":
break;
case "nlpCommand":
break;
}
}
}

{
{
{
case "FindText":
break;
case "nlpCommand":
break;
}
}
}
// What did the user say? e.g. MSDN, "Find Windows Phone Voice Commands"
string recoText = null;
// Which command was recognized in the VCD.XML file? e.g. "FindText"
string voiceCommandName = null;
// What did the user say, for named phrase topic or list "slots"? e.g. "Windows
Phone Voice Commands"
string searchTerms = null;

{
{
{
case "FindText":
break;
case "nlpCommand":
break;
}
}
}
{
case "FindText":
break;
case "nlpCommand":
break;
}

// Windows Runtime App on Windows Phone 8.1, inside OnActivated override in App class
if (args.Kind == ActivationKind.VoiceCommand)
{
VoiceCommandActivatedEventArgs vcArgs = (VoiceCommandActivatedEventArgs)args;
string voiceCommandName = vcArgs.Result.RulePath.First(); // What command launched the app?
switch (voiceCommandName) // Navigate to right page for the voice command
{
case "FindText": // User said "find" or "search"
rootFrame.Navigate(typeof(MSDN.FindText), vcArgs.Result);
break;
case "nlpCommand": // User said something else
rootFrame.Navigate(typeof(MSDN.NlpCommand), vcArgs.Result);
break;
}
}

{
{
break;
break;
}
}
// Windows Runtime App on Windows Phone 8.1, inside OnActivated override
// in App class

{
{
break;
break;
}
}
// What command launched the app?
string voiceCommandName = vcArgs.Result.RulePath.First();
// Navigate to right page for the voice command
switch (voiceCommandName)

{
{
break;
break;
}
}
// Navigate to right page for the voice command
switch (voiceCommandName)
{
break;
break;
}

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo in FindText.xaml.cs
protected override void OnNavigatedTo(NavigationEventArgs e)
{
// Get recognition result from parameter passed in frame.Navigate call
SpeechRecognitionResult vcResult = e.Parameter as SpeechRecognitionResult;
if (vcResult!=null)
{
string recoText = vcResult.Text;
// Store the semantics dictionary for later use
IReadOnlyDictionary<string,IReadOnlyList<string>> semantics = vcResult.SemanticInterpretation.Properties;
string voiceCommandName = vcResult.RulePath.First();
if (voiceCommandName == "FindText")
{
// What did the user say, for named phrase topic or list "slots"? e.g. "Windows Phone Voice Commands"
if (semantics.ContainsKey("dictatedSearchTerms"))
{
HandleFindTextWithSearchTerms(semantics["dictatedSearchTerms"][0]);
}
else
{
HandleNoSearchTerms();
}
}
// Else handle other voice commands
} navigationHelper.OnNavigatedTo(e)
}

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In FindText.xaml.cs
{
if (vcResult!=null)
{
{
{
}
else
{
}
}
}
// Windows Runtime App on Windows Phone 8.1, inside
// OnNavigatedTo in FindText.xaml.cs
{
if (vcResult!=null)
{
IReadOnlyDictionary<string,IReadOnlyList<string>> semantics =
vcResult.SemanticInterpretation.Properties;

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In FindText.xaml.cs
{
if (vcResult!=null)
{
{
{
}
else
{
}
}
}
{
// What did the user say, for named phrase topic or list "slots"?
// e.g. "Windows Phone Voice Commands"
{
}
else
{
}
}

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo in NlpCommand.xaml.cs
{
// Check for null!
string commandMode = vcResult.SemanticInterpretation.Properties["commandMode"][0];
if (commandMode == "voice") // Did the user speak or type the command?
{
SpeakText(audioPlayer, String.Format("MSDN app heard you say {0}", vcResult.Text));
HandleNlpCommand(vcResult);
}
else if(commandMode=="text")
{
messageTextBox.Text = string.Format("Working on your request "{0}"", vcResult.Text);
}
}

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In NlpCommand.xaml.cs
{
// Check for null!
{
}
{
}
}
// Windows Runtime App on Windows Phone 8.1, in OnNavigatedTo in NlpCommand.xaml.cs
{

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In NlpCommand.xaml.cs
{
// Check for null!
{
}
{
}
}
{
SpeakText(audioPlayer, String.Format("MSDN voice heard you say {0}", vcResult.Text));
}
{
msgTextBox.Text= string.Format("Working on your request "{0}"", vcResult.Text);
}

private void HandleNlpCommand(string recoText, bool actSilently)
{
string action = null;
string navigateTo = null;
string searchFor = null;
recoText = recoText.ToLower();
if (recoText.Contains("go to ") || recoText.Contains("goto ") ||
recoText.Contains("find ") || recoText.Contains("search ") ||
recoText.Contains("show me "))
{
action = "navigate";
if (recoText.Contains("windows phone dev center"))
{
navigateTo = "http://dev.windowsphone.com";
}
}
else if (recoText.Contains("learn how to "))
{
action = "find";
searchFor = recoText.Substring(recoText.IndexOf("learn how to ") + 13);
}
else
{
action = "find";
searchFor = recoText;
}
switch (action)
{
case "find":
// ...

{
{
{
}
}
{
action = "find";
}
else
{
action = "find";
}
switch (action)
{
case "find":
// ...
{
{
}
}

{
{
{
}
}
{
action = "find";
}
else
{
action = "find";
}
switch (action)
{
case "find":
// ...
{
action = "find";
}

{
{
{
}
}
{
action = "find";
}
else
{
action = "find";
}
switch (action)
{
case "find":
// ...
else
{
action = "find";
}

// Windows Phone Silverlight App
// Synthesis
private async void SpeakText(string textToSpeak)
{
SpeechSynthesizer synthesizer = new SpeechSynthesizer();
await synthesizer.SpeakTextAsync(textToSpeak);
}
// Recognition
private async Task<SpeechRecognitionUIResult> RecognizeSpeech()
{
SpeechRecognizerUI recognizer = new SpeechRecognizerUI();
// One of three Grammar types available
recognizer.Recognizer.Grammars.AddGrammarFromPredefinedType(
"key1", SpeechPredefinedGrammar.WebSearch);
await recognizer.Recognizer.PreloadGrammarsAsync(); // Optional but recommended
// Put up UI and recognize user's utterance
SpeechRecognitionUIResult result = await recognizer.RecognizeWithUIAsync();
return result;
}
// Calling code uses result.RecognitionResult.Text or result.RecognitionResult.Semantics

// Synthesis
{
}
// Recognition
{
return result;
}
// Synthesis
{
}

// Synthesis
{
}
// Recognition
{
return result;
}
// Recognition
{
// Optional but recommended
await recognizer.Recognizer.PreloadGrammarsAsync();

// Synthesis
{
}
// Recognition
{
return result;
}
SpeechRecognitionUIResult result
= await recognizer.RecognizeWithUIAsync();
return result;

// Windows Phone Store App
// Synthesis

<MediaElement Name="audioPlayer" AutoPlay="True" .../>
// C# code behind
// Function to speak a text string
private async void SpeakText(MediaElement audioPlayer, string textToSpeak)
{
SpeechSynthesisStream ttsStream = await synthesizer.SynthesizeTextToStreamAsync(textToSpeak);
audioPlayer.SetSource(ttsStream, ""); // This starts the player because AutoPlay="True"
}

// Synthesis
// C# code behind
{
}
// Synthesis

// Synthesis
// C# code behind
{
}
// C# code behind
{
SpeechSynthesisStream ttsStream
= await synthesizer.SynthesizeTextToStreamAsync(textToSpeak);
audioPlayer.SetSource(ttsStream, "");
// This starts the player because AutoPlay="True"
}

// Recognition
private async Task<SpeechRecognitionResult> RecognizeSpeech()
{
SpeechRecognizer recognizer = new SpeechRecognizer();
// One of three Constraint types available
SpeechRecognitionTopicConstraint topicConstraint
= new SpeechRecognitionTopicConstraint(SpeechRecognitionScenario.WebSearch, "MSDN");
recognizer.Constraints.Add(topicConstraint);
await recognizer.CompileConstraintsAsync(); // Required
SpeechRecognitionResult result = await recognizer.RecognizeWithUIAsync();
return result;
}
// Calling code uses result.RecognitionResult.Text or
// result.RecognitionResult.SemanticInterpretation

// Recognition
{
return result;
}
= new SpeechRecognitionTopicConstraint(
SpeechRecognitionScenario.WebSearch, "MSDN");

// Recognition
{
return result;
}
SpeechRecognitionResult result
= await recognizer.RecognizeWithUIAsync();
return result;
}


 http://aka.ms/CortanaDev

 http://channel9.msdn.com/Events/Build/2014/2-530

 http://www.bing.com/blogs/site_blogs/b/search/archive/2014/04/14/integrating-store-apps-with-cortana-in-windows-
phone-8-1.aspx

 http://www.bing.com/dev/en-us/speech

 http://msdn.microsoft.com/en-us/magazine/jj721592.aspx

 http://blogs.msdn.com/b/robch


 http://www.windowsphone.com/en-us/store/app/msdn-voice-search/aed09797-
b424-4030-a04d-5ec8ec63847f
 http://code.msdn.microsoft.com/wpapps/MSDN-Voice-Search-for-95c16d92

 https://voicecommands.codeplex.com

 http://code.msdn.microsoft.com/wpapps/Voice-Commands-for-Windows-2f808ac6

 http://code.msdn.microsoft.com/windowsapps/Speech-synthesis-sample-6e07b218

 http://code.msdn.microsoft.com/windowsapps/Bing-Speech-Recognition-210c8978
 http://code.msdn.microsoft.com/windowsapps/Speech-Recognition-19c452b4


 https://developer.apple.com/technologies/ios/accessibility.html

 https://developer.apple.com/library/ios/documentation/AVFoundation/Reference/AVSpeechSynthesizer_Ref/
Reference/Reference.html

 http://nshipster.com/avspeechsynthesizer

 DEVFRIGHT Blog Post: http://www.devfright.com/quick-look-avspeechsynthesizer-class/


 https://play.google.com/store/apps/details?id=com.google.android.launcher

 http://developer.android.com/reference/android/speech/package-summary.html

 http://developer.android.com/reference/android/speech/tts/package-summary.html

http://channel9.msdn.com/Events/Build/2014/

http://channel9.msdn.com/Events/Build/2014















Get Ready to Become a Windows Phone Developer
dev.windowsphone.com
Learn More About Windows Phone Dev via Official Microsoft Videos
http://aka.ms/wp81js
http://aka.ms/wp81devbegin
Check Out Additional Learning Resources
www.pluralsight.com/training/Courses#windows-phone
www.developer.nokia.com
Download Additional Resources & Become an Expert
phone.codeplex.com
http://bit.ly/nokiadevoffers
68
1
2
3
4

 www.AgeofMobility.com
 dev.windowsphone.com
 dev.windows.com
 blogs.windows.com/windows_phone/b/wpdev
 blogs.windows.com/windows_phone/b/windowsphone
 http://developer.nokia.com
 http://bit.ly/godvlup
 http://conversations.nokia.com
 http://appstudio.windows.com
 http://flip.it/95YFG
 www.geekchamp.com
 www.wpcentral.com

Thank You!
Slides will be posted on my Slideshare account.
Slideshare: www.slideshare.net/ActiveNick
Blog: www.AgeofMobility.com
Twitter: @ActiveNick
Mobile Apps: www.bigbaldapps.com
LinkedIn: www.linkedin.com/in/activenick
Website: www.AgeofMobility.com
Email: nick.landry@microsoft.com

Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile Apps

Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile Apps

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile Apps

Similar a Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile Apps (20)

Más de Nick Landry

Más de Nick Landry (18)

Último

Último (20)

Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile Apps