Introduction

IronPython is the .NET implementation of the Python programming language (www.python.org). It's a dynamically typed language with support for many programming paradigms such as object-oriented programming, and also allows seamless use of .NET code. The goal of this tutorial is to demonstrate how easy it is to make an application using the power of the .NET libraries, specifically the ones relating to windows forms and speech. We will also see how the interactive nature of Python is well suited to rapid prototyping, debugging, and experimentation. Due to the complexity of the program, it is assumed that you have already completed the main IronPython tutorial that teaches the basics of Python and how to run applications in IronPython. However, in an effort to be friendly to newcomers who learn better through a case study that can be played with, I have broken down elements of the program into exercises that can be completed independently. This division also corresponds to my steps in coding this application, so it might be worthwhile to see how the use of Python accelerated development.

The following packages are required:

Windows Vista

Visual Studio 2005 is an optional supplement that can be helpful for viewing methods and GUI element properties. However, this is not a requirement.

Getting Started

Please make sure the sample folder Voice (where this readme.htm is located) contains at least the following:

.\Exer1.py

.\Exer2.py

.\Exer3.py

.\Exer4.py

.\favs.txt

Add the folder containing the IronPython distribution to your Path environment variable.

Exercise 1: Building a Web Browser

In this section, we will learn the basics of manipulating UI in IronPython and to do something useful with it: build a web browser!

Specifically, we will use Windows Forms or winforms, the established way of building graphical applications. We will examine how to set parameters, add controls, and display the application. This will lay the foundation for building a graphical web browser that has all the power of the rendering engine of IE. This idea was inspired by the presence of the "Web Browser" control in the toolbox of VS 2005.

The code for this section can be found in Exer1.py

Task 1: Making a Window

Start the IronPython console

Execute the following:

import System

import clr

clr.AddReference("System.Windows.Forms")

clr.AddReference("System.Drawing")

from System.Windows.Forms import *

from System.Drawing import *

Let's examine this code.

import System

import clr

This block is the Python equivalent of the using statement in C# and the #include statement from C/C++. We are loading libraries/modules/namespaces to use for our main program.

clr.AddReference("System.Windows.Forms")

clr.AddReference("System.Drawing")

Here we are calling a basic method of clr. We are adding references to more complicated libraries. For this exercise, we will need to load System.Windows.Forms and System.Drawing in addition to System. Note that we need to add these separately instead of importing them because they are not part of the basic namespaces that we can just import. They are in separate dlls on your hard drive and the information for accessing them must be retrieved through this method. For now, consider basic namespaces such as System and clr as having already been added as references.

from System.Windows.Forms import *

from System.Drawing import *

This might seem complicated. However, what we are doing is just unpacking namespaces. We are importing the contents of these two namespaces into the global environment. Now, if we do dir() before and after, we can see the difference. This trick allows us to make classes with Form() instead of having to type out System.Windows.Forms.Form().

Note that we don't need to add a reference to System.Console because it is not a separate DLL. Rather, it is contained in System. It is important to make the distinction between the libraries that have to be added and then imported and those that are contained in basic namespaces such as System. The documentation for this is on the MSDN website.

Execute the following:

f = Form(Text = "My Web Browser")

f.AutoScroll = True

f.AutoSize = True

f.MinimumSize = Size(40,40)

f.Size = Size(500,500)

The first line creates an instance, f, of the object Form (from System.Windows.Forms if you were wondering). This is a winform element that is basically a blank window to be controlled by the OS.

Next, we set four properties of the form. The syntax of this is almost identical to C# code. The "." operator allows us to access properties of the instance and set them. Words such as True and False are Python Boolean values. Do not confuse this with "True", which is a string. We can also pass the values of the properties as arguments to the constructor. We set the title of the form to be "My Web Browser". We could just as easily use f.Text = "My Web Browser". Likewise, we can move one of the other lines into the constructor.

Autoscroll tells the form to automatically add scroll bars when there is more content than the form can contain. This lets us display large web pages. Autosize resizes the window to fit the content. MinimumSize is the smallest that we can manually resize the window to be. Size is the default size that the form has when first called.

Note that the code is different from C# in that we do not need to prefix declarations by new. Also note that like most high-level scripting languages, objects are untyped in Python.

Task 2: Making a Browser

Execute the following:

browser = WebBrowser()

browser.Dock = DockStyle.Fill

browser.MinimumSize = Size(40,40)

browser.Url = System.Uri("http://www.msn.com")

Likewise, here we are instantiating a Web Browser object found in System.Windows.Forms.WebBrowser. It is a control element that acts like a web browser except that it can be embedded in other applications. Neat, isn't it?

We also set some properties to desirable values. In this case, we are setting the position (DockStyle.Fill tells the element to take up all the space in the form), the minimum size, and the default URL to be shown upon initialization.

If you are wondering how I get the values to set properties, I'm going to point you to the MSDN site. Everything is there, including documentation on attributes and examples of usage. There are no Python examples yet, so use the ones in your favorite language. The rest is experimenting with the environment. For example, if I'm unsure of what to do after I make a browser element, I can do dir(browser) to list all the methods and attributes of the browser element. Then, I can look up the documentation on MSDN and find appropriate values. If you do something wrong, the system will simply throw an exception. It's a lot easier to debug and code like this (freeform experimentation) than to code in an IDE.

Task 3: Putting it Together

Execute the following:

f.Controls.Add(browser)

f.ShowDialog()

Our objects are now created. The next step is to link them by putting the browser in the form f. We do that with the "Add" method of f.Controls by passing "browser" as a parameter.

Verify the results of f.ShowDialog(). This method of "f" draws the form in all its glory to your screen. You should see a web browser embedded there. Congratulations.

Exercise 2: Building a Voice Recognition System

In this section, we will learn about the powerful voice recognition capabilities added to .NET in version 3.0. As part of Windows Presentation Foundation, System.Speech adds two engines, one for rendering text as speech through a variety of synthesized voices and another for interpreting speech as text according to certain grammar rules.

In our example, we will start by building small modules and walking through the code for making grammar rules to recognize speech. We will not be touching on any speech synthesis tools. All of our efforts will focus on recognition. Thus, it is important to have a good microphone and to be operating in a quiet environment.

The code for this section can be found in Exer2.py

Task 1: Configuring Everything

At this point, it's important to make sure the audio recognition system is properly configured. Go to Speech Recognition in the Control Panel and configure the microphone. After that, train a recognition profile. Then test to make sure that your machine can accept audio input. Once you are sure that everything is done, proceed to the next step.

Task 2: Your Speech Recognizer

Now let's build a basic recognizer engine. Open a new console and enter the following:

import System

import clr

clr.AddReference("System.Speech")

from System.Speech import *

def recognized(o, eventArgs):

System.Console.WriteLine(eventArgs.Result.Text)

Let's examine this code. By now, you should recognize the basic import commands and that we are loading the System.Speech library. The library is a component of .NET 3.0's WPF pillar and uses SAPI as an internal speech decoding engine.

However, we are introducing something new here when we define a function. This function is an event handler, meaning that it is executed when a certain event happens. Because Python is untyped, we can write events that we then plug into any situation. Examples of this will be shown later. For now, notice that this handler prints out the content of some object passed to it. System.Console.WriteLine is the basic printing mechanism for .NET. We could just as easily have called Python's native print command.

Execute the following code:

recognizer = Recognition.SpeechRecognizer()

choices = Recognition.Choices()

choices.Add("Sam")

choices.Add("Mary")

choices.Add("Bob")

builder = Recognition.GrammarBuilder()

builder.Append(choices)

Let's examine this code.

First, we create an instance of the Speech Recognizer Engine. This is a class in System.Speech and serves as the connector to the SAPI engine. Once it's constructed, it's automatically initialized and when it detects speech, it will send the signal to the grammar engine to interpret. Right now, it's picking up everything we say into the microphone, but since we have no grammar instances attached, nothing will happen.

Choices can only recognize one of its states at a time. We added three strings to our choice builder, so we can say any of the three names and have the name recognized.

The speech engine must be very precise, so it cannot interpret any arbitrary sound. Thus, we use grammar to define precisely what the engine can recognize. The sound is parsed to match the grammar choices. If there is a match, we trigger the event handler for speech recognized, else, we trigger the speech detected handler or do nothing. Grammar Builder is quite simple to use. We can append strings or choices. If we append a string, it *must* be said for that grammar path to be followed. If we encounter a choice, we must satisfy one of the choice conditions before proceeding. When we reach the end, the speech engine triggers the handler.

Execute the following code:

my Grammar = Recognition.Grammar(builder)

myGrammar.SpeechRecognized += recognized

recognizer.LoadGrammar(myGrammar)

System.Console.ReadLine()

Let's examine this code.

Finally, we're at the stage of building the grammar engine. Initialization proceeds as above. Attaching the handler is done with the += option. We link the function to the port SpeechRecognized. A list of available ports is on MSDN. The last step is to load the grammar processor into the speech recognizer. The ReadLine() statement simply stops the program from closing because we are not executing any more lines. This gives time for the Recognizer(on a separate thread) to read your voice.

After connecting the two, we can try speaking into the microphone. The handler should be triggered as appropriate, and the name of the person you called will be printed in the console.

Exercise 3: Sidetrack: Launcher Application

Voice in general is not a good way to interface completely with a program. As an input method, it is far too unreliable (poor sound quality, misunderstood commands, poor vocabulary/grammar selection, etc.) and inefficient (enormous CPU power to accurately decode speech). Humans can enter text through the keyboard and mouse much faster than through voice. All potential uses of voice systems should be application-specific. That is, they leverage commands through Hash tables, mapping a certain word or phrase to an action within the context of the program. It should not be used for blind text input (which is why our web browser will remain limited in functionality).

One particular use for voice-based input is to launch applications. For office workers, it's often an adjustment when they must switch tasks. They often have to navigate through the start menu and then find the exact path of the file they want to open. We can change that by allowing the execution of any command that can be done through the command line (such as opening a file or querying a database) and abstract that as a single voice command. In this example in Exer3.py, we will do some basic application launching. This will also familiarize you with the complex abstractions coming in the last section.

Task 1: Voice-based Launching

Execute the following code:

import System

import clr

clr.AddReference("System.Speech")

from System.Speech import *

from System.Diagnostics import *

from System.Collections import *

Let's examine this code.

You might notice that we are importing new libraries this time. System.Diagnostics contains the Process class, which lets you launch/terminate applications as well as set command parameters. It allows shallow interactions with a program without fully embellishing it in a .NET shell (if the program is unmanaged code, it will execute as unmanaged). System.Collections contains some basic container classes such as arrays, arraylists, and hashtables. We will primarily be using the hashtable.

Execute the following code:

Programs = Hashtable()

Programs["Word"] = lambda : Process.Start("winword")

Programs["Excel"] = lambda : Process.Start("excel")

Programs["Powerpoint"] = lambda : Process.Start("powerpnt")

Programs["Notepad"] = lambda : Process.Start("notepad")

Programs["Internet"] = lambda : Process.Start("iexplore")

Programs["Program"] = lambda : Process.Start("devenv")

Let's examine this code.

Wow. Lambda statements? These may be unfamiliar to you if you come from a C/C++ background. Essentially, all we're doing here is creating a hashtable and associating strings with functions. Then, later on, when we get a match in the hashtable with a vocal command, we execute the function. There's no need to have complex switch or if-else statements to associate commands with strings. The documentation for lambda is *not* on MSDN. Look to the Python site for documentation about this.

Side Note: part of what's so brilliant about IronPython is that we are free to mix and match the best of all worlds. We can borrow Python libraries and syntax and tools such as lambdas. We have the .NET framework and all the languages and libraries that target it. Isn't it great?

Execute the following code:

def recognized(o, eventArgs):

   if Programs.Contains(eventArgs.Result.Text):

      Programs[eventArgs.Result.Text]()

      synthesizer.Speak(eventArgs.Result.Text)

Let's examine this code.

Simple function declaration - this should be pretty self-explanatory. We're making an event handler that says that if we receive voice input and that voice input matches one of the keys we put into the hashtable, we will execute whatever function is contained in the hashtable at that key and then say the name of the program we executed. Note that we're forward referencing the variable synthesizer as we haven't defined it yet! This is fine according to Python because it's interpreted. However, if we fail to create that variable later on or misname it, we may get headaches trying to debug our app.

Execute the following code:

class VoiceRecognizer (Recognition.SpeechRecognizer):

   def __init__(self):

      self.choices = Recognition.Choices()

      for x in Programs.Keys:

         self.choices.Add(x)

      self.builder = Recognition.GrammarBuilder()

      self.builder.Append(self.choices)

      self.myGrammar = Recognition.Grammar(self.builder)

      self.myGrammar.SpeechRecognized += recognized

      self.LoadGrammar(self.myGrammar)

synthesizer = Synthesis.SpeechSynthesizer()

recognizer = VoiceRecognizer()

System.Console.WriteLine("Press enter to exit...")

System.Console.ReadLine()

Let's examine this code.

Here is a class definition. Notice that we are inheriting from the SpeechRecognizer class and will have all of its attributes. __init__ is equivalent to a constructor and this should all be familiar if you've experienced OOP. Declaring a new variable isn't done through public or private; rather, we just call self.variablename and set it equal to something. Syntax of note here is the for statement. We don't have the numerical incrementing for statement common in C. Think of this like a foreach statement.

Finally, we have our instantiations and they are initialized to default values.

Exercise 4: Basic Functionality in Voice Browsing

At this point, you have learned enough to put together a basic application that combines the previous features. In Exer4.py, we have the code for a voice-controlled web browser with significant modifications.

Task 1: Class Abstraction and Integration

There is virtually nothing new here. We are expanding on our previous exercises to include two classes with more methods. We are also doing text input and output, albeit with no error checking. The only difficult part at this point may be finding the right methods and members for all the different classes. MSDN should be a constantly referenced site when developing .NET apps. Please see Exer4.py for the source to this exercise.

Integrating Voice into .NET

Contributor: Richard Guo

Contents

Introduction

Getting Started

Exercise 1: Building a Web Browser

Task 1: Making a Window

Task 2: Making a Browser

Task 3: Putting it Together

Exercise 2: Building a Voice Recognition System

Task 1: Configuring Everything

Task 2: Your Speech Recognizer

Exercise 3: Sidetrack: Launcher Application

Task 1: Voice-based Launching

Exercise 4: Basic Functionality in Voice Browsing

Task 1: Class Abstraction and Integration