.. image:: https://travis-ci.com/dictation-toolbox/aenea.svg?branch=master :target: https://travis-ci.com/dictation-toolbox/aenea

================= Aenea

A system to allow speech recognition via Dragonfly on one computer to send events to another.

| Alex Roper | alex@aroper.net | http://github.com/calmofthestorm

| With many thanks to Tavis Rudd for showing us it was practical for coding... | (http://ergoemacs.org/emacs/using_voice_to_code.html) | ...and to Joel Gould for NatLink, making it possible... | ...and to the current maintainers of Natlink: Rudiger Wilke, Mark Lillibridge, and Quintijn Hoogenboom... | ...and to Christo Butcher and Dragonfly for making it easy... | ...and to Nuance for being so awesomely hack friendly... | (even if it means we have to write the grammars ourselves :-) )

Summary

Aenea is a project to allow Dragonfly (a Python-based voice macro system for Windows) to send commands to another computer. Typically, this is used to run Dragonfly in a virtual machine while commands are sent to the host operating system. Currently only Linux and OS X hosts are fully supported, but a working subset of the functionality is available for Windows. The primary audience is system administrators and software engineers.

Current Features:

Control keyboard and mouse on one computer using your voice on another. (Supports proxy versions of Dragonfly's Key, Text, and Mouse commands)
Enable or disable voice commands based on the currently active window on the host. (Supports flexible proxy versions of Dragonfly's AppContext)
Access to Dragonfly's powerful grammar specification, using Dragon NaturallySpeaking (via Natlink) or Windows Speech Recognition (not tested).
Many Dragonfly grammars will work with only minor modification via proxy.
Dictate prose directly into any remote application via emulated keystrokes using the keystroke capture client.
Easily add custom Python RPCs run on the server (Linux host) that will be available from your grammars (see server/linux_x11/plugins and client/_server_plugin_example.py)
More of a toolkit to voice-enable your current workflow than an off the shelf development environment.

Primary limitations:

Limited ability to take advantage of context when dictating prose.
Lacks ability to use context dependent editing commands (select that, etc)
Requires some knowledge of Python programming to edit or create new grammars, and in some cases to customize.
Relies on neither free nor gratis Windows and Dragon NaturallySpeaking.
Somewhat complex to set up, depending on your background.

Missing features:

Currently no encryption or authentication for remote control protocol (not a huge issue since it is typically used on single user systems via loopback).
Currently only fully supports a Linux X11 and OS X host. Partial support for Windows is available.

The primary focus of this project is writing code, system administration, terminal use, etc, and it works quite well for those tasks. For writing prose, word processing, etc., this project is quite limited compared to using Dragon natively on Windows, though it is still usable for those tasks.

Current Status

This project currently does everything I need well enough, and no new features are currently planned. I remain astounded by what Nuance has made possible via Natlink, and the flexibility and power of Dragonfly. I'm happy to help with troubles getting set up and take a look at bugs you may encounter, and still welcome pull requests.

Overview

The system consists of a client, Dragon NaturallySpeaking with Natlink and Dragonfly and any voice grammars the user wishes to use running on a Windows virtual machine, and a server, running on the host computer. The client listens on a microphone, recognizes commands based on what you say and the current context on the server, and then sends commands to the server via JSON-RPC to perform actions such as pressing keys and clicking the mouse.

Aenea provides Proxy versions of Dragonfly actions such as Key, Text, and Mouse (ProxyKey, ProxyText, ProxyMouse), which take the same specification language as Dragonfly actions, but instead forward the action to the host to execute. There are also wrapper classes that will respect whether or not the proxy is enabled and delegate execution to either Dragonfly (locally) or via the server. There are also wrappers that will take different actions based on whether the proxy is enabled and/or which OS is running on the server.

Getting Started

Windows VM Software (versions given are ones I used, others likely work too):

Windows 7 Ultimate 32 bit
Dragon NaturallySpeaking Premium 12. (Version 13 also works, with noticeably better accuracy than 12!)
NatLink 4.1echo. 4.1whiskey also works. (look for Natlink download links here https://qh.antenna.nl/unimacro/installation/installation.html)
Python 2.7.5
pywin32-2.1.8
dragonfly2-0.8.0 (please read https://github.com/Danesprite/dragonfly#installation)
python-jsonrpclib-0.1.7
pyparsing-2.0.1

Some notes:

NatLink has some problems on Windows 10 64-bit related to the msvcr100.dll file. It is unclear whether it is the 64-bit or the Windows 10 which resulted in the problem. Therefore it is recommended that you use the version of Windows mentioned above.
If you have problems installing NatLink, this page may help http://qh.antenna.nl/unimacro/installation/problemswithinstallation.html
A previous version of these instructions recommended python-jsonrpclib-0.1.3, but I ran into a bug in it that was fixed in the later version

Setup Instructions

Note: poppe1219 has other installation instructions at https://github.com/dictation-toolbox/dragonfly-scripts; take a look if you're having trouble getting everything working in the VM.

Operating system, Dragon, Natlink, and Dragonfly


0) Install VirtualBox.

1) Install Windows. It works well with 1 GB of RAM, two processors, and a 35 GB dynamically-sized hard disk, of which it is uses about 17-20 GB. (You can increase the RAM to speed up the installation process, and then lower it later to spare system resources.) While it's installing, I suggest you skim the Dragonfly documentation at http://dragonfly2.readthedocs.org/en/latest/

2) Install Dragon, and create your profile according to their directions. IMPORTANT: Ensure that you select BestMatchIV when creating your profile. Recent versions of Dragon default to BestMatchV, which has substantially worse performance with the sorts of grammars we will be using with Dragonfly.

Note: I and others have had problems when creating a profile where only a few seconds into the volume check a pop-up appears complaining about the microphone. To get around this, I memorized the text and continued reading while clicking okay on the dialogue as soon as it appeared. I had to read the text seven or eight times speaking in an unnaturally loud voice to get past this step. You may have to try a few times. I believe this may be a side effect of the USB microphone going through the virtual machine, and as such you may consider creating your profile on a native Windows installation and then moving it over, however I have not tried this. You may also have issues getting past the microphone quality check, as I did, however it worked just fine after that.

3) Install the other software mentioned above, and enable Natlink (by selecting GUI configuration from its start menu entry with Dragon closed). Make sure you install Python and dragonfly into paths with no spaces in them.

4) In VirtualBox's networking settings, set the network to host-only adapter so the VM can't access the network and gets a subnet. If you don't do this, you will need to modify the client and server config files to specify the correct interface to connect to.

5) Now when you start Dragon, a second small window with the title ``Messages from NatLink`` should pop up. If you have issues with this, take a look at the various forums that discuss using NatLink/Dragonfly on Windows.

6) You should now be able to run Natlink and Dragonfly grammars in the VM. Grammars are, by default, located in ``C:\\NatLink\\NatLink\\MacroSystem``. NatLink will load any file named ``_*.py`` (where ``*`` is a wildcard). If your grammars depend on libraries, you can place them (not starting with an ``_``) here. Your grammars will be able to import them, but NatLink will not attempt to load them directly.

7) Test that NatLink is working correctly. Copy ``aenea/client/_hello_world_natlink.py`` to ``C:\\NatLink\\NatLink\\MacroSystem`` and restart Dragon. In the ``Messages from NatLink`` window, you should see ``NatLink hello world module successfully loaded. All it does is print this message:-)`` typed out into Notepad. This means that NatLink successfully loaded your grammar. You can now delete the file you just created inside ``C:\\NatLink\\NatLink\\MacroSystem`` along with its corresponding ``.pyc`` file.

9) Copy ``aenea/client/_hello_world_dragonfly.py`` into the MacroSystem folder, and turn your microphone off and on again. Now open Notepad (or similar) and say ``test hello world grammar``. The phrase ``Hello world grammar: recognition successful!`` should be typed into the active window. (If you are curious to see how it works, open the ``aenea/client/_hello_world_dragonfly.py`` file to have a look - this will be good preparation for your future grammar writing career :P). If this doesn't work, try switching Dragon to command mode first. If it still doesn't work, try restarting Dragon. If it still doesn't work, then there is an issue with the setup of Dragon/NatLink/Dragonfly. Once the ``recognition successful`` has been

Aenea

Install / Use

README