The Virtual Keyboard Interface: Improving Access to Mouse-Intensive Applications
Overview
People need choices in how they operate computers. Well-designed software should allow individuals to interact as easily with an application by mouse or keyboard. Yet despite accessibility standards, guidelines, specifications, and regulations that mandate keyboard access (e.g., W3C WCAG 2.0 and Section 508), many current applications — whether web-based, mainstream, or proprietary — are awkward, inefficient, or impossible to drive by keyboard.
Until all programs are standard-compliant, other methods are needed to enhance usability for people who cannot, or prefer not to, use a mouse. In this paper, I describe how third-party macro scripting utilities may be used to develop "virtual keyboard interfaces." A "virtual keyboard interface" is a set of keyboard commands for an application that, although not native to the application, allows it to be controled by keyboard.
Because keyboard support is not built into the applications, mastering a virtual keyboard interface can be challenging. In this paper, I address the cognitive dimensions associated with "overlaying" an application with a virtual keyboard interface by showing how commands can be made intuitive, discoverable, and easy to remember.
Finally, I outline a simple technique for "recycling" keyboard commands as NaturallySpeaking commands. A virtual keyboard interface allows individuals to control mouse-intensive applications by keyboard, voice, or both.
Keyboard accessibility and usability
Some applications can be fully accessed by keyboard, but are not particularly usable. A program that requires, say, ten or more keystrokes to perform common tasks that are usually done with one or two mouse clicks may be technically accessible, but too physically demanding to be usable.
In addition to the unnecessary physical effort that a poorly designed interface enforces, cognitive demands can also make take a hit on usability. If there is little or no consistency in the design of the interface, ease of learning is affected. This is the case when default hotkeys use inconsistent modifier key combinations: Alt+... for some tasks, Shift+Alt+... for others, etc. In short, accessibility is illusory when keyboard interaction is impractical, and hard to learn and remember.
When applications have inadequate (or no) support for mouseless interaction, a virtual keyboard interface may be the best option for individuals who cannot use a mouse.
Scripting prerequisites
Scripting tools
To build a virtual keyboard interface, a third-party scripting tool is needed. Key features to look for in a scripting utility include:
- Scope options: Macros are active only when, for example, a specific window opens; a specific program is in the foreground; or a specific control has focus.
- Conditional statements: e.g., IF - THEN - ELSE
- Variables and functions: e.g., Let x = Length(Clipboard)
- Event triggers: e.g., an application, web page, window, or control gains or loses focus.
- System monitoring command: e.g., the ability to monitor changes in the colours of pixels and the shape of mouse pointers.
- Control capture: e.g., a way to capture information about Window Controls.
At least four third-party products for Windows® can be used to develop virtual keyboard interfaces: AutoHotkey, AutoIt, Macro Express, and Macro Scheduler.
What makes an application scriptable?
The applications that are easiest to script have:
- Basic keyboard support, e.g., the ability navigate to menus or certain controls via hotkeys or accelerators
- Unique window titles for each screen
- Consistent screen layouts, e.g., objects are always in the same position on every screen
- Standard (as opposed to custom) controls
- Large mouse targets
Unfortunately, some applications fall short of these requirements. It is still possible to construct virtual keyboard interfaces for these applications using the scripting techniques outlined later in this paper.
Scripting techniques
Design philosophy
The "ideal" virtual keyboard interface is reliable, robust, forgiving of errors, easy to use, easy to learn, and easy to remember. Usually a balance must be struck. When practical, I avoid creating a plethora of hotkeys. Instead, I emphasize mastering a finite number of keyboard techniques rather than memorizing dozens (or hundreds) of key combinations. To this end, my approach, whenever possible is to adhere to keyboard interaction standards, and build on the user's understanding of how keyboard interaction should work.
This design philosophy can be summed up as two principles. Design and develop virtual keyboard interfaces that
- Conform to the user's expectations of how the application should work.
- Create interactions that match the user's learning style and personal needs.
Use somewhere...
- Searching pixel by pixel for particular colours
- Monitoring changes in the colour of a pixel at specific coordinates
- Monitoring changes in the shape of the mouse cursor
- Copying part or all content to a variable and analyzing it for key words
Although these scripting techniques are complex, reliable keyboard access to otherwise inaccessible applications is achievable.
Force hotkeys to work as users expect
A hotkey is a key (or key combination) that performs a specific task. In many applications, Ctrl+A selects an entire document, page, or field. When hotkeys do not work as expected, they can be repaired with window, application, or control-specific keyboard macros. If the user expects Ctrl+A to select all, consider creating a macro that does exactly this.
Hotkeys for common tasks, including Ctrl+C for Copy, Ctrl+S for Save, and Ctrl+P for Print are supposed to be available in all applications, but there are exceptions. The Ctrl+P hotkey did not work in Word 2007 (although it was restored in Word 2010).
Repair "broken" accelerators
Whereas hotkeys need to be memorized, accelerators are discoverable. An accelerator is a key combination that is displayed as a highlighted character in the user interface (UI). The character is usually underlined, but sometimes appears in other ways. The character is part of the text associated with a menu, control, etc. Accelerator keys are typically activated by pressing Alt+underlined character — although there are exceptions. For example, in many programs Alt+F opens the File menu.
Accelerators eliminate the need to memorize hotkeys. Accelerators provide users with "clues" on how to perform tasks. If someone knows one "rule" — Alt+ underlined character performs the action — he or she needs only to observe the accelerators to know how to interact with all of them.
Accelerators do not always work as indicated. "Broken" accelerators are common. They are repaired the same way as broken hotkeys: with window, application, or control-specific keyboard macros. For example, if pressing Alt+C fails to activate the Cancel button, consider developing a macro that does.
Reposition and resize windows
A macro can be scripted that activates when a specific window gains or loses focus. Use these macros to automatically resize and reposition windows. Macros that click specific x-y coordinates can be made more reliable by maximizing the window prior to moving the mouse pointer.
Create efficiencies
Macros are customarily used to automate repetitive tasks. If a user always opens a certain application, navigates to a certain field, and enters the information each time, it may make sense to provide a macro that performs all three steps.
A strategy for making this kind of macro easy-to-remember is to choose the activation that a person normally uses to perform the first step. This could be a hotkey, a desktop icon, or a menu or toolbar command.
Voice-enabling applications
In an article illustrating ways to trigger Macro Express scripts with NaturallySpeaking, I describe a simple technique for voice-enabling macros: First, create hotkey macros; second, make NaturallySpeaking custom commands that send the hotkeys to an application. This technique is straightforward enough that adding NaturallySpeaking support for some applications is (almost) effortless.
Case study
In a call centre, staff used three mouse-intensive applications. An employee with mouse-induced repetitive strain injuries was no longer able to perform many software tasks. I developed approximately 150 hotkey macros that send keystrokes; navigate to reliable "anchor" controls; "hunt" for pixels; and click on screen objects.
To reduce the need to memorize, the vast majority of the hotkeys were activated by Alt+character. Because many windows had unique titles, it was possible to reuse many hotkeys. For example, in one application, Alt+N moved keyboard focus to the Name field in one screen, and Alt+N activated the Next button in another screen. In a different application, the same hotkey checked the New Addresscheckbox. Furthermore, I implemented more than ten macros to close pop-up windows, all triggered by pressing Esc.
We also experimented with pop-up menus to further reduce cognitive effort. Rather than memorize large number of hotkeys, we grouped commands into six categories, and assigned each category a hotkey. Each hotkey invoked a pop-up menu that displayed a small number of related commands. Each command could be activated by pressing a clearly-marked accelerator key. For example, pressing Alt+S activated the Sales menu, which consisted of Daily, Weekly, and Monthly. To pick Monthly sales, press Alt+S followed by M. With this system, two keystrokes were required to activate a command instead of one, but there were fewer commands to memorize.
Conclusion
Virtual keyboard interfaces is a scripting technique that produces reliable keyboard access to mouse-intensive applications. The same macros can be easily "recycled" as NaturallySpeaking custom commands. This paper described the scripting techniques available in common third-party macro programs for producing virtual keyboard interfaces, and the ways that the keyboard commands can be made intuitive, discoverable, and easy-to-remember.