Software developers who are blind or visually impaired find it difficult to use productivity-enhancing IDEs, like Visual Studio or Visual Studio Code. CodeTalk is a Visual Studio extension that was previously released to help address these problems by providing accessibility improvements when writing and debugging source code. A core challenge when developing while blind or visually impaired is that these development environments often have highly-complex UI that presents large volumes of related content and this content is often dynamically updated to convey application state. Source code content is structurally interrelated, creating significant impediments to efficient navigation, and dynamic updates can either show continuously updated state or provide context that a user action is required.
VSCodeTalk offers a set of features that address the following categories of usage scenarios i) glanceability, ii) navigability, and iii) real-time alertability. Glanceability is the ability to quickly look at a source code file and ascertain information about the overall code structure. Navigability is the ability to quickly move around the code to the desired code segment. Real-time alertability is the ability to receive notifications about the application state at the desired time. The VSCodeTalk features implemented in this project help improve these use-cases.
- Glanceability, such as the features Function List and Current Context
- Navigability, such as the features Function List, Current Context, and Go to Parent
- Real-time alertability, such as the features TalkPoints Debugging Information and Syntax Error Information
Please watch the following video for additional information about VSCodeTalk, including a demo of some of the most useful features.
The VSCodeTalk project was inspired by the CodeTalk project that is documented at https://www.microsoft.com/en-us/research/project/codetalk/. That project includes a Visual Studio extension that implements many of the ideas discussed in the accompanying research paper https://www.microsoft.com/en-us/research/uploads/prod/2018/04/CodeTalkCHI2018CameraReady.pdf.
The issues that people with visual impairments encounter when using software development environments are of interest to software tools vendors. The Visual Studio Code team has implemented several concepts similar to VSCodeTalk features that help improve the default product's accessibility. One example is the Code Outline feature that overlaps with the Function List and Code Summary features in some regards. Another example is the debugger which has built-in voice notifications for several breakpoint events, such as narrating the words 'stopped' or 'continue'.
VSCodeTalk project helps improve several accessibility use-cases for visually impaired users. There is a Visual Studio Code extension available that can be manually installed from a VSIX extension file. The source code and binary files for this extension are available from the following links.
- Extension VSIX link: https://github.com/kburtram/CodeTalk/releases/download/0.3.0/codetalk-0.3.0.vsix
- Source Code link: https://github.com/kburtram/CodeTalk/tree/0.3.0
The accessibility challenges of i) glanceability, ii) navigability, and iii) real-time alertability have been addressed through the following VSCodeTalk features.
- Function List displays a list of functions in a source file.
- Current Context displays context information about current cursor location such as
source code line, the function being edited, and the parent elements.
- Go to Parent moves the cursor to the parent source code element.
- Talkpoint Debug Information generates an audio alert when a breakpoint is hit.
The alert can be configured to be a tonal, text, or an expression audio sound.
The talkpoint can be configured to either stop or continue.
The following prompt shows how a user can use the Visual Studio Code Command Palette
to create a new talkpoint.
The Talkpoints List shows a list of all the existing talkpoints.
- Syntax Error Information generates an audio alert when a new syntax error is introduced into a source code file.
The VSCodeTalk extension can be installed in Visual Studio Code through the Extension Manager. The extension details for all implemented commands, settings, and views are viewable in VS Code as shown below.
The CodeTalk feature set was based on research that involved close collaboration with the BLV community. VSCodeTalk is largely a direct port of these features so we have some validation that the underlying concepts are sound. A significant aspect of the validation that we did was to ensure that features were implemented correctly and that the user experience passed accessibility testing scenarios.
Screen Reader Compatibility Validation
The accessibility testings procedures focused on screen-reader audio output and keyboard-only navigation scenarios. During screen-reader validation, we would systematically go through each feature with the NVDA software enabled. The spoken text would be compared with the displayed content to confirm that the output is correct and that sufficient text was read to provide source code context.
Keyboard-only Navigation Validation
The keyboard navigation validation scenarios were mostly focused on ensuring the cursor focus could easily be moved back-and-forth between the editor and the VSCodeTalk feature view windows. The content in the views, such as the current context or the function list, has keyboard command handlers to refocus back to the editor on the ENTER keypress.
Language Support Validation
We tried our extension with code written in multiple languages including:
This helped ensure that our extension was not coupled to a single language, but could be language-agnostic.
We met with Venkatesh during office hours to walk through the extension that was built and gathered feedback.
We also shared an early private build of the extension for validation. Feedback included:
- Build features in a language-agnostic way.Status: done
- Add list of talkpoints to keep track of existing talkpoints. Status: done
- Add prompt message when adding a talkpoint. Status: done
- Leave CodeTalk panel with ESC key. Status: done
- Prompt users to install language-specific extension if missing. Status: investigating
Learnings and Future Work
There were many learning opportunities while implementing the VSCodeTalk project. A major learning came from trying to understand the software development process for people for visual impairments. Software development tools display a large amount of information simultaneously. And there is a lot of context that must be maintained about the current cursor position in order to understand what modifications are required in a source file. While implementing and testing the VSCodeTalk feature set, we tried to mimic these user workflows. This exercise was a great way to build a better understanding of some of the challenges for BLV users when using software development tools.
The next steps for the VSCodeTalk project is to try to merge the Visual Studio Code extension source code upstream to the https://github.com/Microsoft/CodeTalk repo. We would then like to collaborate with the CodeTalk project maintainers to help transfer the extension source code to them for a potential release to the Visual Studio Marketplace.
An additional investigation was conducted into how to make the VS Code built-in Code Outline feature more accessible. The Code Outline feature contains much of the same data as the Function List, Code Summary, and Current Context features. Code Outline's user experience is not optimized for accessible use-cases around screen-reader spoken word quality and keyboard-only quick navigation through source code symbols. We looked at various was to reorganize the information in Code Outline to better match the CodeTalk data presentations. The screen-reader user interface metadata was also improved to narrate the symbol datatype as well as the line numbers when describing the symbol. We plan to open a issue on the VS Code repository with suggestions for improvements referencing a PR with our experimental changes.
Other feedback for VSCode:
- Feature: Add ARIA announce capabilities into VSCode
- Feature: Be able to set ARIA labels on Text Decorations
- Feature: Be able to play sounds/tones
- Bug: Screen reader not reading QuickInput description if active item is pre-selected
- Bug: outline.focus does not move keyboard focus to outline
- Bug: Inconsistent navigateToLast command