AXNav: Replaying Accessibility Tests from Natural Language

A key contribution of AXNav is its LLM-based planner that can navigate mobile apps to execute specific tasks or arrive at particular views. Our multi-agent system architecture is loosely based on ResponsibleTA, which presents a framework for facilitating collaboration between LLM agents for web UI navigation tasks [58]. Since AXNav is designed for testing rather than end-user automation, it removes some components (e.g., a system to mask user-specific information), and combines other modules (e.g., AXNav combines evaluation and completeness verification, and AXNav proposes actions and feasibility in the same step). These changes significantly reduce the number of LLM turns taken, which lowers cost and reduces latency.

Other UI navigation works for web and mobile apps have recently emerged. Wang et al.  [52], describe prompting techniques to adapt LLMs for use with mobile UIs, and evaluate an LLM-based agent’s ability to predict the UI element that will perform an action on a given screen. AXNav’s UI navigation system builds upon this work by supporting more complex, multi-step tasks. Other works map from detailed, multi-step instructions to actions in mobile apps [22, 33, 49]. AutoDroid injects known interaction traces from random app crawls into an LLM prompt to help execute actions with an LLM agent [54]. AXNav can interpret a wide variety of instruction types, from highly specific step-by-step instructions to unconstrained goals within an app (“add an item to the cart”), without relying on prior app knowledge. Furthermore, AXNav is able to modify its plan when the UI changes, if it encounters errors, or if the test instructions are incorrect.

The emergence of LLM-based UI navigation systems has motivated the need for more interaction datasets. Android in the Wild presents a large dataset of human demonstrations of tasks on mobile apps for evaluating LLM-based agents [41]. Other datasets, such as PixelHelp [33] and MoTiF [11] also collect mobile app instructions and steps. Unlike prior art, AXNav is designed to work on iOS apps, which can have different navigation flows and complexities than corresponding Android apps.

Most importantly, none of the above works have been used to interact with accessibility features or support accessibility testing workflows. This is the core focus of AXNav’s contribution.

Despite the availability of accessibility guidelines and checklists [1, 10, 53], linters and scanners [2, 3, 27], and platforms for test automation [21, 55], developers and QA testers still often prefer to test their apps manually [34, 35]. Testing manually by using accessibility services can reveal issues that cannot be revealed by scanners alone [37]. However, manual testing is costly and difficult to scale, leading to a variety of automated tools and testing frameworks being developed for accessibility testing [30].

There are a variety of tools to automatically check accessibility properties of apps [48]. Development-time [27] approaches use static analysis to examine code for potential issues. Run-time tools [2, 3, 21, 42] examine a running app to detect accessibility issues, which enables them to detect issues beyond static analysis; however, they still must be activated on each screen of the app to be tested.

Another approach is to automatically crawl the app to detect issues [4, 15, 20, 46]; however, such tools currently adopt random exploration and thus may not fully cover or operate the UI as an end-user might. These crawlers also do not operate through accessibility services which leaves them unable to evaluate whether navigation paths through the app are fully accessible.

Latte [44] starts to bridge this gap by converting GUI tests for navigation flows into accessibility tests that operate using an accessibility service; however, the majority of apps still lack GUI tests [34] and often require updating the code to new navigation flows when a UI changes [40]. Removing the requirements for GUI tests to be available, A11yPuppetry [45] lets developers record UI flows through their app and replay them using accessibility services (i.e., TalkBack [24]). This idea has also been explored in prior work for web applications [9]. However, a key challenge with record and replay approaches is that they can also be brittle and difficult to maintain as the UI evolves [32, 40]. By using LLMs, AXNav can interpret plain text instructions at different levels of granularity, and adapt them to new context when UIs change.

AXNav was not intended to fully scan apps for accessibility issues. Rather, it was designed to flag a subset of potential issues during test replay to aid manual accessibility QA testers, based on feedback from formative interviews. Our system architecture could also be extended to run accessibility audits during each step of the replay, similar to accessibility app crawlers [20, 46]; however, in this work we focus on navigation and replay through accessibility services and not on holistic reporting of accessibility issues.

Participants noted a key benefit of manual testing is to experience the feature as an end user might (P2-P5). One participant, P3, being a VoiceOver user, mentioned this enables them to more realistically test the feature as it is meant to be used: “the advantages are that we can test literally, from the user perspective myself, and a number of my teammates are users of the features because of various accessibility needs that we have. So we are the foremost experts in the functionality of those particular tests and what the expected results would be.” (P3)

Participants from three teams brought up challenges, including an overwhelming scope of features and scenarios, leaving them to target only a few key features and tasks for testing (P2, P4-P6). P5 stated: “It’s not like necessarily difficult. It is just like, repetitive and kind of boring and the scope is so big a lot of times like, if you’re looking at the <App Name Anonymized> app, there’s so many pages and so many views and so many buttons and different types of elements and everything. That is overwhelming and you feel you’re going to miss something”.

Writing manual tests was also noted as a challenge by participants from three teams, who write down or have existing test suites of manual instructions (P3-P6). Participants noted it was easy for those tests to become outdated when apps are updated, challenging less experienced QA testers’ ability to interpret and follow test instructions (P3, P4). Finding the right time for accessibility testing was also mentioned by four participants, as they worked with apps that are frequently updated across various product milestones (P3-P6). Two participants also mentioned trying to develop automated tests in their work, which they described as easily breaking and not covering all possible scenarios (P5, P6).

All participants took part in accessibility testing at various times throughout the product lifecycle. They tested annually as new features were added, or on regular release cycles of app interfaces. The participants’ daily work consists of manually performing tests for accessibility features (e.g., VoiceOver, Dynamic Type) across various products, or additionally writing accessibility frameworks and automation code.

To test purely visual accessibility features, the participants typically toggle on the feature under test and validate that the app’s UI renders or behaves correctly based on the setting. For accessibility services tests (e.g., VoiceOver), they typically enable the feature, and then either navigate the app to perform a task using the feature or navigate to a specific screen to validate the navigation order or another behavior of the feature.

Manual testing instructions are an extremely common artifact within our organization, existing in both manual test databases and bug-tracking tools. One team we interviewed (two participants) noted they own a large database of manual instructions for UI tests (P3, P4), but none of these instructions are specifically for accessibility testing. They also noted that they frequently write down manual instructions for accessibility features, or “repro steps”, when they are filing bugs. In their work, they often work with engineers who may lack familiarity with the accessibility feature under test, so they try to make instructions as specific as possible.

For another two participants on a different team, their accessibility testing instructions primarily consisted of a large regression test suite across ten apps with 300 individual test cases they perform annually (P5, P6). Among these tests, some had concrete low-level steps, but many were abstract, high-level, and assumed the QA tester has a high level of expertise on both the app and the accessibility feature to be tested. Figure 2 contains three example test cases for a video streaming app for the accessibility features Voice Over, Dynamic Type, and Button Shapes. Each test case typically has a title containing the platform, feature, and app to be tested, but only some test cases have step by step instructions, and only some test cases have an “Expected Result” specified.

In the second part of our formative study, we elicited features by having participants imagine a system replaying manual testing instructions on an iPhone, while watching a screen recording of one of the authors performing a manual test. The video was a screen recording only and had no additional features. We then asked participants what features such a system should support in the context of accessibility testing. Here we summarize the key features revealed by both this task and part one of our interviews that we incorporated into the design of AXNav.

Before executing a test, AXNav provisions a remote cloud iOS device and prepares it according to the parameters it extracts from the test instructions. AXNav extracts the name of the app to be tested and the assistive technology to use in the test (e.g., Dynamic Type) from the instructions to automatically install the app and select the assistive feature to test. Instructions typically take the form of those shown in Figure 2.

During setup, AXNav installs a custom application that provides an interface to operating system APIs that silences several system notifications, controls screen recording, and interacts with assistive technologies. AXNav uses an operating system API to toggle and configure the specific accessibility feature under test (e.g., Dynamic Type size). If the test is for VoiceOver, AXNav activates the caption panel (F3; section 3.4.4) and sets the speaking rate to 0.25 to accommodate for speeding up the exported video in the Test Results Export step.

When the device is ready for the test to be executed, AXNav launches the app under test, and begins screen recording. The test execution engine can interact with the cloud device over a remote desktop connection and the accessibility-specific features supported by the custom application (see section 4.2.4).

AXNav uses an LLM-based UI navigation system that can translate from natural language test instructions into a set of actionable steps, execute steps on a live device by calling APIs that interact with a device, and feed results back to improve the navigation plan (see Figure 4). We use OpenAI GPT-4 [38] in our implementation, but AXNav can be easily adapted to use other LLMs. Our system architecture is loosely inspired by ResponsibleTA [58], but eliminates some elements (e.g., masking LLM inputs), and merges other elements (e.g., combining feasibility with actions). It consists of three LLM-based agents: the planner agent, the action agent, and the evaluation agent. To provide device state to the LLM agents, we use existing pixel-based machine learning models to recognize UI elements, text, and icons [14, 57]. AXNav formats detected UI elements as text strings to be ingested by the LLM, described in section 4.2.1. To interact with the device, AXNav provides tools that the LLM invokes to send touch or keyboard input events and VoiceOver gestures.

AXNav can currently flag four types of potential accessibility issues in the output video: VoiceOver navigation loops and missing elements, Dynamic Type text resizing failures, and Button Shapes failures (see Figure 5).

AXNav’s output is a video of the test execution. Throughout the replay process, AXNav records the screen of the cloud device and logs timestamps of every action performed on the device, along with actions and activated UI elements. To improve the navigability of the video, AXNav adds named chapter markers that demarcate each step of the test being performed and each issue flagged by a heuristic (F1 & F2; section 3.4.2 & section 3.4.3). Many video players include features to view all chapter markers by name and navigate directly to the start of a given chapter. To help communicate actions while watching, AXNav overlays markers on the video stream that label each action taken with crosshairs for tap actions and arrows indicating scroll direction. Potential accessibility issues from heuristic results are also overlaid on the video stream with colored bounding boxes in either orange or cyan. AXNav also speeds up the exported video by a factor of 2.5 to minimize pauses due to the latency of its LLM-based agents.

First, we evaluated the system on a large regression manual test suite. Some examples of this test suite are shown in Figure 2. From that test suite, we extracted 64 test cases from 5 apps testing the accessibility features that AXNav supports: VoiceOver, Dynamic Type, Button Shapes, and Bold Text. We discarded two of the tests due to our account lacking the necessary subscription to view the screen(s) being tested. The final set contains 62 test cases. Note that this regression test suite is used for manual testing and is not constructed for the purpose of being used by any automated system. Many of the tests are very high level and assume the QA tester has a high level of expertise on the feature and the app under test. We chose to evaluate AXNav on this dataset since it is a representative set of real-world accessibility tests.

We also constructed a dataset of accessibility testing instructions for publicly available apps. We randomly selected apps from a public list of the 100 most popular free apps in the Apple App Store, ultimately selecting five apps from different app categories. Then for each app, one researcher on our team drafted four manual tests, one for each of AXNav’s supported accessibility features, using the regression testing suite as an example. We validated that the tests were realistic by discussing them with an expert accessibility QA tester from the formative study. The final dataset consists of 20 manual tests across five apps and four accessibility features.

We evaluated the difficulty of each test through a rubric based on prior work [26], which rates each type of instruction task into Easy or Hard categories for evaluation.

To group the tests into the above categories, two authors independently rated each test and then met to discuss and resolve any differences. Table 1 and Table 2 show the total counts for each level across the four supported accessibility feature categories and the two separate datasets.

To repeat each test, we input the test instructions into the system, reset the phone’s current state to match the initial state specified by the test, and then executed the test instructions on the device. During this process, we recorded all interactions between the system and the app. For both datasets, we report navigation replay success, which measures whether our system can follow the instructed steps successfully to reach the desired destination, and accessibility test success for whether the accessibility feature test succeeded. We also report navigation partial success, which indicates that AXNav replayed one or more steps in the test but did not end up in the correct final state. We determined success based on our own manual evaluation based on the expected behavior for each accessibility feature. To ensure consistency, two researchers independently scored the system’s performance on each test case and then met to discuss and resolve any differences.

For the regression testing dataset, our system successfully replayed 95.5% of easy test cases, and 61.1% of hard test cases for an overall success rate of 85.5%. Table 1 summarizes these results. Within our organization’s apps, support for the supported accessibility features is already high; the accessibility test success rate across these tests was 78%. We are also working with the owners of this regression testing dataset to report the accessibility test failures in our internal bug-tracking system.

For the free apps dataset, our system successfully replayed 83.3% of easy test cases, and 64.3% of hard test cases for an overall success rate of 70.0%. Table 2 summarizes these results. Support for the accessibility features of Bold Text, Dynamic Type, and Button Shapes unfortunately were low across the five apps, resulting in an accessibility test success rate for these apps of only 15.0% across the 20 test cases. This further motivates the potential impact of using systems like ours within the app development workflow.

While the navigation replay success of our system is good for both datasets, our system fails to replay some tests. In some cases, the navigation replay fails because the test requires tapping on a certain item in a collection where only some items have the required condition (e.g., have a subscription available) but the planner agent typically suggests activating the first item. In other cases, the planner agent cannot deduce enough knowledge about the app and predicts that key functionality for the replay does not exist in the app. In a few cases, key UI elements needed to be activated for the test that were located offscreen and required scrolling to reach, and AXNav did not continue scrolling long enough to find them. Another challenge we have seen is that the planner agent sometimes is unable to determine when to stop and gets into an infinite loop. These are areas we hope to improve in future work.

We conducted 10 1-to-1 interview-based study sessions. During each session, we first presented an overview of AXNav to the participant. We then showed three videos generated by AXNav and the associated test instructions, in randomized order. Each video showed an accessibility test on iOS media applications for e-books, news stories, and podcasts, respectively, with different UI elements and layouts. The videos were selected from the set of videos used in section 5, based on their coverage of different accessibility features, including VoiceOver, Dynamic Type, and Button Shapes. Two of the tests shown in the videos were selected from those with the difficulty level of Easy, and one test with the difficulty level of Hard. The tests shown in the videos represented real accessibility tests that our participants would perform, as they were selected from the set of test instructions authored and used by testers in the organization. We chose to show videos to participants as they are the primary output produced by AXNav, offering a realistic representation of interaction with our system. Furthermore, since AXNav is not a production system, it was not optimized for speed, and can take several minutes to an hour to produce a video. In practice, this is not a critical limitation, since many tests can be run in parallel, possibly overnight, and reviewed all at once following their completion. The specific videos and associated test instructions that we used for the user studies are as follows:

All three videos contained some accessibility issues, which we prompted the participants to discover using the heuristics as part of the system. Furthermore, all videos deliberately contained errors and imperfect navigation to conservatively showcase the capabilities of our system. Specifically, the VO video shares a podcast itself instead of an episode, and some false positive errors are flagged in the DT and BS videos. We intentionally presented those imperfections to the participants to show the performance of the system conservatively, and to trigger a discussion of limitations and future directions.

For each video, the researcher asked the participant to think aloud as they watched the video to 1) point out any accessibility issues related to the input test, and 2) point out any places where the test performed by the system could be improved. After each video, we interviewed each participant about how well the test in the video met their expectations, and how well the heuristics assisted them in finding any accessibility issues. Besides qualitative questions, we also asked the participants to provide 5-point Likert scale ratings on how similar the tests in the videos are to their manual tests, and how useful the heuristics are for tests to identify accessibility bugs. Following the viewing of all three videos, we asked about the participants’ overall attitude toward the system, how they envisioned incorporating it into their workflow, and any areas they identified for improvement. Additionally, we asked participants to provide 5-point Likert scale ratings assessing our system’s usefulness in its current form and with ideal performance within their workflow.

We recruited 10 participants who are full-time employees at a large technology company. All participants perform manual accessibility tests as part of their professional work, having professional titles of accessibility QA testers and accessibility engineers. We recruited participants via internal communication tools. In contrast to our formative study, all participants in this study were sighted and did not use screen readers. Two participants from our formative study, P5 and P6, also participated in this study. Since we did not collect information on the pronouns of our participants, we used the gender-neutral pronoun “they/them” to refer to all participants in our findings. Interview questions and participant demographics are shared in Supplemental Materials.

The data collected during the study includes audio and video recordings of the study sessions with the consent of the participants. We transcribed all the recordings into text format using an automated tool. The research team also took field notes during the session and used the notes to guide the analysis. The length of the sessions ranged from 29 minutes to 49 minutes, with an average length of 37 minutes. The interview with P9 only covered two videos (VO and BS) due to the participant’s availability.

We performed a thematic analysis on the qualitative data from the user study [25]. Two authors of the paper first individually coded all the transcripts, then presented the codes to each other and collaboratively and iteratively constructed an affinity diagram of quotes and codes together to develop themes. The following findings section presents the resulting themes. We also reported the descriptive statistics of the data collected from the Likert scale rating questions, including the mean, standard deviation (SD) and sample size (N), to supplement our qualitative insights.

AXNav employs one workflow specifically for VoiceOver tests, where the system uses forward swipes until finding a target element before activating it. As shown in the user study, this may not reflect how a VoiceOver user might navigate the task as the user may have prior knowledge of the app structure. This would enable users to skip around to various parts of the screen to activate the desired element. While sometimes such differences can be complementary test strategies, future versions of the system could explore how to simulate alternative patterns of interactions.

While AXNav achieves reasonable test replay accuracy, it can encounter errors arising from a lack of sufficient knowledge about apps or understanding when to stop (see section 5). We expect that improvements in modeling (i.e., by fine-tuning a model on successful navigation paths or integrating existing app knowledge into prompts [54]) can improve navigation performance in future versions of AXNav. Other approaches, such as using multimodal models [28], could be considered for future iterations.

Like all machine learning and heuristic-based systems, AXNav is not expected to always produce perfect output. However, it is important to mitigate the risk of these errors on QA testers. Prior works have shown there is a risk of over-reliance on AI systems since users can view the AI as an authority and be reluctant to challenge it [7, 16]. This is also the case for the navigation and heuristics of AXNav. For example, only 2 out of 10 user study participants were able to spot the navigation error in the VoiceOver example (see section 6.4.1). While evaluating the correctness of LLM-based systems remains an active area of research [29], there are additional techniques that could be considered for future work to enable AXNav to report whether it executed a navigation task correctly. For example, the navigation path itself could be evaluated through heuristics, another LLM, or by using existing knowledge of apps. Another way to mitigate over-reliance in future work would be to provide transparency signals, such as confidence scores and textual explanations of how the predictions were made, echoing design guidelines on transparency and explainability for human-AI collaboration [5].

Our user study had participants watch and comment on videos generated by AXNav. Our study design mimicked how accessibility testers would interact with AXNav in their actual workflows (i.e., reviewing videos generated by an automatic system and spotting accessibility issues, as elaborated in section 6.4.3), but this design has some limitations. First, we only showed the same set of 3 videos to all the participants. Although the set of videos covers different types of accessibility tests, participants’ feedback could be biased by this limited set of examples. Second, we only showed users videos where navigation mostly worked to probe how they would use the system in their workflow. We did not show examples where the replay failed, and therefore were not able to collect user feedback on failed replay and how it would be handled. Third, in order to keep user study sessions short, the participants did not directly write their own tests and generate videos using the tool themselves. In future work, we plan to deploy AXNav in a longitudinal study so that we can better understand how QA testers instruct the system and interact with its output.

One key limitation of AXNav currently is its output video format which is not by default accessible to screen reader users. People with disabilities are commonly employed in accessibility testing such as non-sighted screen reader testers. AXNav should make the video format accessible by ensuring all visual content is described – such as heuristic boxes, screen changes, and chapter annotations. Non-sighted users may also find other output formats more useful. The screen reader user in our formative study requested AXNav replay test cases live on a local device to enable them to take control, which is feasible and something we plan to do in future work. Lastly, future versions of AXNav should be accessible to testers beyond screen reader use cases (e.g., testers with motor impairments).

Our studies uncovered the need to support testing additional accessibility features beyond the four that AXNav supports. Future versions of AXNav can support more navigational accessibility services (e.g., Voice Control) and other accessibility settings (e.g., display features such as contrast adjustment and motion reduction) provided the device’s operating system provides APIs to control those features. AXNav currently surfaces some potential accessibility issues through its heuristics (e.g., Dynamic Type resizing issues); however, these do not cover all accessibility issues we could surface. Future versions of AXNav could incorporate existing accessibility inspection tools similar to Groundhog [46] to report issues such as missing UI element descriptions or minimum target sizes. We could also add a dashboard to summarize the issues found during AXNav’s replay, as study participants proposed. AXNav could also consider focused testing for specific accessibility needs. For example, if a test is for users with motor impairments, issues like target size would be important to surface.

Lastly, we have only built AXNav to work with the iOS operating system. However, the system architecture and workflow should be extensible to other platforms where provided APIs are available to control the accessibility features under test. A body of work has explored general UI navigation in other platforms [22, 33, 49, 54].

We have so far evaluated AXNav for QA testing, but there are many opportunities beyond this as indicated by our user study and other work in this area. One that we would like to explore is using this system as a tool to help novice developers better understand the behaviors of accessibility features and how they should be tested by generating realistic simulations of behavior on their own apps. Additionally, natural language instructions are used in manual UI testing, bug reports, and reproduction steps [22], and natural language automation systems may benefit from the techniques we present in this paper to reconstruct these types of tests. These are examples of use cases we hope to explore in future work.