Repository navigation
feat: a11y testing setup #3177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
matthewlipski
wants to merge
47
commits into
main
Choose a base branch
from
a11y-testing
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
feat: a11y testing setup #3177
Changes from all commits
Commits
Show all changes
47 commits
Select commit
Hold shift + click to select a range
35b14f2
Initial a11y testing setup
matthewlipski a974076
Migrated from Playwright to Vitest Browser Mode
matthewlipski 1fd7af8
Small refactor
matthewlipski 9dee776
Moved axe scans to separate test files
matthewlipski 177e755
Added read-only tests and moved interactive block tests vertical slic…
matthewlipski 1de7a16
Removed `crypto.randomUUID` calls and added lint rule to disallow `ge…
matthewlipski 75e158a
Made `saveTranscript` throw error when `screenReader` is not defined.
matthewlipski a6bafae
Changed animation handling for axe scans
matthewlipski 66f1a48
Rename `spec` -> `test`
matthewlipski c92db4b
Added `@root` shorthand
matthewlipski 463e237
Cleanup
matthewlipski 7cf0798
Implemented remaining PR feedback
matthewlipski 137f4d4
Small axe changes
matthewlipski 8fcb355
Reworked axe scan document fixtures to cover much more issues (contra…
matthewlipski 2c218d8
Merged main
matthewlipski f331a87
Skipped failing tests
matthewlipski e9ff65d
Updated GuidePup
matthewlipski 2a5856f
Added comments for skipped tests
matthewlipski 640faa9
Added ARIA snapshotting, removed screen reader testing from CI
matthewlipski 8941dc5
Merge branch 'main' into a11y-testing
matthewlipski e7dd7ba
Merged main
matthewlipski f4a2e18
Updated visual snapshots with masks for videos
matthewlipski 71930dd
Fixed flaky behaviour in formatting toolbar block type select focus
matthewlipski 72eaa23
Updated a11y.yml
matthewlipski 8e6108f
Reverted better auth version update in lockfile
matthewlipski b02e584
Added setup script to always wait 20ms when editor is focused to wait…
matthewlipski 3503052
Refactored formatting toolbar tests
matthewlipski 26abc63
Refactored remaining tests
matthewlipski d5cdcc5
Implemented PR feedback
matthewlipski f821d53
Removed separate config and GH workflow for a11y tests
matthewlipski fb25cdf
Implemented PR feedback
matthewlipski 9ab8d5f
Merge branch 'main' into a11y-testing
matthewlipski ad280a5
Fixed build
matthewlipski 1cf937b
Updated screenshots
matthewlipski 89450d9
Fixed test
matthewlipski 2354059
Skipped failing a11y test
matthewlipski 18bcf7e
Updated agent skills
matthewlipski ae305d4
feat: a11y focus ring checks (#3308)
matthewlipski c8d90c3
Implemented PR feedback
matthewlipski 2dc8bfc
Updated skill
matthewlipski c138511
Small fixes
matthewlipski 354034c
Removed obsolete snapshots
matthewlipski 6c4be74
Refactored a11y test helpers
matthewlipski 48f95a6
Fixed review feedback
matthewlipski 852e1e7
Added lint rules for browser tests
matthewlipski c6337ef
Updated full document keyboard navigation test
matthewlipski 7208742
Added key press limit to tests in addition to time limit
matthewlipski File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 2 additions & 0 deletions
2
.claude/skills/playwright-cli/references/spec-driven-testing.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
79 changes: 79 additions & 0 deletions
79
.claude/skills/testing-skill/references/accessibility-tests.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| # Colocated component and accessibility tests | ||
|
|
||
| All paths below are relative to the repository root. These are browser integration tests, identified by the `.browser.test.ts` or `.browser.test.tsx` suffix. They use the existing e2e configuration and CI jobs, not a separate accessibility browser project or workflow. | ||
|
|
||
| ## Files and fixtures | ||
|
|
||
| Place files directly beside the component or block: | ||
|
|
||
| ```text | ||
| FormattingToolbar.tsx | ||
| FormattingToolbar.browser.test.tsx | ||
| FormattingToolbar.screen-reader.test.ts # optional, native/local only | ||
| __snapshots__/ # ARIA snapshots | ||
| __screenshots__/ # browser-specific visual baselines | ||
| ``` | ||
|
|
||
| Examples to read before extending an area: | ||
|
|
||
| - `packages/react/src/components/FormattingToolbar/FormattingToolbar.browser.test.tsx` | ||
| - `packages/react/src/editor/BlockNoteView.browser.test.tsx` | ||
| - `packages/core/src/blocks/Heading/block.browser.test.tsx` | ||
|
|
||
| Keep the browser scenarios in one file, grouped by assertion purpose. Do not split them into `.keyboard.test.ts`, `.aria.test.ts`, `.axe.test.ts`, or `.visual.test.ts` files or put them in a component-local `a11y/` directory. Only add groups relevant to the component. | ||
|
|
||
| Declare document fixtures in their consuming test files, with names ending in `Document` and typed as `PartialBlock[]`. Prefer plain data; avoid fixture-selection factories and separate `fixtures.ts` files. If browser and screen-reader tests share a fixture or helper, export it from the owning test file and import it in the other. Guard browser suite registration with `if (!screenReader)` when the native suite imports that file, so importing helpers does not collect browser scenarios in the native run. | ||
|
|
||
| Write explicit named tests for the distinct user interactions covered by the component. Name each test after the outcome it asserts, rather than only the actions it performs. Share repeated test logic through named functions near the start of the file or describe block, rather than registering different interactions in a loop. Use names specific to the component and element being tested, such as `toggleButton`, `linkInput`, or `toolbarMenu`, rather than `control`. | ||
|
|
||
| ## Describe blocks and assertion scope | ||
|
|
||
| The same keyboard actions may appear in several groups. Each group checks a different outcome; avoid adding every kind of assertion to every test. | ||
|
|
||
| | Group | Actions and assertions | | ||
| | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ||
| | `behavior` | Navigate using the keyboard and activate with Enter. Assert the component's functional effect. For document edits, subscribe with `editor.onChange(vi.fn())` after navigation and before activation, register the unsubscribe with `onTestFinished`, and assert the callback ran. View-only changes may need another observable effect: toggle-heading tests check persisted expansion state. Keep detailed editor-command unit tests colocated with those commands rather than duplicating them here. | | ||
| | `keyboard` | Assert the relevant focus transitions explicitly with `toHaveFocus()`: entering the editor, reaching the toolbar/menu/item, and returning to the editor or leaving it. Assert visibility and, for document traversal, selection/block/table-cell changes through the selection command. Check the visual focus indicator as well; DOM focus alone does not prove that focus is visible. | | ||
| | `ARIA` | Follow the keyboard interaction and compare named ARIA snapshots before and after activation to check names, roles, pressed/expanded state, etc. Scope item tests to the item: adding another toolbar button must not invalidate Bold's snapshots. Use a separate whole-toolbar tree snapshot to cover membership and overall structure. If activation changes the item's accessible name, locate its resulting name explicitly. | | ||
| | `axe` | Open the relevant UI state and run the axe command. Scan a toolbar, menu, or popover independently where appropriate; use a whole-document scan for broad schema coverage. Attach the returned manual-review warnings to the test. Axe scans do not replace keyboard navigation or focus-indicator checks. | | ||
| | `visual` | Cover visual accessibility scenarios such as text-spacing overrides and 200% text resizing. Use `screenshotFull` for tall documents, wait for fonts/media, and mask nondeterministic video frames where needed. These checks concern layout/readability rather than individual button activation and should only ever be used for specific requirements defined in the WCAG 2.0-2.2 guidelines. | | ||
|
|
||
| ## Test setup and commands | ||
|
|
||
| Define a local `TestEditor` component using `useCreateBlockNote` and `BlockNoteView`. Use `.tsx` and JSX when rendering React components. Add a `renderTestEditor` function when the suite needs repeated rendering or readiness steps. Keep document data, initial focus, and setup specific to the scenario; do not introduce a shared editor fixture for unrelated suites. | ||
|
|
||
| Consume browser operations through the typed `commands` export in `tests/src/utils/context.ts`. The implementations live in `tests/src/utils/accessibility.ts` and `tests/src/utils/keyboard.ts`; their JSDoc describes inputs, results, synchronization, and usage. Register new reusable browser operations in the relevant Vite configuration and add their client signatures to `context.ts`. Do not import command implementations into tests, add per-caller type casts, or wrap a command just to forward its arguments. | ||
|
|
||
| Keep scenario-specific sequences, such as navigating to a toolbar item, in the consuming test file. Share them only where another test actually needs the same sequence. Keep action sequences separate from behavior assertions; keyboard tests should explicitly assert the focus and selection transitions they cover. | ||
|
|
||
| The default browser setup already disables motion and tracks ProseMirror focus reconciliation. Do not duplicate those mechanisms in fixtures. Wait for observable readiness with retrying assertions instead of arbitrary sleeps. For example, an opened menu can be visible before its initial item receives focus: assert that focus before sending navigation keys. | ||
|
|
||
| Use accessible locators (`getByRole` with a name or `getByLabel`) and pass their selectors to commands. Prefer live locators for asynchronously mounted elements. When tabbing or arrowing to an item, stop as soon as the target is reached and enforce an explicit maximum number of keypresses. Choose a generous bound for the fixture and fail immediately with the target and limit in the message if it is exhausted; do not rely on the test timeout. The bound is a safety limit, not a fixed navigation sequence or a duplicate list of controls. | ||
|
|
||
| ## Native screen-reader edge cases | ||
|
|
||
| Use a separate `*.screen-reader.test.ts` only for actual reader announcements/navigation that ARIA snapshots cannot establish. Reuse keyboard navigation helpers, render the suite’s editor and call `commands.startReader(locator.selector)`, and assert the transcript and stop the reader in `afterEach`. Avoid duplicating every browser scenario. Transcripts are reader-specific; review changes rather than accepting them blindly. | ||
|
|
||
| These tests run locally with VoiceOver on macOS or NVDA on Windows and require the installed reader and GuidePup permissions. They must stay excluded from regular e2e, node tests, and CI. `tests/screen-reader/vite.config.screen-reader.ts` is for this native runner only. | ||
|
|
||
| ## Discovery and commands | ||
|
|
||
| Browser files use `*.browser.test.ts` (or `.tsx`); native screen-reader tests use `*.screen-reader.test.ts`; node tests use `*.test.ts`. Existing glob patterns discover browser tests and exclude them from node runs without per-file configuration. Accessibility checks are part of the regular suite. Production files must not import test files; lint enforces this boundary. | ||
|
|
||
| Browser and screen-reader files also have lint rules for focused tests, malformed assertions, unhandled promises, direct command-implementation imports, test-ID locators, and explicit sleep APIs (`waitForTimeout`, `.sleep`, and Node timer imports). Readiness waits should use retrying assertions. | ||
|
|
||
| Run from the repository root: | ||
|
|
||
| ```bash | ||
| # Existing e2e suite, always Docker; narrow by file, group, and/or browser. | ||
| vp run e2e FormattingToolbar.browser.test.tsx | ||
| vp run e2e FormattingToolbar.browser.test.tsx -t keyboard --project "e2e (webkit)" --retry=0 | ||
|
|
||
| # Update only the selected suite, then review and rerun without updating. | ||
| vp run e2e:updateSnaps FormattingToolbar.browser.test.tsx | ||
|
|
||
| # Native/local only, never part of regular e2e or CI. | ||
| vp run test:screen-reader FormattingToolbar.screen-reader.test.ts | ||
| ``` | ||
|
|
||
| Evaluate known functionality gaps individually; any intentional skip needs a comment explaining the specific failure. Do not skip timing failures or increase retries to hide them: fix readiness/locator handling and verify repeated runs with retries disabled. Do not refresh snapshots to accept an unexplained regression. | ||
20 changes: 20 additions & 0 deletions
20
packages/core/src/blocks/Heading/__snapshots__/block.browser.test.tsx.snap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| // Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html | ||
|
|
||
| exports[`ARIA > read-only toggle heading: Tab reaches the toggle and Enter expands/collapses it > collapsed 1`] = ` | ||
| "- textbox: | ||
| - button "Expand or collapse" | ||
| - heading "Details" [level=2]" | ||
| `; | ||
|
|
||
| exports[`ARIA > read-only toggle heading: Tab reaches the toggle and Enter expands/collapses it > collapsed again 1`] = ` | ||
| "- textbox: | ||
| - button "Expand or collapse" | ||
| - heading "Details" [level=2]" | ||
| `; | ||
|
|
||
| exports[`ARIA > read-only toggle heading: Tab reaches the toggle and Enter expands/collapses it > expanded 1`] = ` | ||
| "- textbox: | ||
| - button "Expand or collapse" [expanded] | ||
| - heading "Details" [level=2] | ||
| - paragraph: Nested read-only content" | ||
| `; |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so I understand correctly, this means that in the
.browser.test.tsfiles we don't check that an action did the "correct" thing, but just that it did "something"?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I assume (also based on standup discussions) that "full behavior" tests are then out of scope for now, which is fair. But maybe we should do the thought exercise of where the full behavior tests would / should live if / when we'd invest in those. In the same file under a different group? would they replace the
behaviorgroup? Or, the philosophy is that we should always test them at a lower level?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note from our discussion: full integration style tests would override the need for "behavior" tests in the case of the formatting toolbar, but there are cases for other UI elements where both test types are relevant. @nperez0111 will provide examples of this.