WebNN Testing

This guide covers the testing methodologies and tools available for validating WebNN implementations across different browsers and devices.

Prerequisites

Before testing WebNN functionality, ensure you have properly configured your browser by following the WebNN Installation guide for your target backend (e.g., WebNN Windows ML / OpenVINO EP GPU).

Test Plan

WebNN testing encompasses multiple validation approaches to cover various test suites to ensure correctness, performance, security and stability.

Test Infrastructure

• Chromium bots: Continuous integration testing covers unit tests and WPT conformance tests
• Microsoft Edge team testing facilities
  • WebNN Automation Tests , covers WPT, samples and demos
• W3C WebML community testing facilities
  • WebNN Samples Test Framework , covers WebNN samples and demos
• Other hardware vendor testing facilities

Backends

WebNN is validated against multiple versions of backends implementation:

• Public releases: Stable Windows ML, Execution Providers, and GPU, NPU drivers
  • IHV EP certification test suites
• Pre-releases

Test Devices

WebNN ensures testing is conducted across all major hardware configurations and will promptly support the latest devices.

Test Cadence

In addition to continuous integration testing for all code changes, WebNN follows a scheduled testing strategy:

• Nightly: WPT and demo automation tests
• Daily: Performance and model tests
• Continuous Fuzzing: Fuzz targets are automatically discovered by buildbots, built with different fuzzing engines, then uploaded to the distributed ClusterFuzz system to run at scale
• On-demand: PageHeap testing before major releases for debugging purposes

WebNN Test Suites

Web Platform Tests (WPT)

The WebNN API follows W3C standards and includes comprehensive test suites to ensure cross-browser compatibility.

WebNN Web Platform Tests / Conformance Test Suite

The WebNN Conformance Test Suite (CTS)  is part of the web-platform-tests  project, a cross-browser test suite that ensures Web platform compatibility across all major browsers.

These tests provide confidence to:

• Browser implementers that their WebNN implementation is compatible with other browsers
• Web developers that they can rely on consistent WebNN behavior across browsers and devices

Web IDL Tests

File: idlharness.https.any.js

Validates that the WebNN API correctly implements the interfaces defined in the Web IDL specification. The test uses a daily-updated snapshot of the spec’s Web IDL located at ../interfaces/webnn.idl.

Validation Tests

Location: validation_tests/

These tests verify that WebNN methods properly validate input parameters according to the specification algorithms. They ensure invalid inputs are correctly rejected.

Example: validation_tests/matmul.https.any.js confirms that:

• Exceptions are thrown when input tensor ranks are less than 2
• Input tensor shapes must be broadcastable
• Other parameter validation requirements are met

Conformance Tests

Location: conformance_tests/

These tests verify that WebNN operators produce correct computational results.

Example: conformance_tests/matmul.https.any.js validates that N-dimensional matrix multiplication produces mathematically accurate outputs.

Test Execution Contexts

Device Variations

WebNN supports multiple compute devices (CPU, GPU, NPU). Tests use META: variant=?... to run the same test across different device types automatically.

Global Contexts

WebNN interfaces are available in both Window and Worker contexts via navigator.ml. Tests use:

• .any.js convention
• META: global=window,dedicatedworker

This allows the same test to run in multiple JavaScript execution contexts.

Note: Most operator-specific tests only specify global=window to reduce compute overhead, as behavior differences between contexts for individual operators are unlikely.

Running WPT Tests

You can also run WPT Automation Tests via WebNN Automation Tests 

Samples & Developer Preview Demos Testing

Interactive samples and demos help validate real-world WebNN usage patterns.

• W3C WebNN Samples  demonstrating practical WebNN API usage in web applications
• WebNN Developer Preview Demos  showcases featuring WebNN API integration with ONNX Runtime Web
• HuggingFace: WebNN Spaces  implements more Transformers.js based WebNN examples

Testing Sample and Demo Applications

Automated test frameworks are available for comprehensive sample and demo testing:

Fuzz Testing in Chromium

Fuzzing is an automated software testing technique that provides invalid, unexpected, or random data as inputs to a program. It is one of the most effective methods for discovering security vulnerabilities and stability issues in code.

In the context of WebNN, fuzz testing helps identify edge cases and potential crashes by generating a wide variety of input combinations that might not be covered by traditional test cases.

Testing under PageHeap

Note: Full-page heap verification can cause system memory shortages due to its high memory consumption. It should be used selectively during development and testing, not in production environments.

What is PageHeap?

PageHeap is a Windows diagnostic feature that helps detect memory access violations. It reserves memory at the boundary of each allocation to detect attempts to access memory beyond the allocated region. This catches common memory bugs such as:

• Buffer overruns
• Use-after-free errors
• Out-of-bounds memory access

PageHeap Modes

GFlags provides two PageHeap verification modes:

• Standard heap verification Writes fill patterns at the end of each heap allocation and examines these patterns when allocations are freed. This mode has minimal performance impact.
• Full-page heap verification Places an inaccessible guard page at the end of each allocation, causing the program to stop immediately if it accesses memory beyond the allocation. This provides stronger guarantees but uses significantly more memory (one full page per allocation).

Why test WebNN under PageHeap?

Testing WebNN under PageHeap provides several benefits:

• Early detection: Memory violations are caught immediately at the point of occurrence, making bugs easier to diagnose and fix.
• Security assurance: Helps identify potential security vulnerabilities related to memory handling before they reach production.
• Comprehensive coverage: When combined with fuzzing, PageHeap ensures that automatically-generated test inputs don’t trigger hidden memory safety issues.

Testing WebNN under PageHeap

Performance Testing

WebNN performance is evaluated across hundreds of AI models to ensure optimal performance across different WebNN backends. Each WebNN backend is tested against an extensive model suite spanning various architectures, sizes, and use cases to provide comprehensive performance insights.

For comprehensive WebNN performance testing tools and Web AI benchmarking website, contact the Intel Web Team.

Additional Resources

• WebNN API Specification  - Complete technical specification
• Web Platform Tests  - Cross-browser testing initiative
• GFlags and PageHeap