Visitar URL original
Report individual test results to LUCI ResultDB by chingjun · Pull Request #190254 · flutter/flutter · GitHub
Skip to content

Report individual test results to LUCI ResultDB - #190254

Merged
auto-submit[bot] merged 4 commits into
flutter:masterfrom
chingjun:test_result
Aug 12, 2026
Merged

Report individual test results to LUCI ResultDB#190254
auto-submit[bot] merged 4 commits into
flutter:masterfrom
chingjun:test_result

Conversation

@chingjun

@chingjun chingjun commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Framework test shards previously only surfaced a single pass/fail status at the shard level, so the "Test Results" tab for a build in LUCI was effectively empty. This reports each individual test case to the build's ResultDB invocation, so the tab lists every test that ran.

This PR populates the "test results" tab with actual information about the tests. Example: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20framework_tests_widgets/109467/test-results

@chingjun chingjun added the CICD Run CI/CD label Jul 29, 2026
@chingjun
chingjun requested a review from jtmcdole July 30, 2026 00:10
@chingjun
chingjun marked this pull request as ready for review July 30, 2026 00:10
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jul 30, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request integrates LUCI ResultDB reporting into the Flutter test runner, allowing individual test results to be uploaded directly. It introduces the ResultDbRecorder client, parses test cases from the JSON reporter, and updates runDartTest and runFlutterTest to report results. Unit tests are also added to verify the implementation. Feedback on the changes suggests handling potential exceptions during LUCI_CONTEXT reading and recorder initialization to prevent test runner crashes, as well as optimizing string truncation to avoid performance bottlenecks with very long test names.

Comment thread dev/bots/luci_resultdb.dart Resolved
Comment thread dev/bots/utils.dart Resolved
Comment thread dev/bots/luci_resultdb.dart Resolved
jtmcdole
jtmcdole reviewed Aug 10, 2026
View reviewed changes
Comment thread dev/bots/luci_resultdb.dart Resolved
jtmcdole
jtmcdole reviewed Aug 11, 2026
View reviewed changes
Comment thread dev/bots/test/luci_resultdb_test.dart
Comment on lines +51 to +54
for (final LuciTestResult(
testId: LuciStructuredTestId(:moduleName, :moduleScheme, :moduleVariant),
)
in results) {

@jtmcdole jtmcdole Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

jtmcdole
jtmcdole previously approved these changes Aug 11, 2026
View reviewed changes
Comment thread dev/bots/luci_resultdb.dart Resolved
@jtmcdole jtmcdole added the autosubmit Merge PR when tree becomes green via auto submit App label Aug 12, 2026
@auto-submit auto-submit Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Aug 12, 2026
@auto-submit

auto-submit Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

autosubmit label was removed for flutter/flutter/190254, because - The status or check suite Dashboard Checks has failed. Please fix the issues identified (or deflake) before re-applying this label.

@chingjun
Report individual test results to LUCI ResultDB
e971dbd 
Framework test shards previously only surfaced a single pass/fail status at the
shard level, so the "Test Results" tab for a build in LUCI was effectively
empty. This reports each individual test case to the build's ResultDB
invocation, so the tab lists every test that ran.

Details:
- Parse the per-shard test metrics (already emitted by the JSON reporter) into
  structured TestResult objects (tool_subsharding.dart).
- Add a ResultDB Recorder client (luci_resultdb.dart) that uploads results via
  BatchCreateTestResults, authorized by the update_token from
  LUCI_CONTEXT["resultdb"]. This works for the Flutter recipe, which exposes an
  invocation but does not run `rdb stream`. It is a no-op off LUCI.
- Convert each case to a structured test id (testIdStructured, `flat` scheme) so
  the test file (module) and the test name (case) show as separate columns.
- Report module names relative to the repository root: relative suite paths are
  resolved against the shard's working directory and separators normalized, so
  ids are consistent regardless of where the shard ran.
- Mark results from expect-failure runs (e.g. test_smoke_test) as expected
  failures so they are not surfaced as red regressions.
- Model the ResultDB format with typed classes (LuciTestResult /
  LuciStructuredTestId) and add unit tests.
@chingjun
Address review: harden ResultDB reporting against failures
c40c7cc 
- readLuciContext: wrap read+decode in try/catch so an unreadable file or
  malformed JSON returns null instead of crashing the test run (it is called
  outside the reporting try/catch).
- reportTestResultsToResultDb: initialize ResultDbRecorder.fromEnvironment()
  inside the try/catch so initialization failures are logged, not fatal.
- _truncateToBytes: truncate to maxBytes code units before the byte-length
  loop to avoid O(N^2) behavior on very long names.
- Add a regression test for malformed LUCI_CONTEXT.
@chingjun
Use modern Dart features
16aff42
@chingjun
Address review comment
6861af9
@chingjun chingjun added the autosubmit Merge PR when tree becomes green via auto submit App label Aug 12, 2026
@auto-submit
auto-submit Bot added this pull request to the merge queue Aug 12, 2026
Merged via the queue into flutter:master with commit 8dee36b Aug 12, 2026
22 checks passed
@flutter-dashboard flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Aug 12, 2026
@engine-flutter-autoroll engine-flutter-autoroll mentioned this pull request Aug 12, 2026
Merged
auto-submit Bot pushed a commit to flutter/packages that referenced this pull request Aug 13, 2026
@engine-flutter-autoroll
Manual roll Flutter from 27b098811f3b to c2437523d308 (179 revisions) (…
a3e763e 
…#12453)

Manual roll Flutter from 27b098811f3b to c2437523d308 (179 revisions)

Manual roll requested by tarrinneal@google.com

flutter/flutter@27b0988...c243752

2026-08-12 matt.boetger@gmail.com Enable Gradle CI cache on all test targets that require android_sdk (flutter/flutter#190723)
2026-08-12 bkonyi@google.com [analysis] Reland "Added initial implementation of the flutter_analyzer_plugin (#175679)" (flutter/flutter#191022)
2026-08-12 matt.boetger@gmail.com Switch testing to gradle bin distribution type instead of all (flutter/flutter#190738)
2026-08-12 matt.boetger@gmail.com Convert Mockito instances in Kotlin to Mockk (flutter/flutter#189884)
2026-08-12 chingjun@google.com Report individual test results to LUCI ResultDB (flutter/flutter#190254)
2026-08-12 victorsanniay@gmail.com Toggleable reaction duration respects overrides (flutter/flutter#190857)
2026-08-12 engine-flutter-autoroll@skia.org Roll Skia from e00dbd7448c4 to fee7272f5bc2 (1 revision) (flutter/flutter#191007)
2026-08-12 30870216+gaaclarke@users.noreply.github.com Started caching text shadows by content. (flutter/flutter#190681)
2026-08-12 30870216+gaaclarke@users.noreply.github.com Adds agent skill for spawning led tasks. (flutter/flutter#190937)
2026-08-12 engine-flutter-autoroll@skia.org Roll Packages from aaaf246 to 94485f1 (8 revisions) (flutter/flutter#191008)
2026-08-12 82978131+herdiyana256@users.noreply.github.com flutter_tools: validate plugin identifiers before generating registrant code (flutter/flutter#190462)
2026-08-12 engine-flutter-autoroll@skia.org Roll Skia from 112f36148949 to e00dbd7448c4 (3 revisions) (flutter/flutter#190993)
2026-08-12 engine-flutter-autoroll@skia.org Roll Skia from 7d366c802307 to 112f36148949 (3 revisions) (flutter/flutter#190983)
2026-08-12 okorohelijah@google.com remove bringup for flavors test (flutter/flutter#190940)
2026-08-12 engine-flutter-autoroll@skia.org Roll Skia from 1f10a20bdd61 to 7d366c802307 (2 revisions) (flutter/flutter#190980)
2026-08-12 116356835+AbdeMohlbi@users.noreply.github.com Remove `--no-sim-use-hardfp` flag (flutter/flutter#190790)
2026-08-12 137456488+flutter-pub-roller-bot@users.noreply.github.com Roll pub packages (flutter/flutter#190977)
2026-08-12 engine-flutter-autoroll@skia.org Roll Skia from 339bedab6766 to 1f10a20bdd61 (1 revision) (flutter/flutter#190975)
2026-08-12 victorsanniay@gmail.com RawTooltip respects AnimationStyle updates and reverseCurve (flutter/flutter#190889)
2026-08-12 chris@bracken.jp ci: Support --target_arch option in prepare_package.dart (flutter/flutter#190960)
2026-08-12 engine-flutter-autoroll@skia.org Roll Fuchsia Linux SDK from SFq4FVodIOQAS26Lr... to -uHuSGv3wt7QAlDwa... (flutter/flutter#190973)
2026-08-12 aam@google.com Removes building of ci/android_debug_x86 as nobody should be consuming it. (flutter/flutter#190951)
2026-08-12 30870216+gaaclarke@users.noreply.github.com Adds error about wimp_heavy not being implemented. (flutter/flutter#189945)
2026-08-11 robert.ancell@canonical.com Add clang, cmake, and ninja deps to Linux windowing_test (flutter/flutter#190119)
2026-08-11 269567208+reidbaker-agent@users.noreply.github.com [AGP 9.1.0 Migration #1] Add Android Gradle Plugin Public API migration documentation (flutter/flutter#190842)
2026-08-11 bkonyi@google.com [flutter_tools] Fix deadlock in debug adapters when process exits early (flutter/flutter#190931)
2026-08-11 bkonyi@google.com [tool] Define modular dependency injection containers and bootstrapper (flutter/flutter#190724)
2026-08-11 1961493+harryterkelsen@users.noreply.github.com [web] Unify MaskFilter and ColorFilter primitives across CanvasKit and Skwasm (flutter/flutter#190314)
2026-08-11 137456488+flutter-pub-roller-bot@users.noreply.github.com Roll pub packages (flutter/flutter#190958)
2026-08-11 33794642+FelixMittermeier@users.noreply.github.com [Impeller] Move image upload scheduling waits to GPU disable (flutter/flutter#190445)
2026-08-11 30870216+gaaclarke@users.noreply.github.com Started generating the windows platform for macrobenchmarks (flutter/flutter#190932)
2026-08-11 1961493+harryterkelsen@users.noreply.github.com [web] Unify ui.Vertices (flutter/flutter#190563)
2026-08-11 jhy03261997@gmail.com Fix accessibility_inspector service extensions map mutability (flutter/flutter#190888)
2026-08-11 47866232+chunhtai@users.noreply.github.com Add batch3 a11y_assessment for vpat (flutter/flutter#189042)
2026-08-11 15619084+vashworth@users.noreply.github.com Remove Xcode environment when building swift tools in Xcode pre-action (flutter/flutter#190848)
2026-08-11 mdebbar@google.com [tool] Add missing play element in web test index.html to fix warning (flutter/flutter#190675)
2026-08-11 kkmk1999@gmail.com Offload blocking work in ProcessTextPlugin to the background (flutter/flutter#189823)
2026-08-11 bkonyi@google.com [flutter_tools] Replace usages of package:dds/dap.dart with package:dap_adapters/dap_adapters.dart (flutter/flutter#190667)
2026-08-11 15619084+vashworth@users.noreply.github.com Always update swift package dependencies (flutter/flutter#190886)
2026-08-11 bkonyi@google.com [flutter_tools] Add --preset option to flutter test (flutter/flutter#190878)
2026-08-11 jmccandless@google.com Include the examples cross imports checker in the analzyer. (flutter/flutter#190674)
2026-08-11 bkonyi@google.com [devicelab] Remove orphaned screenshot test files (flutter/flutter#190879)
2026-08-11 jason-simmons@users.noreply.github.com Remove the bringup flag from the linux_arm_host_desktop_engine builder (flutter/flutter#190935)
2026-08-11 mdebbar@google.com Reduce web_skwasm_tests subshards from 8 to 2 (flutter/flutter#190728)
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jtmcdole jtmcdole jtmcdole approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CICD Run CI/CD

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@chingjun @jtmcdole