[chromium-dev] Stale test expectations

[Łukasz Anforowicz]

Hello,

I have a spreadsheet that lists 152 bugs
<https://docs.google.com/spreadsheets/d/14NKy1kbi2FQ3J9_eXCYxpeLajBQLz7Wc7Tw1AMlu9Ns>
that are closed (fixed / wontfix / etc) but that are still associated with
skipped or failing tests in
third_party/WebKit/LayoutTests/TestExpectations. Either 1) the bug got
closed correctly and the test expectations are stale and should be removed
or 2) the bug is still not addressed as shown by tests not passing. In
both cases, things don't seem right. Right?

QUESTION: What is the best way to help get rid of these stale
LayoutTests/TestExpectations?

- Just ask chromium-dev(s)@ to take a look at the spreadsheet to check
if they are an owner of one such bug / test expectation?
- Add a comment to such closed bugs? Reactivate them (some of them are
launch bugs)? Open new bugs?
- Does anyone have a list of best practices for doing automated bug
operations? (e.g. mark a bug with a label like
"StaleLayoutTestExpectation" and avoid touching the bug (and adding a
duplicate comment or reactivating again) if the label is already there?)


Thanks,

Lukasz

PS. The spreadsheet was gathered as part of https://crbug.com/899367.
Suggestions are welcomed for how to detect similarly stale, disabled gtest
tests (and TODO comments in general?)

PPS. Note that the spreadsheet should be visible to @google.com accounts (I
had trouble creating the spreadsheet using my @chromium.org account - see
https://bugs.chromium.org/p/monorail/issues/detail?id=4509#c5) and also to
***@chromium.org.

[Łukasz Anforowicz]

+blink-dev@ since there were no responses on chromium-dev@ (apart from one
private/direct email that I replied to directly)

Post by Łukasz Anforowicz
Hello,
I have a spreadsheet that lists 152 bugs
<https://docs.google.com/spreadsheets/d/14NKy1kbi2FQ3J9_eXCYxpeLajBQLz7Wc7Tw1AMlu9Ns>
that are closed (fixed / wontfix / etc) but that are still associated with
skipped or failing tests in
third_party/WebKit/LayoutTests/TestExpectations. Either 1) the bug got
closed correctly and the test expectations are stale and should be removed
or 2) the bug is still not addressed as shown by tests not passing. In
both cases, things don't seem right. Right?
QUESTION: What is the best way to help get rid of these stale
LayoutTests/TestExpectations?
if they are an owner of one such bug / test expectation?
- Add a comment to such closed bugs? Reactivate them (some of them
are launch bugs)? Open new bugs?
- Does anyone have a list of best practices for doing automated bug
operations? (e.g. mark a bug with a label like
"StaleLayoutTestExpectation" and avoid touching the bug (and adding a
duplicate comment or reactivating again) if the label is already there?)
Thanks,
Lukasz
PS. The spreadsheet was gathered as part of https://crbug.com/899367.
Suggestions are welcomed for how to detect similarly stale, disabled gtest
tests (and TODO comments in general?)
see https://bugs.chromium.org/p/monorail/issues/detail?id=4509#c5) and

[Peter Kasting]

Post by Łukasz Anforowicz
Hello,
I have a spreadsheet that lists 152 bugs
<https://docs.google.com/spreadsheets/d/14NKy1kbi2FQ3J9_eXCYxpeLajBQLz7Wc7Tw1AMlu9Ns>
that are closed (fixed / wontfix / etc) but that are still associated with
skipped or failing tests in
third_party/WebKit/LayoutTests/TestExpectations. Either 1) the bug got
closed correctly and the test expectations are stale and should be removed
or 2) the bug is still not addressed as shown by tests not passing. In
both cases, things don't seem right. Right?
QUESTION: What is the best way to help get rid of these stale
LayoutTests/TestExpectations?

This seems like the ideal, but is a lot of work on your side:

Try to remove the expectations. For things which pass locally and on the
trybots, land, and watch the bots (in the case of flaky tests, for the next
week) for subsequent failures. In all cases, for failing tests, ensure
there's an open bug that accurately captures the failure you saw; assign
those bugs to relevant owners to fix.

This is the sort of thing that can be parallelized among several people to
work on if you want; my team is doing something very similar for disabled
UI tests right now. However, those people have to agree in advance to work
on this -- trying to parallelize by sharding to unsuspecting owners
up-front will result in a lot of your requests being ignored :/

PK

[Stephen Chenney]

For Rendering Core we have a quarterly goal of reviewing and organizing
expectations. I'll be reviewing all of the paint, compositing, filters,
hit-testing, and svg before the end of the year.

There is value in the responsible teams doing it because they can asses
when rebaselines are acceptable.

Responsible teams can be found in the OWNERS for various test directories,
including emails to contact.

Stephen.

Post by Peter Kasting

Post by Łukasz Anforowicz
Hello,
I have a spreadsheet that lists 152 bugs
<https://docs.google.com/spreadsheets/d/14NKy1kbi2FQ3J9_eXCYxpeLajBQLz7Wc7Tw1AMlu9Ns>
that are closed (fixed / wontfix / etc) but that are still associated with
skipped or failing tests in
third_party/WebKit/LayoutTests/TestExpectations. Either 1) the bug got
closed correctly and the test expectations are stale and should be removed
or 2) the bug is still not addressed as shown by tests not passing. In
both cases, things don't seem right. Right?
QUESTION: What is the best way to help get rid of these stale
LayoutTests/TestExpectations?

Try to remove the expectations. For things which pass locally and on the
trybots, land, and watch the bots (in the case of flaky tests, for the next
week) for subsequent failures. In all cases, for failing tests, ensure
there's an open bug that accurately captures the failure you saw; assign
those bugs to relevant owners to fix.
This is the sort of thing that can be parallelized among several people to
work on if you want; my team is doing something very similar for disabled
UI tests right now. However, those people have to agree in advance to work
on this -- trying to parallelize by sharding to unsuspecting owners
up-front will result in a lot of your requests being ignored :/
PK
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFD%3DPLY4bJL6mFdbBO1VC2YMXfn8Wz2pRbqVTj3J1jrrLg%40mail.gmail.com
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFD%3DPLY4bJL6mFdbBO1VC2YMXfn8Wz2pRbqVTj3J1jrrLg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.