[chromium-dev] Test procedure for stable Chromium releases

[sergey y]

Hello,

Could anybody clarify following question:
How stable Chromium releases are tested before tagging with "stable tag"?

I'm asking because I downloaded Chromium with some tag (68.0.3440.91) which
was specified as stable in https://omahaproxy.appspot.com/. Then I run
layout tests (as described in
https://chromium.googlesource.com/chromium/src/+/1ded89f3aa84b59f29c34ccdf9dc2bf074fe6bae/docs/layout_tests_linux.md
<https://email-uk.huawei.com/owa/redir.aspx?C=vuP7Ai72kcqoIYkj5yzpT23K8rCv1pjzTkdz0bxZKWYBrMOKJlXWCA..&URL=https%3a%2f%2fchromium.googlesource.com%2fchromium%2fsrc%2f%2b%2f1ded89f3aa84b59f29c34ccdf9dc2bf074fe6bae%2fdocs%2flayout_tests_linux.md>).
And I got number of errors (32).
So there are some questions:
- Is there any special procedure for stable releases testing?
- Which tests are run for stable releases for each platform (i.e. which
layout tests, which unit tests, etc.)?
- How situation with failed tests is managed?
- Is it possible to find any stable versions test results somewhere (in
order to compare with my own results)?

I will very appreciated for clarification because I really worry about is
Google really releases stable versions with known but not fixed bugs.

Thank you,
Sergey.

[Peter Kasting]

I really worry about is Google really releases stable versions with known
but not fixed bugs.

The bug tracker at crbug.com lists 80,000 open bugs. Even assuming that
3/4 of them are bogus (which I suspect is too high) and that 10% of them
are too new to apply to stable (no idea, I just made up that number), that
would mean 18,000 known unfixed bugs in the current stable release.

So yes, all stable versions of Chrome (and basically all commercial
software) are shipped with known unfixed bugs. The question is the
magnitude of those bugs and the relative costs of the different ways to
address them.

PK

[Dave Tapuska]

The other thing you need to consider is that the web is a moving platform.
The standards change, new issues are found, it evolves every release if you
want to see the results of layout tests against releases checkout:
https://wpt.fyi/results/

dave.

Post by Peter Kasting

I really worry about is Google really releases stable versions with known
but not fixed bugs.

The bug tracker at crbug.com lists 80,000 open bugs. Even assuming that
3/4 of them are bogus (which I suspect is too high) and that 10% of them
are too new to apply to stable (no idea, I just made up that number), that
would mean 18,000 known unfixed bugs in the current stable release.
So yes, all stable versions of Chrome (and basically all commercial
software) are shipped with known unfixed bugs. The question is the
magnitude of those bugs and the relative costs of the different ways to
address them.
PK
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFA%3Difs2dUGuK6wuL_a04iU9qa0-m0o2fMCc5E-xRpS4gw%40mail.gmail.com
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAAHOzFA%3Difs2dUGuK6wuL_a04iU9qa0-m0o2fMCc5E-xRpS4gw%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

[sergey y]

At first thank you for the answers.

But the picture still unclear. Let me explain what I want to know in
details.
I know that a lot of tests is running for each master-branch commit and the
detail results are available at https://ci.chromium.org/p/chromium
<https://email-uk.huawei.com/owa/redir.aspx?C=Dn_pRwpBzt_3GPAhNGJYitQHjl9fDpzAqP8xuHlkddKr8vGr3FXWCA..&URL=https%3a%2f%2fci.chromium.org%2fp%2fchromium>

But according my observation some layout tests (I mean blink specific
layout test which are stored in ./third_party/blink/LayoutTests/ former
./third_party/WebKit/LayoutTests/) are fails on Chromium releases which are
announced as stable. And there no issues submitted against this fails.
Moreover I observe that sometime number of failed tests is increased from
release to release.
I get this observation when I build Chromium by myself and run Layout tests
by myself.

So my questions are more specific and related to stable releases test
procedure.
1. Is any special test procedure using for stable releases? Which tests are
run for these releases? Can I look at test results for these releases
somewhere?
2. What happens if some tests fails during stable release testing and this
fail is a new one (i.e. there is no appropriate issue in
https://bugs.chromium.org/p/chromium/issues/list
<https://email-uk.huawei.com/owa/redir.aspx?C=YpwKe57M5l4GAHNK6NmY6hqKVy5zz84MtcT8boIyHiCr8vGr3FXWCA..&URL=https%3a%2f%2fbugs.chromium.org%2fp%2fchromium%2fissues%2flist>
)?

[PhistucK]

Can you share your failures?
If you build the master branch, do you get those errors as well?
I ask because it can be an environment issue.

☆*PhistucK*

Post by sergey y
At first thank you for the answers.
But the picture still unclear. Let me explain what I want to know in
details.
I know that a lot of tests is running for each master-branch commit and
the detail results are available at https://ci.chromium.org/p/chromium
<https://email-uk.huawei.com/owa/redir.aspx?C=Dn_pRwpBzt_3GPAhNGJYitQHjl9fDpzAqP8xuHlkddKr8vGr3FXWCA..&URL=https%3a%2f%2fci.chromium.org%2fp%2fchromium>
But according my observation some layout tests (I mean blink specific
layout test which are stored in ./third_party/blink/LayoutTests/ former
./third_party/WebKit/LayoutTests/) are fails on Chromium releases which are
announced as stable. And there no issues submitted against this fails.
Moreover I observe that sometime number of failed tests is increased from
release to release.
I get this observation when I build Chromium by myself and run Layout
tests by myself.
So my questions are more specific and related to stable releases test
procedure.
1. Is any special test procedure using for stable releases? Which tests
are run for these releases? Can I look at test results for these releases
somewhere?
2. What happens if some tests fails during stable release testing and this
fail is a new one (i.e. there is no appropriate issue in
https://bugs.chromium.org/p/chromium/issues/list
<https://email-uk.huawei.com/owa/redir.aspx?C=YpwKe57M5l4GAHNK6NmY6hqKVy5zz84MtcT8boIyHiCr8vGr3FXWCA..&URL=https%3a%2f%2fbugs.chromium.org%2fp%2fchromium%2fissues%2flist>
)?
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a73bb59f-7569-49f2-ae60-40ca99275614%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a73bb59f-7569-49f2-ae60-40ca99275614%40chromium.org?utm_medium=email&utm_source=footer>
.

[sergey y]

Sure, let me replay my actions step by step

1. I download chromium repo.
2. I check-out version 68.0.3440.84 which was announced as stable for Linux.
3. I started to build Layout tests in accordance with the manual.
Then wow! I get two compilation errors! It is really strange because I'm
using "stable" version.
The errors are attached (blink_tests_build_log_68.0.3440.84.txt). These
issues do not look like environment issues.
4. I check-out next stable 68.0.3440.106 and I have same errors.
5. I find and cherry-pick upstream commits which fixes these errors (
https://chromium-review.googlesource.com/1072222 and
https://chromium-review.googlesource.com/1083491).
6. I build layout tests and run them. And I got 31 errors. Results are
attached too (blink_tests_run_log_68.0.3440.84.txt)
One issue is related to environment. I already submitted bug against it (
https://bugs.chromium.org/p/chromium/issues/detail?id=872718). But I'm not
sure about other ones.
7. Finally I check out last stable Linux version (70.0.3538.110). I build
blink_tests target with no compilation errors.
But when I run tests I get ~4500 regression errors!

And I would like to know Chromium stable versions testing process in order
to understand why I have so many issues with Layout tests.
Is it my specific issues?
Or there is no any defined test procedure for Chromium stable versions and
each one can be released with unlimited number of non-documented errors?
(sounds pretty unbelievable))

The ideal case is to look into Google's description of test procedure, test
environment and test results for stable releases. And compare these staff
with my one.
I will be very appreciated for any help.

Post by PhistucK
Can you share your failures?
If you build the master branch, do you get those errors as well?
I ask because it can be an environment issue.
☆*PhistucK*

Post by sergey y
At first thank you for the answers.
But the picture still unclear. Let me explain what I want to know in
details.
I know that a lot of tests is running for each master-branch commit and
the detail results are available at https://ci.chromium.org/p/chromium
But according my observation some layout tests (I mean blink specific
layout test which are stored in ./third_party/blink/LayoutTests/ former
./third_party/WebKit/LayoutTests/) are fails on Chromium releases which are
announced as stable. And there no issues submitted against this fails.
Moreover I observe that sometime number of failed tests is increased from
release to release.
I get this observation when I build Chromium by myself and run Layout
tests by myself.
So my questions are more specific and related to stable releases test
procedure.
1. Is any special test procedure using for stable releases? Which tests
are run for these releases? Can I look at test results for these releases
somewhere?
2. What happens if some tests fails during stable release testing and
this fail is a new one (i.e. there is no appropriate issue in
https://bugs.chromium.org/p/chromium/issues/list)?
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a73bb59f-7569-49f2-ae60-40ca99275614%40chromium.org
.

[Yuta Kitamura]

A few comments:


- Automated tests, including layout tests, are an aid for every day
development on tip-of-tree source code, so we can detect severe regressions
early. On the other hand, release branches (first beta, then stable) do not
get a lot of commits, and the focus is shifted towards fixing stability and
user issues. At this stage, we generally don't rely on automated testing
too much, and we do more manual quality-assurance testing. We generally
don't care much about automated test breakage at this stage, because the
changes to the release branches are eventually discarded.
- Layout tests have a huge number of tests, and actually so many tests
are failing "as expected". Expected failures aren't shown by default, but
you can see the list of failed tests on the results dashboard. Also, layout
tests are known to be sensitive to local environment (installed fonts,
locale, etc.), so ~30 failures don't seem too bad actually. As I said,
automated test failures on the release branch are not our main focus, so
it's not surprising there are some failures.
- I recommend running layout tests on the ToT and see if you'll get the
same results as our automated bots. They probably differ (and that's
natural).

Post by sergey y
Sure, let me replay my actions step by step
1. I download chromium repo.
2. I check-out version 68.0.3440.84 which was announced as stable for Linux.
3. I started to build Layout tests in accordance with the manual.
Then wow! I get two compilation errors! It is really strange because I'm
using "stable" version.
The errors are attached (blink_tests_build_log_68.0.3440.84.txt). These
issues do not look like environment issues.
4. I check-out next stable 68.0.3440.106 and I have same errors.
5. I find and cherry-pick upstream commits which fixes these errors (
https://chromium-review.googlesource.com/1072222 and
https://chromium-review.googlesource.com/1083491).
6. I build layout tests and run them. And I got 31 errors. Results are
attached too (blink_tests_run_log_68.0.3440.84.txt)
One issue is related to environment. I already submitted bug against it (
https://bugs.chromium.org/p/chromium/issues/detail?id=872718). But I'm
not sure about other ones.
7. Finally I check out last stable Linux version (70.0.3538.110). I build
blink_tests target with no compilation errors.
But when I run tests I get ~4500 regression errors!
And I would like to know Chromium stable versions testing process in order
to understand why I have so many issues with Layout tests.
Is it my specific issues?
Or there is no any defined test procedure for Chromium stable versions and
each one can be released with unlimited number of non-documented errors?
(sounds pretty unbelievable))
The ideal case is to look into Google's description of test procedure,
test environment and test results for stable releases. And compare these
staff with my one.
I will be very appreciated for any help.

Post by PhistucK
Can you share your failures?
If you build the master branch, do you get those errors as well?
I ask because it can be an environment issue.
☆*PhistucK*

Post by sergey y
At first thank you for the answers.
But the picture still unclear. Let me explain what I want to know in
details.
I know that a lot of tests is running for each master-branch commit and
the detail results are available at https://ci.chromium.org/p/chromium
But according my observation some layout tests (I mean blink specific
layout test which are stored in ./third_party/blink/LayoutTests/ former
./third_party/WebKit/LayoutTests/) are fails on Chromium releases which are
announced as stable. And there no issues submitted against this fails.
Moreover I observe that sometime number of failed tests is increased from
release to release.
I get this observation when I build Chromium by myself and run Layout
tests by myself.
So my questions are more specific and related to stable releases test
procedure.
1. Is any special test procedure using for stable releases? Which tests
are run for these releases? Can I look at test results for these releases
somewhere?
2. What happens if some tests fails during stable release testing and
this fail is a new one (i.e. there is no appropriate issue in
https://bugs.chromium.org/p/chromium/issues/list)?
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a73bb59f-7569-49f2-ae60-40ca99275614%40chromium.org
.

--

--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a6a0d443-8a62-4b4a-908a-ac47bd4d5f40%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a6a0d443-8a62-4b4a-908a-ac47bd4d5f40%40chromium.org?utm_medium=email&utm_source=footer>
.

[sergey y]

Thank you for clarification.

Just couple of question.

- What is ToT?
- Is it possible to setup absolutely same environment as environment which
is used for automated testing?

Post by Yuta Kitamura
- Automated tests, including layout tests, are an aid for every day
development on tip-of-tree source code, so we can detect severe regressions
early. On the other hand, release branches (first beta, then stable) do not
get a lot of commits, and the focus is shifted towards fixing stability and
user issues. At this stage, we generally don't rely on automated testing
too much, and we do more manual quality-assurance testing. We generally
don't care much about automated test breakage at this stage, because the
changes to the release branches are eventually discarded.
- Layout tests have a huge number of tests, and actually so many tests
are failing "as expected". Expected failures aren't shown by default, but
you can see the list of failed tests on the results dashboard. Also, layout
tests are known to be sensitive to local environment (installed fonts,
locale, etc.), so ~30 failures don't seem too bad actually. As I said,
automated test failures on the release branch are not our main focus, so
it's not surprising there are some failures.
- I recommend running layout tests on the ToT and see if you'll get
the same results as our automated bots. They probably differ (and that's
natural).

Post by sergey y
Sure, let me replay my actions step by step
1. I download chromium repo.
2. I check-out version 68.0.3440.84 which was announced as stable for Linux.
3. I started to build Layout tests in accordance with the manual.
Then wow! I get two compilation errors! It is really strange because
I'm using "stable" version.
The errors are attached (blink_tests_build_log_68.0.3440.84.txt). These
issues do not look like environment issues.
4. I check-out next stable 68.0.3440.106 and I have same errors.
5. I find and cherry-pick upstream commits which fixes these errors (
https://chromium-review.googlesource.com/1072222 and
https://chromium-review.googlesource.com/1083491).
6. I build layout tests and run them. And I got 31 errors. Results are
attached too (blink_tests_run_log_68.0.3440.84.txt)
One issue is related to environment. I already submitted bug against it
(https://bugs.chromium.org/p/chromium/issues/detail?id=872718). But I'm
not sure about other ones.
7. Finally I check out last stable Linux version (70.0.3538.110). I build
blink_tests target with no compilation errors.
But when I run tests I get ~4500 regression errors!
And I would like to know Chromium stable versions testing process in
order to understand why I have so many issues with Layout tests.
Is it my specific issues?
Or there is no any defined test procedure for Chromium stable versions
and each one can be released with unlimited number of non-documented
errors? (sounds pretty unbelievable))
The ideal case is to look into Google's description of test procedure,
test environment and test results for stable releases. And compare these
staff with my one.
I will be very appreciated for any help.

Post by PhistucK
Can you share your failures?
If you build the master branch, do you get those errors as well?
I ask because it can be an environment issue.
☆*PhistucK*

Post by sergey y
At first thank you for the answers.
But the picture still unclear. Let me explain what I want to know in
details.
I know that a lot of tests is running for each master-branch commit and
the detail results are available at https://ci.chromium.org/p/chromium
But according my observation some layout tests (I mean blink specific
layout test which are stored in ./third_party/blink/LayoutTests/ former
./third_party/WebKit/LayoutTests/) are fails on Chromium releases which are
announced as stable. And there no issues submitted against this fails.
Moreover I observe that sometime number of failed tests is increased from
release to release.
I get this observation when I build Chromium by myself and run Layout
tests by myself.
So my questions are more specific and related to stable releases test
procedure.
1. Is any special test procedure using for stable releases? Which tests
are run for these releases? Can I look at test results for these releases
somewhere?
2. What happens if some tests fails during stable release testing and
this fail is a new one (i.e. there is no appropriate issue in
https://bugs.chromium.org/p/chromium/issues/list)?
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a73bb59f-7569-49f2-ae60-40ca99275614%40chromium.org
.

--

--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a6a0d443-8a62-4b4a-908a-ac47bd4d5f40%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a6a0d443-8a62-4b4a-908a-ac47bd4d5f40%40chromium.org?utm_medium=email&utm_source=footer>
.

[Yuta Kitamura]

Post by sergey y
Thank you for clarification.
Just couple of question.
- What is ToT?

Tip-of-tree. The latest revision of master branch. See also
https://www.chromium.org/glossary

Post by sergey y
- Is it possible to setup absolutely same environment as environment which
is used for automated testing?

Hmmm, I don't have first-hand knowledge of this. This largely depends on
the platform (Win, Mac, Linux, Android?), but it should not be too hard to
have an environment that's mostly the same.

Post by sergey y

Post by Yuta Kitamura
- Automated tests, including layout tests, are an aid for every day
development on tip-of-tree source code, so we can detect severe regressions
early. On the other hand, release branches (first beta, then stable) do not
get a lot of commits, and the focus is shifted towards fixing stability and
user issues. At this stage, we generally don't rely on automated testing
too much, and we do more manual quality-assurance testing. We generally
don't care much about automated test breakage at this stage, because the
changes to the release branches are eventually discarded.
- Layout tests have a huge number of tests, and actually so many
tests are failing "as expected". Expected failures aren't shown by default,
but you can see the list of failed tests on the results dashboard. Also,
layout tests are known to be sensitive to local environment (installed
fonts, locale, etc.), so ~30 failures don't seem too bad actually. As I
said, automated test failures on the release branch are not our main focus,
so it's not surprising there are some failures.
- I recommend running layout tests on the ToT and see if you'll get
the same results as our automated bots. They probably differ (and that's
natural).

Post by sergey y
Sure, let me replay my actions step by step
1. I download chromium repo.
2. I check-out version 68.0.3440.84 which was announced as stable for Linux.
3. I started to build Layout tests in accordance with the manual.
Then wow! I get two compilation errors! It is really strange because
I'm using "stable" version.
The errors are attached (blink_tests_build_log_68.0.3440.84.txt).
These issues do not look like environment issues.
4. I check-out next stable 68.0.3440.106 and I have same errors.
5. I find and cherry-pick upstream commits which fixes these errors (
https://chromium-review.googlesource.com/1072222 and
https://chromium-review.googlesource.com/1083491).
6. I build layout tests and run them. And I got 31 errors. Results are
attached too (blink_tests_run_log_68.0.3440.84.txt)
One issue is related to environment. I already submitted bug against
it (https://bugs.chromium.org/p/chromium/issues/detail?id=872718). But
I'm not sure about other ones.
7. Finally I check out last stable Linux version (70.0.3538.110). I
build blink_tests target with no compilation errors.
But when I run tests I get ~4500 regression errors!
And I would like to know Chromium stable versions testing process in
order to understand why I have so many issues with Layout tests.
Is it my specific issues?
Or there is no any defined test procedure for Chromium stable versions
and each one can be released with unlimited number of non-documented
errors? (sounds pretty unbelievable))
The ideal case is to look into Google's description of test procedure,
test environment and test results for stable releases. And compare these
staff with my one.
I will be very appreciated for any help.

Post by PhistucK
Can you share your failures?
If you build the master branch, do you get those errors as well?
I ask because it can be an environment issue.
☆*PhistucK*

Post by sergey y
At first thank you for the answers.
But the picture still unclear. Let me explain what I want to know in
details.
I know that a lot of tests is running for each master-branch commit
and the detail results are available at
https://ci.chromium.org/p/chromium
But according my observation some layout tests (I mean blink specific
layout test which are stored in ./third_party/blink/LayoutTests/ former
./third_party/WebKit/LayoutTests/) are fails on Chromium releases which are
announced as stable. And there no issues submitted against this
fails. Moreover I observe that sometime number of failed tests is increased
from release to release.
I get this observation when I build Chromium by myself and run Layout
tests by myself.
So my questions are more specific and related to stable releases test
procedure.
1. Is any special test procedure using for stable releases? Which
tests are run for these releases? Can I look at test results for these
releases somewhere?
2. What happens if some tests fails during stable release testing and
this fail is a new one (i.e. there is no appropriate issue in
https://bugs.chromium.org/p/chromium/issues/list)?
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a73bb59f-7569-49f2-ae60-40ca99275614%40chromium.org
.

--

--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a6a0d443-8a62-4b4a-908a-ac47bd4d5f40%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a6a0d443-8a62-4b4a-908a-ac47bd4d5f40%40chromium.org?utm_medium=email&utm_source=footer>
.

--

--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/2380dce9-89e1-4882-a5ff-29d5c0217610%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/2380dce9-89e1-4882-a5ff-29d5c0217610%40chromium.org?utm_medium=email&utm_source=footer>
.

[sergey y]

Post by Yuta Kitamura
Hmmm, I don't have first-hand knowledge of this. This largely depends on
the platform (Win, Mac, Linux, Android?), but it should not be too hard to
have an environment that's mostly the same.

Could you please advise what is the best way to find knowledge about
"reference" Google environment for some specific platform?

[PhistucK]

Wait, so do all of the layout tests (and any other test) run on release
branches?
Are unexpected results being looked at at all?

☆*PhistucK*

Post by Yuta Kitamura
- Automated tests, including layout tests, are an aid for every day
development on tip-of-tree source code, so we can detect severe regressions
early. On the other hand, release branches (first beta, then stable) do not
get a lot of commits, and the focus is shifted towards fixing stability and
user issues. At this stage, we generally don't rely on automated testing
too much, and we do more manual quality-assurance testing. We generally
don't care much about automated test breakage at this stage, because the
changes to the release branches are eventually discarded.
- Layout tests have a huge number of tests, and actually so many tests
are failing "as expected". Expected failures aren't shown by default, but
you can see the list of failed tests on the results dashboard. Also, layout
tests are known to be sensitive to local environment (installed fonts,
locale, etc.), so ~30 failures don't seem too bad actually. As I said,
automated test failures on the release branch are not our main focus, so
it's not surprising there are some failures.
- I recommend running layout tests on the ToT and see if you'll get
the same results as our automated bots. They probably differ (and that's
natural).

Post by sergey y
Sure, let me replay my actions step by step
1. I download chromium repo.
2. I check-out version 68.0.3440.84 which was announced as stable for Linux.
3. I started to build Layout tests in accordance with the manual.
Then wow! I get two compilation errors! It is really strange because
I'm using "stable" version.
The errors are attached (blink_tests_build_log_68.0.3440.84.txt). These
issues do not look like environment issues.
4. I check-out next stable 68.0.3440.106 and I have same errors.
5. I find and cherry-pick upstream commits which fixes these errors (
https://chromium-review.googlesource.com/1072222 and
https://chromium-review.googlesource.com/1083491).
6. I build layout tests and run them. And I got 31 errors. Results are
attached too (blink_tests_run_log_68.0.3440.84.txt)
One issue is related to environment. I already submitted bug against it
(https://bugs.chromium.org/p/chromium/issues/detail?id=872718). But I'm
not sure about other ones.
7. Finally I check out last stable Linux version (70.0.3538.110). I build
blink_tests target with no compilation errors.
But when I run tests I get ~4500 regression errors!
And I would like to know Chromium stable versions testing process in
order to understand why I have so many issues with Layout tests.
Is it my specific issues?
Or there is no any defined test procedure for Chromium stable versions
and each one can be released with unlimited number of non-documented
errors? (sounds pretty unbelievable))
The ideal case is to look into Google's description of test procedure,
test environment and test results for stable releases. And compare these
staff with my one.
I will be very appreciated for any help.

Post by PhistucK
Can you share your failures?
If you build the master branch, do you get those errors as well?
I ask because it can be an environment issue.
☆*PhistucK*

Post by sergey y
At first thank you for the answers.
But the picture still unclear. Let me explain what I want to know in
details.
I know that a lot of tests is running for each master-branch commit and
the detail results are available at https://ci.chromium.org/p/chromium
But according my observation some layout tests (I mean blink specific
layout test which are stored in ./third_party/blink/LayoutTests/ former
./third_party/WebKit/LayoutTests/) are fails on Chromium releases which are
announced as stable. And there no issues submitted against this fails.
Moreover I observe that sometime number of failed tests is increased from
release to release.
I get this observation when I build Chromium by myself and run Layout
tests by myself.
So my questions are more specific and related to stable releases test
procedure.
1. Is any special test procedure using for stable releases? Which tests
are run for these releases? Can I look at test results for these releases
somewhere?
2. What happens if some tests fails during stable release testing and
this fail is a new one (i.e. there is no appropriate issue in
https://bugs.chromium.org/p/chromium/issues/list)?
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a73bb59f-7569-49f2-ae60-40ca99275614%40chromium.org
.

--

--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a6a0d443-8a62-4b4a-908a-ac47bd4d5f40%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/a6a0d443-8a62-4b4a-908a-ac47bd4d5f40%40chromium.org?utm_medium=email&utm_source=footer>
.

--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAFJcur9ZkYTJ8hJJrAu8pkzSpCpWZASBtRHRyO3_Dtq4%2BWQLFw%40mail.gmail.com
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAFJcur9ZkYTJ8hJJrAu8pkzSpCpWZASBtRHRyO3_Dtq4%2BWQLFw%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

[Adam Rice]

Post by sergey y
I will very appreciated for clarification because I really worry about is
Google really releases stable versions with known but not fixed bugs.

It may seem counter-intuitive, but stable releases of Chrome have more
known bugs than dev channel releases.

But if you actually use a dev channel release day-to-day, you'll notice
more bugs than you would using the stable channel.

The reason for this is that the stabilisation process involves fixing the
highest-impact bugs. Every bug fix carries a risk of introducing worse
problems, and so only the most critical fixes are merged to the stable
branch.

I have experienced test failures when building stable releases too, so I'd
be interested to know the answers to your other questions.

Post by sergey y
Hello,
How stable Chromium releases are tested before tagging with "stable tag"?
I'm asking because I downloaded Chromium with some tag (68.0.3440.91)
which was specified as stable in https://omahaproxy.appspot.com/. Then I
run layout tests (as described in
https://chromium.googlesource.com/chromium/src/+/1ded89f3aa84b59f29c34ccdf9dc2bf074fe6bae/docs/layout_tests_linux.md
<https://email-uk.huawei.com/owa/redir.aspx?C=vuP7Ai72kcqoIYkj5yzpT23K8rCv1pjzTkdz0bxZKWYBrMOKJlXWCA..&URL=https%3a%2f%2fchromium.googlesource.com%2fchromium%2fsrc%2f%2b%2f1ded89f3aa84b59f29c34ccdf9dc2bf074fe6bae%2fdocs%2flayout_tests_linux.md>).
And I got number of errors (32).
- Is there any special procedure for stable releases testing?
- Which tests are run for stable releases for each platform (i.e. which
layout tests, which unit tests, etc.)?
- How situation with failed tests is managed?
- Is it possible to find any stable versions test results somewhere (in
order to compare with my own results)?
I will very appreciated for clarification because I really worry about is
Google really releases stable versions with known but not fixed bugs.
Thank you,
Sergey.
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/61c6758e-d13a-4fb3-8fb7-2de7bbd89217%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/61c6758e-d13a-4fb3-8fb7-2de7bbd89217%40chromium.org?utm_medium=email&utm_source=footer>
.