[chromium-dev] Git CL Upload Frequently Failing with 404

[Andy Perelson]

tl;dr: git cl upload has often been failing with a 404, resulting in an
issue link that cannot be followed and requiring re-upload, potentially
hitting the same issue. Tracked as bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> (Sorry,
Googlers only). I will send a follow up email daily to chromium-dev@ to
update you on our progress until we resolve the issue.

Details:
The root cause of this bug is that git cl upload makes several requests to
Gerrit, the latter of which fail when they hit stale replicas and cannot
find expected data from earlier requests.

We are working as quickly as possible to mitigate and fix this issue.

We previously added better routing to help Gerrit find better replicas, and
as of last night are explicitly making retries to different git mirrors in
an attempt to find a non-stale one. This will hopefully ensure success
after a retry or two.

We are working now on changes to do more operations in the initial git
push, reducing the number of subsequent requests that can hit these
replication problems. If this work isn't sufficient we'll continue working
with the Gerrit team to find a solution.

Longer term we want to work with the Gerrit team to make changes to the
Gerrit API that will allow us to further simplify the git cl implementation
to use even fewer requests with the goal of faster cl uploading with fewer
chances for issues such as this one.

If you have any questions feel free to reach out to me at ***@chromium.org
or comment on bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> or it's
dependent issues.

Thank you,

Andy Perelson
On behalf of Chrome Operations

[Andy Perelson]

If you experience a git cl upload failure after updating depot tools please
let us know on bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>. (depot
tools should auto update, but run `depot_tools/update_depot_tools` if you
want to be sure). We believe the change made to use explicit mirrors should
drastically reduce how often git cl upload completely fails after retries,
but we don't have great analytics into how much people are hitting this
issue.

We're continuing work with the Gerrit team to fully resolve this issue.
They have said they should have Gerrit API changes that we need in
production by the end of next week. We'll be modifying git cl upload to
reduce the number of requests we sent to Gerrit that could hit the 404
problems, with finishing that work blocked on Gerrit.

I will send a further update next week when I get an update from the Gerrit
team, or if anything significantly changes. But rest assured we're working
hard on resolving this issue.

Thanks,

Andy

Post by Andy Perelson
tl;dr: git cl upload has often been failing with a 404, resulting in an
issue link that cannot be followed and requiring re-upload, potentially
hitting the same issue. Tracked as bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> (Sorry,
update you on our progress until we resolve the issue.
The root cause of this bug is that git cl upload makes several requests to
Gerrit, the latter of which fail when they hit stale replicas and cannot
find expected data from earlier requests.
We are working as quickly as possible to mitigate and fix this issue.
We previously added better routing to help Gerrit find better replicas,
and as of last night are explicitly making retries to different git mirrors
in an attempt to find a non-stale one. This will hopefully ensure success
after a retry or two.
We are working now on changes to do more operations in the initial git
push, reducing the number of subsequent requests that can hit these
replication problems. If this work isn't sufficient we'll continue working
with the Gerrit team to find a solution.
Longer term we want to work with the Gerrit team to make changes to the
Gerrit API that will allow us to further simplify the git cl implementation
to use even fewer requests with the goal of faster cl uploading with fewer
chances for issues such as this one.
or comment on bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> or it's
dependent issues.
Thank you,
Andy Perelson
On behalf of Chrome Operations

[Andy Perelson]

Chromium developers are continuing to experience git cl upload failures
with 404s (881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>) (and with
lower frequency other errors <https://crbug.com/892488>), though at a lower
rate than before our short term mitigation efforts.

We are continuing to work hard on a long term fix and still prioritizing
this as a p1 bug. The changes to git cl upload that will fix this are
blocked on a lot of work both in Gerrit and Chrome Infrastructure.
Currently in progress:

- Gerrit API changes <http://b/116622189> (sorry, Googler's only), have
proved more complex than they expected. Getting them to production is now
targeted to next week.
- Chrome Infrastructure changes towards fixing blocking bugs: 633572
<https://crbug.com/633572>, 892433 <https://crbug.com/892433>, 877717
<https://crbug.com/877717>

I'll send another update next week.

Andy

Post by Andy Perelson
If you experience a git cl upload failure after updating depot tools
please let us know on bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>. (depot
tools should auto update, but run `depot_tools/update_depot_tools` if you
want to be sure). We believe the change made to use explicit mirrors should
drastically reduce how often git cl upload completely fails after retries,
but we don't have great analytics into how much people are hitting this
issue.
We're continuing work with the Gerrit team to fully resolve this issue.
They have said they should have Gerrit API changes that we need in
production by the end of next week. We'll be modifying git cl upload to
reduce the number of requests we sent to Gerrit that could hit the 404
problems, with finishing that work blocked on Gerrit.
I will send a further update next week when I get an update from the
Gerrit team, or if anything significantly changes. But rest assured we're
working hard on resolving this issue.
Thanks,
Andy

Post by Andy Perelson
tl;dr: git cl upload has often been failing with a 404, resulting in an
issue link that cannot be followed and requiring re-upload, potentially
hitting the same issue. Tracked as bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> (Sorry,
update you on our progress until we resolve the issue.
The root cause of this bug is that git cl upload makes several requests
to Gerrit, the latter of which fail when they hit stale replicas and cannot
find expected data from earlier requests.
We are working as quickly as possible to mitigate and fix this issue.
We previously added better routing to help Gerrit find better replicas,
and as of last night are explicitly making retries to different git mirrors
in an attempt to find a non-stale one. This will hopefully ensure success
after a retry or two.
We are working now on changes to do more operations in the initial git
push, reducing the number of subsequent requests that can hit these
replication problems. If this work isn't sufficient we'll continue working
with the Gerrit team to find a solution.
Longer term we want to work with the Gerrit team to make changes to the
Gerrit API that will allow us to further simplify the git cl implementation
to use even fewer requests with the goal of faster cl uploading with fewer
chances for issues such as this one.
If you have any questions feel free to reach out to me at
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> or it's
dependent issues.
Thank you,
Andy Perelson
On behalf of Chrome Operations

[Andy Perelson]

Work to fix git cl upload continues. When this bug occurs you will now see
instructions to upload a log file to bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> that will
help Gerrit team understand the root cause of this issue. Thank you to
folks who have done that already! Current status towards a fix:

- Blocking Gerrit API changes <http://b/116622189> (sorry, Googler's
only), are all in CL and under review or merged so we have higher
confidence they can get to production next week.
- We've fixed blocking bug 892433 <https://crbug.com/892433>, expect a
separate PSA soon announcing the new support for specifying trybots in CQ
configuration.
- Next week we expect to be able to start making the significant updates
to git cl upload that will reduce the number of requests we make and fix
this issue! Stay tuned for separate PSAs based on the changes we need to
make to PRESUBMITs

Thanks,

Andy

Post by Andy Perelson
Chromium developers are continuing to experience git cl upload failures
with 404s (881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>) (and with
lower frequency other errors <https://crbug.com/892488>), though at a
lower rate than before our short term mitigation efforts.
We are continuing to work hard on a long term fix and still prioritizing
this as a p1 bug. The changes to git cl upload that will fix this are
blocked on a lot of work both in Gerrit and Chrome Infrastructure.
- Gerrit API changes <http://b/116622189> (sorry, Googler's only),
have proved more complex than they expected. Getting them to production is
now targeted to next week.
- Chrome Infrastructure changes towards fixing blocking bugs: 633572
<https://crbug.com/633572>, 892433 <https://crbug.com/892433>, 877717
<https://crbug.com/877717>
I'll send another update next week.
Andy

Post by Andy Perelson
If you experience a git cl upload failure after updating depot tools
please let us know on bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>. (depot
tools should auto update, but run `depot_tools/update_depot_tools` if
you want to be sure). We believe the change made to use explicit mirrors
should drastically reduce how often git cl upload completely fails after
retries, but we don't have great analytics into how much people are hitting
this issue.
We're continuing work with the Gerrit team to fully resolve this issue.
They have said they should have Gerrit API changes that we need in
production by the end of next week. We'll be modifying git cl upload to
reduce the number of requests we sent to Gerrit that could hit the 404
problems, with finishing that work blocked on Gerrit.
I will send a further update next week when I get an update from the
Gerrit team, or if anything significantly changes. But rest assured we're
working hard on resolving this issue.
Thanks,
Andy

Post by Andy Perelson
tl;dr: git cl upload has often been failing with a 404, resulting in an
issue link that cannot be followed and requiring re-upload, potentially
hitting the same issue. Tracked as bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> (Sorry,
update you on our progress until we resolve the issue.
The root cause of this bug is that git cl upload makes several requests
to Gerrit, the latter of which fail when they hit stale replicas and cannot
find expected data from earlier requests.
We are working as quickly as possible to mitigate and fix this issue.
We previously added better routing to help Gerrit find better replicas,
and as of last night are explicitly making retries to different git mirrors
in an attempt to find a non-stale one. This will hopefully ensure success
after a retry or two.
We are working now on changes to do more operations in the initial git
push, reducing the number of subsequent requests that can hit these
replication problems. If this work isn't sufficient we'll continue working
with the Gerrit team to find a solution.
Longer term we want to work with the Gerrit team to make changes to the
Gerrit API that will allow us to further simplify the git cl implementation
to use even fewer requests with the goal of faster cl uploading with fewer
chances for issues such as this one.
If you have any questions feel free to reach out to me at
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> or it's
dependent issues.
Thank you,
Andy Perelson
On behalf of Chrome Operations

[Andy Perelson]

git cl uploads are still failing with 404 errors. Thank you everyone for
your bug reports. They are helping the Gerrit team root cause this issue.
We're continually adding to the set of information we're collecting to
debug as they learn more and get further along.

We've started merging updates to git cl that are reducing the number of
requests, which may reduce how often this bug happens, but we're definitely
not done yet. Specifically:

- Blocking Gerrit API changes <http://b/116622189> (sorry, Googler's
only), are in canary. However, I was unable to verify they work. Will be
following up with the Gerrit team next week.
- Blocking bug 633572 <https://crbug.com/633572> has been fixed for the
chromium/src repo where we expect it to be problematic. Edward has
migrated <https://crrev.com/c/1265735/10/infra/config/branch/cq.cfg> all
the conditional trybots for that repo to cq.cfg and cleaned up the
PRESUBMIT files <https://crrev.com/c/1266055>.
- Progress on blocking bug 877717 <https://crbug.com/877717>
- Andrii has made changes to set reviewers and cc in the initial git
push for all reviewers and cc's that the push API will accept.
- Remaining reviewers and ccs Gerrit API changes.
- Also need to set labels in git push

I expect limited progress next week as all of Chrome Operations is at an
extended off site.

Thanks,

Andy

Post by Andy Perelson
Work to fix git cl upload continues. When this bug occurs you will now see
instructions to upload a log file to bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> that will
help Gerrit team understand the root cause of this issue. Thank you to
- Blocking Gerrit API changes <http://b/116622189> (sorry, Googler's
only), are all in CL and under review or merged so we have higher
confidence they can get to production next week.
- We've fixed blocking bug 892433 <https://crbug.com/892433>, expect a
separate PSA soon announcing the new support for specifying trybots in CQ
configuration.
- Next week we expect to be able to start making the significant
updates to git cl upload that will reduce the number of requests we make
and fix this issue! Stay tuned for separate PSAs based on the changes we
need to make to PRESUBMITs
Thanks,
Andy

Post by Andy Perelson
Chromium developers are continuing to experience git cl upload failures
with 404s (881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>) (and
with lower frequency other errors <https://crbug.com/892488>), though at
a lower rate than before our short term mitigation efforts.
We are continuing to work hard on a long term fix and still prioritizing
this as a p1 bug. The changes to git cl upload that will fix this are
blocked on a lot of work both in Gerrit and Chrome Infrastructure.
- Gerrit API changes <http://b/116622189> (sorry, Googler's only),
have proved more complex than they expected. Getting them to production is
now targeted to next week.
- Chrome Infrastructure changes towards fixing blocking bugs: 633572
<https://crbug.com/633572>, 892433 <https://crbug.com/892433>, 877717
<https://crbug.com/877717>
I'll send another update next week.
Andy

Post by Andy Perelson
If you experience a git cl upload failure after updating depot tools
please let us know on bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>. (depot
tools should auto update, but run `depot_tools/update_depot_tools` if you
want to be sure). We believe the change made to use explicit mirrors should
drastically reduce how often git cl upload completely fails after retries,
but we don't have great analytics into how much people are hitting this
issue.
We're continuing work with the Gerrit team to fully resolve this issue.
They have said they should have Gerrit API changes that we need in
production by the end of next week. We'll be modifying git cl upload to
reduce the number of requests we sent to Gerrit that could hit the 404
problems, with finishing that work blocked on Gerrit.
I will send a further update next week when I get an update from the
Gerrit team, or if anything significantly changes. But rest assured we're
working hard on resolving this issue.
Thanks,
Andy

Post by Andy Perelson
tl;dr: git cl upload has often been failing with a 404, resulting in an
issue link that cannot be followed and requiring re-upload, potentially
hitting the same issue. Tracked as bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> (Sorry,
to update you on our progress until we resolve the issue.
The root cause of this bug is that git cl upload makes several requests
to Gerrit, the latter of which fail when they hit stale replicas and cannot
find expected data from earlier requests.
We are working as quickly as possible to mitigate and fix this issue.
We previously added better routing to help Gerrit find better replicas,
and as of last night are explicitly making retries to different git mirrors
in an attempt to find a non-stale one. This will hopefully ensure success
after a retry or two.
We are working now on changes to do more operations in the initial git
push, reducing the number of subsequent requests that can hit these
replication problems. If this work isn't sufficient we'll continue working
with the Gerrit team to find a solution.
Longer term we want to work with the Gerrit team to make changes to the
Gerrit API that will allow us to further simplify the git cl implementation
to use even fewer requests with the goal of faster cl uploading with fewer
chances for issues such as this one.
If you have any questions feel free to reach out to me at
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> or it's
dependent issues.
Thank you,
Andy Perelson
On behalf of Chrome Operations

[Andy Perelson]

We've landed several CLs in depot_tools to fix issues with git cl upload
and believe there should be a substantial drop in the number of 404s. If
you update depot_tools you should get these latest changes.

- Blocking Gerrit API changes <http://b/116622189> (sorry, Googler's
only) are in production and have been validated to work for all our public
repos. There is a remaining issue that is preventing us from reducing the
number of requests for private, internal repos. However, those repos are
all smaller and much less likely to have this problem when uploading.
- Andrii's made significant progress on blocking bug 877717
<https://crbug.com/877717>. There are only a few edge cases where we
send requests that we think could 404 most notably:
- When setting labels and submitting a change TBR
- When uploading to internal repos that are not hosted on
chromium.googlesource.com

We will continue to work on the long tail of edge cases, but we're hoping
to validate that this issue is fixed for most people. We'll be keeping an
eye out for continued reports of the issue on bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> (Sorry,
Googlers only).

Andy

Post by Andy Perelson
git cl uploads are still failing with 404 errors. Thank you everyone for
your bug reports. They are helping the Gerrit team root cause this issue.
We're continually adding to the set of information we're collecting to
debug as they learn more and get further along.
We've started merging updates to git cl that are reducing the number of
requests, which may reduce how often this bug happens, but we're definitely
- Blocking Gerrit API changes <http://b/116622189> (sorry, Googler's
only), are in canary. However, I was unable to verify they work. Will be
following up with the Gerrit team next week.
- Blocking bug 633572 <https://crbug.com/633572> has been fixed for
the chromium/src repo where we expect it to be problematic. Edward has
migrated <https://crrev.com/c/1265735/10/infra/config/branch/cq.cfg>
all the conditional trybots for that repo to cq.cfg and cleaned up the
PRESUBMIT files <https://crrev.com/c/1266055>.
- Progress on blocking bug 877717 <https://crbug.com/877717>
- Andrii has made changes to set reviewers and cc in the initial
git push for all reviewers and cc's that the push API will accept.
- Remaining reviewers and ccs Gerrit API changes.
- Also need to set labels in git push
I expect limited progress next week as all of Chrome Operations is at an
extended off site.
Thanks,
Andy

Post by Andy Perelson
Work to fix git cl upload continues. When this bug occurs you will now
see instructions to upload a log file to bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> that will
help Gerrit team understand the root cause of this issue. Thank you to
- Blocking Gerrit API changes <http://b/116622189> (sorry, Googler's
only), are all in CL and under review or merged so we have higher
confidence they can get to production next week.
- We've fixed blocking bug 892433 <https://crbug.com/892433>, expect
a separate PSA soon announcing the new support for specifying trybots in CQ
configuration.
- Next week we expect to be able to start making the significant
updates to git cl upload that will reduce the number of requests we make
and fix this issue! Stay tuned for separate PSAs based on the changes we
need to make to PRESUBMITs
Thanks,
Andy

Post by Andy Perelson
Chromium developers are continuing to experience git cl upload failures
with 404s (881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>) (and
with lower frequency other errors <https://crbug.com/892488>), though
at a lower rate than before our short term mitigation efforts.
We are continuing to work hard on a long term fix and still prioritizing
this as a p1 bug. The changes to git cl upload that will fix this are
blocked on a lot of work both in Gerrit and Chrome Infrastructure.
- Gerrit API changes <http://b/116622189> (sorry, Googler's only),
have proved more complex than they expected. Getting them to production is
now targeted to next week.
- Chrome Infrastructure changes towards fixing blocking bugs: 633572
<https://crbug.com/633572>, 892433 <https://crbug.com/892433>, 877717
<https://crbug.com/877717>
I'll send another update next week.
Andy

Post by Andy Perelson
If you experience a git cl upload failure after updating depot tools
please let us know on bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>. (depot
tools should auto update, but run `depot_tools/update_depot_tools` if you
want to be sure). We believe the change made to use explicit mirrors should
drastically reduce how often git cl upload completely fails after retries,
but we don't have great analytics into how much people are hitting this
issue.
We're continuing work with the Gerrit team to fully resolve this issue.
They have said they should have Gerrit API changes that we need in
production by the end of next week. We'll be modifying git cl upload to
reduce the number of requests we sent to Gerrit that could hit the 404
problems, with finishing that work blocked on Gerrit.
I will send a further update next week when I get an update from the
Gerrit team, or if anything significantly changes. But rest assured we're
working hard on resolving this issue.
Thanks,
Andy

Post by Andy Perelson
tl;dr: git cl upload has often been failing with a 404, resulting in
an issue link that cannot be followed and requiring re-upload, potentially
hitting the same issue. Tracked as bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>
to update you on our progress until we resolve the issue.
The root cause of this bug is that git cl upload makes several
requests to Gerrit, the latter of which fail when they hit stale replicas
and cannot find expected data from earlier requests.
We are working as quickly as possible to mitigate and fix this issue.
We previously added better routing to help Gerrit find better
replicas, and as of last night are explicitly making retries to different
git mirrors in an attempt to find a non-stale one. This will hopefully
ensure success after a retry or two.
We are working now on changes to do more operations in the initial git
push, reducing the number of subsequent requests that can hit these
replication problems. If this work isn't sufficient we'll continue working
with the Gerrit team to find a solution.
Longer term we want to work with the Gerrit team to make changes to
the Gerrit API that will allow us to further simplify the git cl
implementation to use even fewer requests with the goal of faster cl
uploading with fewer chances for issues such as this one.
If you have any questions feel free to reach out to me at
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> or
it's dependent issues.
Thank you,
Andy Perelson
On behalf of Chrome Operations

[Andy Perelson]

It looks like the fix we released last week has been working well. The
number of reports of git cl upload failures has dropped dramatically and
our (new and improved) data collection also shows only a very few 404s in
the expected long tail of cases where this bug can still happen such as
private repositories.

I am going to downgrade bug 881860 <https://crbug.com/881860> to a p2 and
stop sending PSA updates on the issue. The bug will continue to be updated
as we make progress on the remaining edge cases that can be problematic.

Please reach out if you have any questions or concerns.

Thank you,

Andy

Post by Andy Perelson
We've landed several CLs in depot_tools to fix issues with git cl upload
and believe there should be a substantial drop in the number of 404s. If
you update depot_tools you should get these latest changes.
- Blocking Gerrit API changes <http://b/116622189> (sorry, Googler's
only) are in production and have been validated to work for all our public
repos. There is a remaining issue that is preventing us from reducing the
number of requests for private, internal repos. However, those repos are
all smaller and much less likely to have this problem when uploading.
- Andrii's made significant progress on blocking bug 877717
<https://crbug.com/877717>. There are only a few edge cases where we
- When setting labels and submitting a change TBR
- When uploading to internal repos that are not hosted on
chromium.googlesource.com
We will continue to work on the long tail of edge cases, but we're hoping
to validate that this issue is fixed for most people. We'll be keeping an
eye out for continued reports of the issue on bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> (Sorry,
Googlers only).
Andy

Post by Andy Perelson
git cl uploads are still failing with 404 errors. Thank you everyone for
your bug reports. They are helping the Gerrit team root cause this issue.
We're continually adding to the set of information we're collecting to
debug as they learn more and get further along.
We've started merging updates to git cl that are reducing the number of
requests, which may reduce how often this bug happens, but we're definitely
- Blocking Gerrit API changes <http://b/116622189> (sorry, Googler's
only), are in canary. However, I was unable to verify they work. Will be
following up with the Gerrit team next week.
- Blocking bug 633572 <https://crbug.com/633572> has been fixed for
the chromium/src repo where we expect it to be problematic. Edward has
migrated <https://crrev.com/c/1265735/10/infra/config/branch/cq.cfg>
all the conditional trybots for that repo to cq.cfg and cleaned up
the PRESUBMIT files <https://crrev.com/c/1266055>.
- Progress on blocking bug 877717 <https://crbug.com/877717>
- Andrii has made changes to set reviewers and cc in the initial
git push for all reviewers and cc's that the push API will accept.
- Remaining reviewers and ccs Gerrit API changes.
- Also need to set labels in git push
I expect limited progress next week as all of Chrome Operations is at an
extended off site.
Thanks,
Andy

Post by Andy Perelson
Work to fix git cl upload continues. When this bug occurs you will now
see instructions to upload a log file to bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> that
will help Gerrit team understand the root cause of this issue. Thank you to
- Blocking Gerrit API changes <http://b/116622189> (sorry, Googler's
only), are all in CL and under review or merged so we have higher
confidence they can get to production next week.
- We've fixed blocking bug 892433 <https://crbug.com/892433>, expect
a separate PSA soon announcing the new support for specifying trybots in CQ
configuration.
- Next week we expect to be able to start making the significant
updates to git cl upload that will reduce the number of requests we make
and fix this issue! Stay tuned for separate PSAs based on the changes we
need to make to PRESUBMITs
Thanks,
Andy

Post by Andy Perelson
Chromium developers are continuing to experience git cl upload failures
with 404s (881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>) (and
with lower frequency other errors <https://crbug.com/892488>), though
at a lower rate than before our short term mitigation efforts.
We are continuing to work hard on a long term fix and still
prioritizing this as a p1 bug. The changes to git cl upload that will fix
this are blocked on a lot of work both in Gerrit and Chrome Infrastructure.
- Gerrit API changes <http://b/116622189> (sorry, Googler's only),
have proved more complex than they expected. Getting them to production is
now targeted to next week.
- Chrome Infrastructure changes towards fixing blocking bugs: 633572
<https://crbug.com/633572>, 892433 <https://crbug.com/892433>,
877717 <https://crbug.com/877717>
I'll send another update next week.
Andy

Post by Andy Perelson
If you experience a git cl upload failure after updating depot tools
please let us know on bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>. (depot
tools should auto update, but run `depot_tools/update_depot_tools` if you
want to be sure). We believe the change made to use explicit mirrors should
drastically reduce how often git cl upload completely fails after retries,
but we don't have great analytics into how much people are hitting this
issue.
We're continuing work with the Gerrit team to fully resolve this
issue. They have said they should have Gerrit API changes that we need in
production by the end of next week. We'll be modifying git cl upload to
reduce the number of requests we sent to Gerrit that could hit the 404
problems, with finishing that work blocked on Gerrit.
I will send a further update next week when I get an update from the
Gerrit team, or if anything significantly changes. But rest assured we're
working hard on resolving this issue.
Thanks,
Andy

Post by Andy Perelson
tl;dr: git cl upload has often been failing with a 404, resulting in
an issue link that cannot be followed and requiring re-upload, potentially
hitting the same issue. Tracked as bug 881860
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860>
to update you on our progress until we resolve the issue.
The root cause of this bug is that git cl upload makes several
requests to Gerrit, the latter of which fail when they hit stale replicas
and cannot find expected data from earlier requests.
We are working as quickly as possible to mitigate and fix this issue.
We previously added better routing to help Gerrit find better
replicas, and as of last night are explicitly making retries to different
git mirrors in an attempt to find a non-stale one. This will hopefully
ensure success after a retry or two.
We are working now on changes to do more operations in the initial
git push, reducing the number of subsequent requests that can hit these
replication problems. If this work isn't sufficient we'll continue working
with the Gerrit team to find a solution.
Longer term we want to work with the Gerrit team to make changes to
the Gerrit API that will allow us to further simplify the git cl
implementation to use even fewer requests with the goal of faster cl
uploading with fewer chances for issues such as this one.
If you have any questions feel free to reach out to me at
<https://bugs.chromium.org/p/chromium/issues/detail?id=881860> or
it's dependent issues.
Thank you,
Andy Perelson
On behalf of Chrome Operations