[chromium-dev] Fwd: Failed to symbolize chromium heap dump

[Alexander Yashkin]

I will try my luck in chromium-dev, since ***@chromium.org is silent.


Hi all.
I am trying to become acquainted with chromium memory tracing infrastructure.
I am trying to record and symbolize heap dump as described in https://chromium.googlesource.com/chromium/src/+/lkcr/docs/memory-infra/heap_profiler.md

I have successfully recorded memory trace with heap dump and have problem with symbolizing it.
According to instruction I am performing "./third_party/catapult/tracing/bin/symbolize_trace <path_to_trace>"

I get following output
"c:\Workspace\chrome\src>python "third_party/catapult/tracing/bin/symbolize_trace" C:\Workspace\Bugs\91887\trace_with_heap_dump.json.gz
Reading trace file...
Trace loaded for Windows NT/70.0.3535.3
Traceback (most recent call last):
  File "third_party/catapult/tracing/bin/symbolize_trace", line 17, in <module>
    sys.exit(symbolize_trace.main(sys.argv[1:]))
  File "c:\Workspace\chrome\src\third_party\catapult\tracing\tracing\extras\symbolizer\symbolize_trace.py", line 1678, in main
    options.cloud_storage_bucket)
  File "c:\Workspace\chrome\src\third_party\catapult\tracing\tracing\extras\symbolizer\symbolize_trace.py", line 1549, in FetchAndExtractSymbolsWin
    if not cloud_storage.Exists(cloud_storage_bucket, gcs_folder):
  File "c:\Workspace\chrome\src\third_party\catapult\common\py_utils\py_utils\cloud_storage.py", line 203, in Exists
    _RunCommand(['ls', 'gs://%s/%s' % (bucket, remote_path)])
  File "c:\Workspace\chrome\src\third_party\catapult\common\py_utils\py_utils\cloud_storage.py", line 159, in _RunCommand
    raise GetErrorObjectForCloudStorageStderr(stderr)
py_utils.cloud_storage.PermissionError: Attempted to access a file from Cloud Storage but you don't have permission. To configure your credentials:
  1. Run "c:\Workspace\chrome\src\third_party\catapult\third_party\gsutil\gsutil config" and follow its instructions.
  2. If you have a @google.com account, use that account.
  3. For the project-id, just enter 0."

My problem is that after performing all actions in steps 1,2,3, authorizing, entering code for gsutil and project-id=0 I still get the same PermissionError.
I have tried on my win and mac and get the same PermissionError.

Did anyone successfully tried to symbolize heap dumps outside of chromium team?
What I can do to investigate problem?

--
WBR, Alexander Yashkin.

[Erik Chen]

If you're using a local build -- e.g. Chromium, you'll want to run:
"""
./third_party/catapult/tracing/bin/symbolize_trace --is-local-build
<path_to_trace>
"""

Google Chrome breakpad symbols are currently not publicly available. For
Chrome on Windows, I believe symbols are available via a public symbol
server. The symbolize_trace script is not hooked up to that though...

Post by Alexander Yashkin
Hi all.
I am trying to become acquainted with chromium memory tracing
infrastructure.
I am trying to record and symbolize heap dump as described in
https://chromium.googlesource.com/chromium/src/+/lkcr/docs/
memory-infra/heap_profiler.md
I have successfully recorded memory trace with heap dump and have problem
with symbolizing it.
According to instruction I am performing "./third_party/catapult/tracing/bin/symbolize_trace
<path_to_trace>"
I get following output
"c:\Workspace\chrome\src>python "third_party/catapult/tracing/bin/symbolize_trace"
C:\Workspace\Bugs\91887\trace_with_heap_dump.json.gz
Reading trace file...
Trace loaded for Windows NT/70.0.3535.3
File "third_party/catapult/tracing/bin/symbolize_trace", line 17, in
<module>
sys.exit(symbolize_trace.main(sys.argv[1:]))
File "c:\Workspace\chrome\src\third_party\catapult\tracing\
tracing\extras\symbolizer\symbolize_trace.py", line 1678, in main
options.cloud_storage_bucket)
File "c:\Workspace\chrome\src\third_party\catapult\tracing\
tracing\extras\symbolizer\symbolize_trace.py", line 1549, in
FetchAndExtractSymbolsWin
File "c:\Workspace\chrome\src\third_party\catapult\common\
py_utils\py_utils\cloud_storage.py", line 203, in Exists
_RunCommand(['ls', 'gs://%s/%s' % (bucket, remote_path)])
File "c:\Workspace\chrome\src\third_party\catapult\common\
py_utils\py_utils\cloud_storage.py", line 159, in _RunCommand
raise GetErrorObjectForCloudStorageStderr(stderr)
py_utils.cloud_storage.PermissionError: Attempted to access a file from
1. Run "c:\Workspace\chrome\src\third_party\catapult\third_party\gsutil\gsutil
config" and follow its instructions.
3. For the project-id, just enter 0."
My problem is that after performing all actions in steps 1,2,3,
authorizing, entering code for gsutil and project-id=0 I still get the same
PermissionError.
I have tried on my win and mac and get the same PermissionError.
Did anyone successfully tried to symbolize heap dumps outside of chromium team?
What I can do to investigate problem?
--
WBR, Alexander Yashkin.
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/
chromium.org/d/msgid/chromium-dev/368691535641876%40sas1-
bf4ab558af9f.qloud-c.yandex.net.

[Idrees Shah]

Hi, I have followed the help page mentioned above and have built chrome 69 on linux with debug symbols. I am passing --is-local-build to symbolize_trace tool. But it returns with no symbols added to the profiled data.

On digging further in the profiled data and symbolize_trace code, I found out that "heap_v2" symbol is missing from the profiled json data. As a result, symbolize_trace skip all the stack frames. I did enabled all the memlog options above.

What I am doing wrong here?

[Erik Chen]

Idrees: It sounds like you're not successfully taking a heap dump. Try
going to chrome://memory-internals, make sure that profiling is turned on
for the process you care about, and then click "save dump"

Post by Idrees Shah
Hi, I have followed the help page mentioned above and have built chrome 69
on linux with debug symbols. I am passing --is-local-build to
symbolize_trace tool. But it returns with no symbols added to the profiled
data.
On digging further in the profiled data and symbolize_trace code, I found
out that "heap_v2" symbol is missing from the profiled json data. As a
result, symbolize_trace skip all the stack frames. I did enabled all the
memlog options above.
What I am doing wrong here?
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/8785a63f-2658-47f1-878f-fc21986204e8%40chromium.org
.

[Idrees Shah]

Yes, using memory-internals does give the correct json format. But it only dumps only one purple M, while I need multiple points of M, so to compare them. Chrome://tracing does not give that json, which I can symbolize.

[Erik Chen]

If you want multiple heap dumps, you can take a chrome://tracing trace.
Make sure to select "memory-infra" in the column on the right.

Post by Idrees Shah
Yes, using memory-internals does give the correct json format. But it only
dumps only one purple M, while I need multiple points of M, so to compare
them. Chrome://tracing does not give that json, which I can symbolize.
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/982a513f-f33c-4232-89b7-6923e400995a%40chromium.org
.

[Idrees Shah]

That's what I do but symbolize_trace fails to resolve the symbols for me on that json file. Browser can display the result fine. I was just hoping to get symbols loaded as well.

[Erik Chen]

Is this the same problem you were referring to earlier? "I found out that
"heap_v2" symbol is missing from the profiled json data.". If there are
multiple purple Ms, then I would also expect heap_v2 to be present in the
JSON file.

Post by Idrees Shah
That's what I do but symbolize_trace fails to resolve the symbols for me
on that json file. Browser can display the result fine. I was just hoping
to get symbols loaded as well.
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/51e8ade3-8493-4ec4-b42c-3aacabac3837%40chromium.org
.

[Idrees Shah]

Yes

[Erik Chen]

That is unexpected.

Given that you can successfully take heap dumps using
chrome://memory-internals, can you just do that repeatedly instead of
relying on chrome://tracing?

Post by Alexander Yashkin
Yes
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/2cdd6907-8556-40b4-803a-c72857cc867b%40chromium.org
.

[Idrees Shah]

I got the tracing fine. It was my mistake. I selected 'out of process heap profiling start mode' to None by default... After selecting All processes I got the correct traces. Thanks for you help