[chromium-dev] Memory pressure in an embedded linux environment

[i***@noisolation.com]

Hello!

I'm having problems using the Chromium in an embedded device, due to what
seems to be too aggressive memory consumption.

Architecture: aarch64 (armv8-a).
OS: Linux (kernel 4.9.39).

*Test situation:*
Chromium Embedded Framework (CEF; cefsimple example program) running with a
cgroup limiting memory to a max of 320MB. If I hadn't had this cgroup limit
the same happens only much later when the whole ram is spent (1GB). A
simple html+images+js webpage that uses js to switch between images in a
loop. No new content over time, just the same over and over again.

*Expected:*
Chromium to consume the size of all binaries as the memory mapped libraries
are used and loaded into ram + os process/thread overhead + webpage content
size ++.

*What actually happens:*
Chromium uses all available memory to within 1MB, not right away but it
slowly grows from half of the given max until there is nothing left, then
hugs that limit so closely that either it gets itself killed by the OOM, or
if the OOM is turned off it fails to create new threads (pthread_create)
due to those resource limits. This depends a bit on the type of content.
With the image switcher described above it grows in steps, which if graphed
look like stairs. However, if the page has a row of images and use CSS
animations to translate them back and forth the memory "leak" / usage grows
slow and steady, completely linearly, until there is nothing left and it
hugs that limit closely until it fails.

*Questions:*

1. Is there a concept of Memory Pressure for Linux? I cannot see it in
the source. I assume this is supposed to hint to Chromiums allocators to
ease off and GC when closing on the limit.
2. Why does Chromium insist on using all the memory? It is clearly more
than it needs since it spends some time getting to the max.
3. How can I make Chromium believe that it has 20-30MB less memory than
it does by patching the source? I'm thinking this may be enough to hide the
crash since it does seem to respect the limit somewhat, just not well
enough.
1. I'm hoping that when it over-steps the allocators will GC to
correct itself. Might be wrong?
2. Keeping it a bit below the limit will also prevent the kernel from
slowing the CPU to punish the process for using all its allowed memory
(high vs. max memory limits in cgroups).
4. Any other ideas for me?



I really appreciate any help or guidance you may provide me with!

[Mario Sanchez Prada]

Hi

I don't have all the answers to your questions and it's even possible that what I'm going to comment is a bit outdated or plainly wrong, so please anyone else with more knowledge on this do correct me if needed.

Post by i***@noisolation.com
[...]
*Questions:*
1. Is there a concept of Memory Pressure for Linux? I cannot see it in the source. I assume this is supposed to hint to Chromiums allocators to ease off and GC when closing on the limit.

As far as I know there's still the old system based on the MemoryPressureMonitor class in place under //base/memory [1], which is still in use in some platforms but unfortunately not in Linux: just on Windows, Mac and ChromeOs.

Also as far as I remember, there have been attempts in the past to refactor this code into a |memory_pressure| component under //components/memory_pressure, including a Linux implementation (see [2]). However, and this is why you can't find anything in the sources, checking the commit history [3] I found out that this has been abandoned like 1.5 years ago in favour of the "memory coordinator work" (see [4]).

So, I think I'd suggest getting familiar with that "memory coordinator" work and figuring out the status of it for Linux, and whether you could work in that area to get your use case supported. Mind you, I'm not familiar with that so please take my word with a grain of salt, you'd better get in touch with the right people there and get advice from them if this is something you consider exploring.

Post by i***@noisolation.com
[...]
4. Any other ideas for me?

Alternative, you could follow another approach and do what Endless have done downstream to support their low-memory ARM devices a long time ago: hack your own version of the old MemoryPressureMonitor for your platform and adapt it to your needs. Endless did it based on ChromiumOS' implementation of MemoryPressureMonitor here, see the main commits here:

https://github.com/endlessm/chromium-browser/commit/6196cc79
https://github.com/endlessm/chromium-browser/commit/b6df3633
https://github.com/endlessm/chromium-browser/commit/146408fd

(you might find other interesting commits in the whole set of downstream patches in [5])

However, this came not without problems (e.g. heuristics not being entirely right, memory still filing up in some configurations...) so, based on my own experience and if it's feasible for you, I think I'd advise you to first check the status "memory coordinator" work first and see if that can fit your needs, before going the "downstream-only" route :-).

Post by i***@noisolation.com
I really appreciate any help or guidance you may provide me with!

As I said a couple of times already, take this mail with a grain of salt, but other than that I hope this is useful in some way.

Thanks,
Mario

[1] https://cs.chromium.org/chromium/src/base/memory/memory_pressure_monitor.h
[2] https://bugs.chromium.org/p/chromium/issues/detail?id=463606
[3] https://chromium.googlesource.com/chromium/src/+log/master/components/memory_pressure/direct_memory_pressure_calculator_linux.cc
[4] https://chromium.googlesource.com/chromium/src/+/4d1357d65728d99aef9de10f4ab8a914cb09b96d
[5] https://github.com/endlessm/chromium-browser/commits/eos3.5

[i***@noisolation.com]

Thanks Mario.

I will look into those references and patches tomorrow. I discovered
earlier today from the source that linux fetches its memory stats from
"/proc/meminfo", and thus probably knows nothing about the cgroup limits
I've given it. Which could explain why it seemingly goes CPU to a 100% when
it reaches the limit and actively hugs the line (I'm assuming a loop trying
to malloc somewhere, or similar).

My plan right now is to reimplement it to get its memory stats info from
the cgroup stats instead, and check how that effects Chromium :).

Post by Mario Sanchez Prada
Hi
I don't have all the answers to your questions and it's even possible that
what I'm going to comment is a bit outdated or plainly wrong, so please
anyone else with more knowledge on this do correct me if needed.
In any case, here it goes anyway in case it can be of any help for your

Post by i***@noisolation.com
[...]
*Questions:*
1. Is there a concept of Memory Pressure for Linux? I cannot see it in

the source. I assume this is supposed to hint to Chromiums allocators to
ease off and GC when closing on the limit.
As far as I know there's still the old system based on the
MemoryPressureMonitor class in place under //base/memory [1], which is
still in use in some platforms but unfortunately not in Linux: just on
Windows, Mac and ChromeOs.
Also as far as I remember, there have been attempts in the past to
refactor this code into a |memory_pressure| component under
//components/memory_pressure, including a Linux implementation (see [2]).
However, and this is why you can't find anything in the sources, checking
the commit history [3] I found out that this has been abandoned like 1.5
years ago in favour of the "memory coordinator work" (see [4]).
So, I think I'd suggest getting familiar with that "memory coordinator"
work and figuring out the status of it for Linux, and whether you could
work in that area to get your use case supported. Mind you, I'm not
familiar with that so please take my word with a grain of salt, you'd
better get in touch with the right people there and get advice from them if
this is something you consider exploring.

Post by i***@noisolation.com
[...]
4. Any other ideas for me?

Alternative, you could follow another approach and do what Endless have
hack your own version of the old MemoryPressureMonitor for your platform
and adapt it to your needs. Endless did it based on ChromiumOS'
https://github.com/endlessm/chromium-browser/commit/6196cc79
https://github.com/endlessm/chromium-browser/commit/b6df3633
https://github.com/endlessm/chromium-browser/commit/146408fd
(you might find other interesting commits in the whole set of downstream patches in [5])
However, this came not without problems (e.g. heuristics not being
entirely right, memory still filing up in some configurations...) so, based
on my own experience and if it's feasible for you, I think I'd advise you
to first check the status "memory coordinator" work first and see if that
can fit your needs, before going the "downstream-only" route :-).

Post by i***@noisolation.com
I really appreciate any help or guidance you may provide me with!

As I said a couple of times already, take this mail with a grain of salt,
but other than that I hope this is useful in some way.
Thanks,
Mario
[1]
https://cs.chromium.org/chromium/src/base/memory/memory_pressure_monitor.h
[2] https://bugs.chromium.org/p/chromium/issues/detail?id=463606
[3]
https://chromium.googlesource.com/chromium/src/+log/master/components/memory_pressure/direct_memory_pressure_calculator_linux.cc
[4]
https://chromium.googlesource.com/chromium/src/+/4d1357d65728d99aef9de10f4ab8a914cb09b96d
[5] https://github.com/endlessm/chromium-browser/commits/eos3.5

[Luke Halliwell]

Btw, we face a similar issue on Chromecast. Currently we run our own
memory pressure monitor and generate pressure signals, without which we
would hit OOM crashes pretty quickly. You can find this code under
chromecast/ dir.

Some possible downsides to look out for:
1) memory pressure signals can cause some extra work and possible jank
(e.g. GC pauses, and sometimes dropping compositor tiles which need to be
regenerated immediately)
2) code can get added anywhere to respond to memory pressure signals, so it
can be an ongoing maintenance issue to look out for unexpected behaviour
changes (e.g. I remember one where idle sockets got closed that caused us
problems, you'll see we have some code hanging around still to deal with
that).

Post by i***@noisolation.com
Thanks Mario.
I will look into those references and patches tomorrow. I discovered
earlier today from the source that linux fetches its memory stats from
"/proc/meminfo", and thus probably knows nothing about the cgroup limits
I've given it. Which could explain why it seemingly goes CPU to a 100% when
it reaches the limit and actively hugs the line (I'm assuming a loop trying
to malloc somewhere, or similar).
My plan right now is to reimplement it to get its memory stats info from
the cgroup stats instead, and check how that effects Chromium :).

Post by Mario Sanchez Prada
Hi
I don't have all the answers to your questions and it's even possible
that what I'm going to comment is a bit outdated or plainly wrong, so
please anyone else with more knowledge on this do correct me if needed.
In any case, here it goes anyway in case it can be of any help for your

Post by i***@noisolation.com
[...]
*Questions:*
1. Is there a concept of Memory Pressure for Linux? I cannot see it in

the source. I assume this is supposed to hint to Chromiums allocators to
ease off and GC when closing on the limit.
As far as I know there's still the old system based on the
MemoryPressureMonitor class in place under //base/memory [1], which is
still in use in some platforms but unfortunately not in Linux: just on
Windows, Mac and ChromeOs.
Also as far as I remember, there have been attempts in the past to
refactor this code into a |memory_pressure| component under
//components/memory_pressure, including a Linux implementation (see [2]).
However, and this is why you can't find anything in the sources, checking
the commit history [3] I found out that this has been abandoned like 1.5
years ago in favour of the "memory coordinator work" (see [4]).
So, I think I'd suggest getting familiar with that "memory coordinator"
work and figuring out the status of it for Linux, and whether you could
work in that area to get your use case supported. Mind you, I'm not
familiar with that so please take my word with a grain of salt, you'd
better get in touch with the right people there and get advice from them if
this is something you consider exploring.

Post by i***@noisolation.com
[...]
4. Any other ideas for me?

Alternative, you could follow another approach and do what Endless have
hack your own version of the old MemoryPressureMonitor for your platform
and adapt it to your needs. Endless did it based on ChromiumOS'
https://github.com/endlessm/chromium-browser/commit/6196cc79
https://github.com/endlessm/chromium-browser/commit/b6df3633
https://github.com/endlessm/chromium-browser/commit/146408fd
(you might find other interesting commits in the whole set of
downstream patches in [5])
However, this came not without problems (e.g. heuristics not being
entirely right, memory still filing up in some configurations...) so, based
on my own experience and if it's feasible for you, I think I'd advise you
to first check the status "memory coordinator" work first and see if that
can fit your needs, before going the "downstream-only" route :-).

Post by i***@noisolation.com
I really appreciate any help or guidance you may provide me with!

As I said a couple of times already, take this mail with a grain of salt,
but other than that I hope this is useful in some way.
Thanks,
Mario
[1]
https://cs.chromium.org/chromium/src/base/memory/memory_pressure_monitor.h
[2] https://bugs.chromium.org/p/chromium/issues/detail?id=463606
[3]
https://chromium.googlesource.com/chromium/src/+log/master/components/memory_pressure/direct_memory_pressure_calculator_linux.cc
[4]
https://chromium.googlesource.com/chromium/src/+/4d1357d65728d99aef9de10f4ab8a914cb09b96d
[5] https://github.com/endlessm/chromium-browser/commits/eos3.5