[chromium-dev] CP-437 is not a valid ICU converter

[Claudio deSouza]

Hi,

I have been using CodepageToUTF16
<https://cs.chromium.org/chromium/src/base/i18n/icu_string_conversions.h?l=44&gs=kythe%253A%252F%252Fchromium%253Flang%253Dc%25252B%25252B%253Fpath%253Dsrc%252Fbase%252Fi18n%252Ficu_string_conversions.h%2523gJuPcZiw64PZKrvqmg1Wn%25252Bxi8TM78Cse%25252F5uZd%25252BeIUY0%25253D&gsn=CodepageToUTF16&ct=xref_usages> to
convert data coming from custom codepages to UTF16, but it seems that
CP-437 is not supported for some reason, even though it is clearly listed
in the following URL:

http://demo.icu-project.org/icu-bin/convexp

As mentioned in the documentation, codepage 437 is handled by the following
mapper: http://demo.icu-project.org/icu-bin/convexp?conv=ibm-437_P100-1995&s=ALL

This mapper is not a valid mapper somehow though, even though the file for
this mapper is present in the repository.

When listing all the valid mappers with the following code:

int end = ucnv_countAvailable();
for (int i = 0; i != end; ++i) {
LOG(INFO) << ucnv_getAvailableName(i);
}

I'm getting this result

UTF-8
utf-16be
utf-16le
ibm866-html
iso-8859-2-html
iso-8859-3-html
iso-8859-4-html
iso-8859-5-html
iso-8859-6-html
iso-8859-7-html
iso-8859-8-html
iso-8859-10-html
iso-8859-13-html
iso-8859-14-html
iso-8859-15-html
iso-8859-16-html
koi8-r-html
koi8-u-html
macintosh-html
windows-874-html
windows-1250-html
windows-1251-html
windows-1252-html
windows-1253-html
windows-1254-html
windows-1255-html
windows-1256-html
windows-1257-html
windows-1258-html
x-mac-cyrillic-html
windows-936-2000
gb18030
big5-html
euc-jp-html
ISO_2022,locale=ja,version=0
shift_jis-html
euc-kr-html

As can be seen, the mapper *ibm-437_P100-1995* is not listed in there. May
I please have some help understanding why that's the case?

[PhistucK]

A reason could be - if it is not needed by the browser, it might have been
excluded in order to save disk space.

☆*PhistucK*

Post by Claudio deSouza
Hi,
I have been using CodepageToUTF16
<https://cs.chromium.org/chromium/src/base/i18n/icu_string_conversions.h?l=44&gs=kythe%253A%252F%252Fchromium%253Flang%253Dc%25252B%25252B%253Fpath%253Dsrc%252Fbase%252Fi18n%252Ficu_string_conversions.h%2523gJuPcZiw64PZKrvqmg1Wn%25252Bxi8TM78Cse%25252F5uZd%25252BeIUY0%25253D&gsn=CodepageToUTF16&ct=xref_usages> to
convert data coming from custom codepages to UTF16, but it seems that
CP-437 is not supported for some reason, even though it is clearly listed
http://demo.icu-project.org/icu-bin/convexp
As mentioned in the documentation, codepage 437 is handled by the
http://demo.icu-project.org/icu-bin/convexp?conv=ibm-437_P100-1995&s=ALL
This mapper is not a valid mapper somehow though, even though the file for
this mapper is present in the repository.
int end = ucnv_countAvailable();
for (int i = 0; i != end; ++i) {
LOG(INFO) << ucnv_getAvailableName(i);
}
I'm getting this result
UTF-8
utf-16be
utf-16le
ibm866-html
iso-8859-2-html
iso-8859-3-html
iso-8859-4-html
iso-8859-5-html
iso-8859-6-html
iso-8859-7-html
iso-8859-8-html
iso-8859-10-html
iso-8859-13-html
iso-8859-14-html
iso-8859-15-html
iso-8859-16-html
koi8-r-html
koi8-u-html
macintosh-html
windows-874-html
windows-1250-html
windows-1251-html
windows-1252-html
windows-1253-html
windows-1254-html
windows-1255-html
windows-1256-html
windows-1257-html
windows-1258-html
x-mac-cyrillic-html
windows-936-2000
gb18030
big5-html
euc-jp-html
ISO_2022,locale=ja,version=0
shift_jis-html
euc-kr-html
As can be seen, the mapper *ibm-437_P100-1995* is not listed in there.
May I please have some help understanding why that's the case?
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/3d615dda-5f07-4b17-9e1b-f1db6a4bcc4e%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/3d615dda-5f07-4b17-9e1b-f1db6a4bcc4e%40chromium.org?utm_medium=email&utm_source=footer>
.

[Claudio deSouza]

Yes, it appears to be so. I wonder what would be the less disruptive way to
re-activate these mappers, without much disruption/patches.

[PhistucK]

It might be a one-liner -
https://cs.chromium.org/chromium/src/third_party/icu/icu.gyp?l=29&rcl=407b39301e71006b68bd38e770f35d32398a7b14

☆*PhistucK*

Post by Claudio deSouza
Yes, it appears to be so. I wonder what would be the less disruptive way
to re-activate these mappers, without much disruption/patches.
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e284e6c6-f6dc-4986-8f1b-674d50f30946%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e284e6c6-f6dc-4986-8f1b-674d50f30946%40chromium.org?utm_medium=email&utm_source=footer>
.

[Claudio deSouza]

Let me try... this looks two good to be true.

Post by PhistucK
It might be a one-liner -
https://cs.chromium.org/chromium/src/third_party/icu/icu.gyp?l=29&rcl=407b39301e71006b68bd38e770f35d32398a7b14
☆*PhistucK*

Post by Claudio deSouza
Yes, it appears to be so. I wonder what would be the less disruptive way
to re-activate these mappers, without much disruption/patches.
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e284e6c6-f6dc-4986-8f1b-674d50f30946%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e284e6c6-f6dc-4986-8f1b-674d50f30946%40chromium.org?utm_medium=email&utm_source=footer>
.

[Nico Weber]

If this is for a Chromium build, you probably want
https://cs.chromium.org/chromium/src/third_party/icu/BUILD.gn?g=0&l=73 --
we stopped using gyp a while ago.

Post by Claudio deSouza
Let me try... this looks two good to be true.

Post by PhistucK
It might be a one-liner -
https://cs.chromium.org/chromium/src/third_party/icu/icu.gyp?l=29&rcl=407b39301e71006b68bd38e770f35d32398a7b14
☆*PhistucK*

Post by Claudio deSouza
Yes, it appears to be so. I wonder what would be the less disruptive way
to re-activate these mappers, without much disruption/patches.
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e284e6c6-f6dc-4986-8f1b-674d50f30946%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e284e6c6-f6dc-4986-8f1b-674d50f30946%40chromium.org?utm_medium=email&utm_source=footer>
.

--

--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/416a1872-c857-4a5a-b21e-e7414be85dc5%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/416a1872-c857-4a5a-b21e-e7414be85dc5%40chromium.org?utm_medium=email&utm_source=footer>
.

[PhistucK]

Whoops! I knew that, I thought it was .gn because it was below .gni in
CodeSearch. :( Thank you for the correction.

☆*PhistucK*

Post by Nico Weber
If this is for a Chromium build, you probably want
https://cs.chromium.org/chromium/src/third_party/icu/BUILD.gn?g=0&l=73 --
we stopped using gyp a while ago.

Post by Claudio deSouza
Let me try... this looks two good to be true.

Post by PhistucK
It might be a one-liner -
https://cs.chromium.org/chromium/src/third_party/icu/icu.gyp?l=29&rcl=407b39301e71006b68bd38e770f35d32398a7b14
☆*PhistucK*

Post by Claudio deSouza
Yes, it appears to be so. I wonder what would be the less disruptive
way to re-activate these mappers, without much disruption/patches.
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e284e6c6-f6dc-4986-8f1b-674d50f30946%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e284e6c6-f6dc-4986-8f1b-674d50f30946%40chromium.org?utm_medium=email&utm_source=footer>
.

--

--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/416a1872-c857-4a5a-b21e-e7414be85dc5%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/416a1872-c857-4a5a-b21e-e7414be85dc5%40chromium.org?utm_medium=email&utm_source=footer>
.

--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAMGbLiGOVDEuUJCksDZgWxRTOZMvWVgQwBvzHR%3D_MKHsMrDh%2BQ%40mail.gmail.com
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAMGbLiGOVDEuUJCksDZgWxRTOZMvWVgQwBvzHR%3D_MKHsMrDh%2BQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

[Claudio deSouza]

I have check it out but there's more...

I think that the icudtl.dat file is bundled with the mappers, necessary for
the converters, and only the ones mention in the following file get bundled:

https://cs.chromium.org/chromium/src/third_party/icu/source/data/mappings/convrtrs.txt?g=0

However, by what I've noticed in the heading happens not during the build,
but it is rather pushed to git regularly by a maintainer in the chromium
team... I might be mistaken on this, but that's the impression I'm getting
from this explanation:

https://cs.chromium.org/chromium/src/third_party/icu/README.chromium?g=0&l=122

Any clues, I really want to bundle these code page mappings on a build I'm
doing, but it is not very clear to me how to get them included.

Cheers.

[PhistucK]

Jungshik - can you assist? You seem to be committing the .dat files
regularly.

☆*PhistucK*

Post by Claudio deSouza
I have check it out but there's more...
I think that the icudtl.dat file is bundled with the mappers, necessary
for the converters, and only the ones mention in the following file get
https://cs.chromium.org/chromium/src/third_party/icu/source/data/mappings/convrtrs.txt?g=0
However, by what I've noticed in the heading happens not during the build,
but it is rather pushed to git regularly by a maintainer in the chromium
team... I might be mistaken on this, but that's the impression I'm getting
https://cs.chromium.org/chromium/src/third_party/icu/README.chromium?g=0&l=122
Any clues, I really want to bundle these code page mappings on a build I'm
doing, but it is not very clear to me how to get them included.
Cheers.
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e1b424d6-243e-4bb1-82a2-c8ddbae62701%40chromium.org
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e1b424d6-243e-4bb1-82a2-c8ddbae62701%40chromium.org?utm_medium=email&utm_source=footer>
.

[Claudio deSouza]

Ideally, it would be nice if we could have a gn config to opt-in into a ICU
version that has all ICU converters.

I would be more than willing to liason Jungshik on getting that
code-reviewed/merged.

Cheers,

Claudio DeSouza

[Daniel Bratell]

As you have noticed, the data files for icu are built offline by jshin and
checked into the tree. You can compile them locally and replace what is
checked in. The instructions are in third_party/icu/README.chromium and I
think they were correct as of a couple of years ago. It's not trivial, but
more fiddly than hard.

The current setup isn't exactly optimal. A couple of years ago I tried to
move the generation of data files to the normal build systems both to
avoid binary files in git and to make it easier for us to modify what data
is included but couldn't get anyone to look at the patch. (Opera adds some
locales that are important to our users but are not important enough to
Chrome's users.)

/Daniel

On Sun, 25 Nov 2018 22:57:14 +0100, Claudio deSouza

Post by Claudio deSouza
Ideally, it would be nice if we could have a gn config to opt-in into a
ICU version that has all ICU converters.
I would be more than willing to liason Jungshik on getting that
code-reviewed/merged.

Post by Claudio deSouza
Cheers,
Claudio DeSouza

http://groups.google.com/a/chromium.org/group/chromium-dev
---You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/74926274-b3fd-4cf5-921a->aaafbf044081%40chromium.org.

--
/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Peter Kasting]

Post by Daniel Bratell
The current setup isn't exactly optimal. A couple of years ago I tried to
move the generation of data files to the normal build systems both to avoid
binary files in git and to make it easier for us to modify what data is
included but couldn't get anyone to look at the patch. (Opera adds some
locales that are important to our users but are not important enough to
Chrome's users.)

This seems worth fixing. If you still have the patch I will either review
it or find someone who can.

PK

[Daniel Bratell]

Post by Peter Kasting

Post by Daniel Bratell
The current setup isn't exactly optimal. A couple of years ago I tried
to move the generation of data files to the normal build systems both
to avoid >>binary files in git and to make it easier for us to modify
what data is included but couldn't get anyone to look at the patch.
(Opera adds some locales >>that are important to our users but are not
important enough to Chrome's users.)

This seems worth fixing. If you still have the patch I will either
review it or find someone who can.

I've not kept it alive so I don't know how much work it is to update but
it's in https://codereview.chromium.org/1000163003/ . The mass of the
patch is listing all data files (The 10+ MB icu file contains data from
hundreds or thousands of data source files) and a ported build system of
the tools that build the icu data file.

I'll see if I can get it going.

/Daniel
--
/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Nico Weber]

(If this adds a 40s and a 35s build step on Windows for something only
out-of-tree clients use, I'm not sure if keeping this patch out-of-tree
wouldn't be better. That's fairly expensive.)

Post by Peter Kasting

Post by Daniel Bratell
The current setup isn't exactly optimal. A couple of years ago I tried to
move the generation of data files to the normal build systems both to avoid
binary files in git and to make it easier for us to modify what data is
included but couldn't get anyone to look at the patch. (Opera adds some
locales that are important to our users but are not important enough to
Chrome's users.)

This seems worth fixing. If you still have the patch I will either review
it or find someone who can.
I've not kept it alive so I don't know how much work it is to update but
it's in https://codereview.chromium.org/1000163003/ . The mass of the
patch is listing all data files (The 10+ MB icu file contains data from
hundreds or thousands of data source files) and a ported build system of
the tools that build the icu data file.
I'll see if I can get it going.
/Daniel
--
/* Opera Software, Linköping, Sweden: CET (UTC+1) */
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.zs3gnlisrbppqq%40cicero2.linkoping.osa
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.zs3gnlisrbppqq%40cicero2.linkoping.osa?utm_medium=email&utm_source=footer>
.

[Daniel Bratell]

It adds many small steps so I don't know what the increase would be in
wall time in a typical full build clean build on a massively parallel
computer. I would have to get it running to see but I'd guess not much.

If the numbers I quoted there still are correct, then it would be an
increase of the work in CPU cycles by about 0.3%, or the increase we get
from a couple of days of average development. It was relatively speaking
much more in 2015.

/Daniel

Post by Nico Weber
(If this adds a 40s and a 35s build step on Windows for something only
out-of-tree clients use, I'm not sure if keeping this patch out-of-tree
wouldn't >be better. That's fairly expensive.)

On Mon, 26 Nov 2018 16:45:07 +0100, Peter Kasting

Post by Peter Kasting

Post by Daniel Bratell
The current setup isn't exactly optimal. A couple of years ago I
tried to move the generation of data files to the normal build
systems both to >>>>avoid binary files in git and to make it easier
for us to modify what data is included but couldn't get anyone to
look at the patch. (Opera adds >>>>some locales that are important to
our users but are not important enough to Chrome's users.)

This seems worth fixing. If you still have the patch I will either
review it or find someone who can.

I've not kept it alive so I don't know how much work it is to update
but it's in https://codereview.chromium.org/1000163003/ . The mass of
the patch >>is listing all data files (The 10+ MB icu file contains
data from hundreds or thousands of data source files) and a ported
build system of the tools >>that build the icu data file.
I'll see if I can get it going.
/Daniel
--/* Opera Software, Linköping, Sweden: CET (UTC+1) */
http://groups.google.com/a/chromium.org/group/chromium-dev
---You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.zs3gnlisrbppqq>>%40cicero2.linkoping.osa.

--
/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Nico Weber]

Ah, if it's many small build steps that's better. (Note that full goma
builds are like 6 min wall time, and non-compile steps run locally.)

Worst case, we could land the generation behind off-by-defaul gn arg, but
maybe it's not needed.

Post by Daniel Bratell
It adds many small steps so I don't know what the increase would be in
wall time in a typical full build clean build on a massively parallel
computer. I would have to get it running to see but I'd guess not much.
If the numbers I quoted there still are correct, then it would be an
increase of the work in CPU cycles by about 0.3%, or the increase we get
from a couple of days of average development. It was relatively speaking
much more in 2015.
/Daniel
(If this adds a 40s and a 35s build step on Windows for something only
out-of-tree clients use, I'm not sure if keeping this patch out-of-tree
wouldn't be better. That's fairly expensive.)

Post by Peter Kasting

Post by Daniel Bratell
The current setup isn't exactly optimal. A couple of years ago I tried
to move the generation of data files to the normal build systems both to
avoid binary files in git and to make it easier for us to modify what data
is included but couldn't get anyone to look at the patch. (Opera adds some
locales that are important to our users but are not important enough to
Chrome's users.)

This seems worth fixing. If you still have the patch I will either
review it or find someone who can.
I've not kept it alive so I don't know how much work it is to update but
it's in https://codereview.chromium.org/1000163003/ . The mass of the
patch is listing all data files (The 10+ MB icu file contains data from
hundreds or thousands of data source files) and a ported build system of
the tools that build the icu data file.
I'll see if I can get it going.
/Daniel
--
/* Opera Software, Linköping, Sweden: CET (UTC+1) */
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.zs3gnlisrbppqq%40cicero2.linkoping.osa
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/op.zs3gnlisrbppqq%40cicero2.linkoping.osa?utm_medium=email&utm_source=footer>
.

--
/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Claudio deSouza]

Daniel,

This seems indeed non-trivial work to be done. I think that having it being
generated during the build would be the ideal solution. I would gladly help
in that, to get that CL you were working merged, but I don't have in-depth
knowledge on the subject. I still can help if you guide me.

[Daniel Bratell]

The patch is non-trivial to apply because of its age and all the changes
that have happened to Chromium's build system. I'm giving it a go and I'll
let you know if it still works, or how much work it would be.

/Daniel

On Mon, 26 Nov 2018 18:00:24 +0100, Claudio deSouza

Post by Claudio deSouza
Daniel,
This seems indeed non-trivial work to be done. I think that having it
being generated during the build would be the ideal solution. I would
gladly help in >that, to get that CL you were working merged, but I
don't have in-depth knowledge on the subject. I still can help if you
guide me.
http://groups.google.com/a/chromium.org/group/chromium-dev
---You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/bfc34ef4-a6cb-47a6-96a5->d11e69800a66%40chromium.org.

--
/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Claudio deSouza]

Daniel

I forgot to mention but I had reported an issue before you chimed in into
this discussion.

https://crbug.com/908373

Let me know if I can provide any assistance.

[Daniel Bratell]

If the generation moves to the build system, then this will be easier,
though everyone should be aware that the file contains 1000+ resources
picked from many more thousand in the icu project. There is a lot of data
hidden in the icu data file.

I have been looking at reviving my old branch, and it seems to work, but
it needs to catch up on 4 years of patches to the scripts and regexps.

/Daniel

On Wed, 28 Nov 2018 00:06:17 +0100, Claudio deSouza

Post by Daniel Bratell
Daniel
I forgot to mention but I had reported an issue before you chimed in
into this discussion.
https://crbug.com/908373
Let me know if I can provide any assistance.
http://groups.google.com/a/chromium.org/group/chromium-dev
---You received this message because you are subscribed to the Google
Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/e4a54f5c-5dff-4e36-9f55-97e24993016d>%40chromium.org.

--
/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Claudio deSouza]

I would appreciate when you have one CL up for review... it would be nice
to try it out on my end.

Cheers,

Claudio.

[Claudio deSouza]

I mean, I would appreciate if you pinged here back.