[chromium-dev] simulating key presses

[Luigi Semenzato]

he telemetry code simulates key presses (for instance, for typing
name and password at login prompts) by sending a json dictionary to a
debug socket with "method": "Input.dispatchKeyEvent". It appears that
it would also be possible to do the same thing by injecting some
Javascript that calls document.dispatchEvent(keyboardEvent). I am
wondering if these are exactly equivalent or if there is some benefit
for one vs. the other.

(I am implementing this functionality in Go and I slightly favor the
second choice, mostly because the existing API uses a single devtools
socket.)

Thanks!

[Dave Tapuska]

No they are not equivalent. Injecting events via the debugging protocol
<https://chromedevtools.github.io/devtools-protocol/tot/Input> (devtools)
causes the events to appear from the browser.

Calling dispatchEvent in your example:
1) Avoids dispatching the event to the window (if you are calling
document.dispatchEvent)).
2) The events are untrusted so they don't create user gestures nor perform
the default action.
3) They avoid the browser processing; things like keyboard accelerators
(ie; think Ctrl-N for new tab, etc).

dave.

Post by Luigi Semenzato
he telemetry code simulates key presses (for instance, for typing
name and password at login prompts) by sending a json dictionary to a
debug socket with "method": "Input.dispatchKeyEvent". It appears that
it would also be possible to do the same thing by injecting some
Javascript that calls document.dispatchEvent(keyboardEvent). I am
wondering if these are exactly equivalent or if there is some benefit
for one vs. the other.
(I am implementing this functionality in Go and I slightly favor the
second choice, mostly because the existing API uses a single devtools
socket.)
Thanks!
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAA25o9SKg3P0YGYAHPVPmgu9Jj8ueyBq8F%3DB6m1Bk3F9TrQULQ%40mail.gmail.com
.

[Jacob Dufault]

Have you considered instead generating events as-if they come from
hardware? desktopui_ScreenLocker does that here
<http://cs/chromeos_public/src/third_party/autotest/files/client/site_tests/desktopui_ScreenLocker/desktopui_ScreenLocker.py?l=110-111&rcl=9f759f8e3e397a0c5b273b5daf9003cd0f3ebf74>
.

The advantages being:
- it will catch focus regressions
- does not depend on UI being implemented as a webui
- more similar to how a user actually uses the device since it uses more of
the input stack

Post by Dave Tapuska
No they are not equivalent. Injecting events via the debugging protocol
<https://chromedevtools.github.io/devtools-protocol/tot/Input> (devtools)
causes the events to appear from the browser.
1) Avoids dispatching the event to the window (if you are calling
document.dispatchEvent)).
2) The events are untrusted so they don't create user gestures nor perform
the default action.
3) They avoid the browser processing; things like keyboard accelerators
(ie; think Ctrl-N for new tab, etc).
dave.

Post by Luigi Semenzato
he telemetry code simulates key presses (for instance, for typing
name and password at login prompts) by sending a json dictionary to a
debug socket with "method": "Input.dispatchKeyEvent". It appears that
it would also be possible to do the same thing by injecting some
Javascript that calls document.dispatchEvent(keyboardEvent). I am
wondering if these are exactly equivalent or if there is some benefit
for one vs. the other.
(I am implementing this functionality in Go and I slightly favor the
second choice, mostly because the existing API uses a single devtools
socket.)
Thanks!
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAA25o9SKg3P0YGYAHPVPmgu9Jj8ueyBq8F%3DB6m1Bk3F9TrQULQ%40mail.gmail.com
.
--

--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHgVhZWAVWdNOD7MnK7a%3DYwURuK9ZQmDfca%3DRnKkbAmr7Fib-A%40mail.gmail.com
<https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHgVhZWAVWdNOD7MnK7a%3DYwURuK9ZQmDfca%3DRnKkbAmr7Fib-A%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

[Luigi Semenzato]

Thank you for the clarification and the suggestion. Given that this
is a chromeos-specific application, evemu-play (used by
desktopui_ScreenLocker) is a very appealing option. :)

Have you considered instead generating events as-if they come from hardware? desktopui_ScreenLocker does that here.
- it will catch focus regressions
- does not depend on UI being implemented as a webui
- more similar to how a user actually uses the device since it uses more of the input stack

No they are not equivalent. Injecting events via the debugging protocol (devtools) causes the events to appear from the browser.
1) Avoids dispatching the event to the window (if you are calling document.dispatchEvent)).
2) The events are untrusted so they don't create user gestures nor perform the default action.
3) They avoid the browser processing; things like keyboard accelerators (ie; think Ctrl-N for new tab, etc).
dave.

Post by Luigi Semenzato
he telemetry code simulates key presses (for instance, for typing
name and password at login prompts) by sending a json dictionary to a
debug socket with "method": "Input.dispatchKeyEvent". It appears that
it would also be possible to do the same thing by injecting some
Javascript that calls document.dispatchEvent(keyboardEvent). I am
wondering if these are exactly equivalent or if there is some benefit
for one vs. the other.
(I am implementing this functionality in Go and I slightly favor the
second choice, mostly because the existing API uses a single devtools
socket.)
Thanks!
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAA25o9SKg3P0YGYAHPVPmgu9Jj8ueyBq8F%3DB6m1Bk3F9TrQULQ%40mail.gmail.com.

--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHgVhZWAVWdNOD7MnK7a%3DYwURuK9ZQmDfca%3DRnKkbAmr7Fib-A%40mail.gmail.com.

[Luigi Semenzato]

It turns out that there are drawbacks to using evemu-play to emulate
key presses (it's racy).

Going back to the element.dispatchEvent approach, I am now wondering
if and how it can be made to work. I was planning to use
chrome.tabs.ExecuteScript, but according to the docs here:

https://developer.chrome.com/extensions/content_scripts#execution-environment

the API available to content scripts seems quite limited.

More concretely, the injection would be performed by a promise created as shown:

new Promise((resolve, reject) => {
chrome.tabs.executeScript(1732467094, {code: "
{
var kbEvent = document.createEvent('KeyboardEvent');
var initKind = typeof kbEvent.initKeyboardEvent !== 'undefined' ?
'initKeyboardEvent' : 'initKeyEvent';
kbEvent[initKind]('keydown',
true,
true,
window,
false,
false,
false,
false,
42, // keycode
0);
document.dispatchEvent(keyboardEvent);
}"}, () => { resolve() }) });

Am I delusional?

Post by Luigi Semenzato
Thank you for the clarification and the suggestion. Given that this
is a chromeos-specific application, evemu-play (used by
desktopui_ScreenLocker) is a very appealing option. :)

Have you considered instead generating events as-if they come from hardware? desktopui_ScreenLocker does that here.
- it will catch focus regressions
- does not depend on UI being implemented as a webui
- more similar to how a user actually uses the device since it uses more of the input stack

No they are not equivalent. Injecting events via the debugging protocol (devtools) causes the events to appear from the browser.
1) Avoids dispatching the event to the window (if you are calling document.dispatchEvent)).
2) The events are untrusted so they don't create user gestures nor perform the default action.
3) They avoid the browser processing; things like keyboard accelerators (ie; think Ctrl-N for new tab, etc).
dave.

Post by Luigi Semenzato
he telemetry code simulates key presses (for instance, for typing
name and password at login prompts) by sending a json dictionary to a
debug socket with "method": "Input.dispatchKeyEvent". It appears that
it would also be possible to do the same thing by injecting some
Javascript that calls document.dispatchEvent(keyboardEvent). I am
wondering if these are exactly equivalent or if there is some benefit
for one vs. the other.
(I am implementing this functionality in Go and I slightly favor the
second choice, mostly because the existing API uses a single devtools
socket.)
Thanks!
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAA25o9SKg3P0YGYAHPVPmgu9Jj8ueyBq8F%3DB6m1Bk3F9TrQULQ%40mail.gmail.com.

--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHgVhZWAVWdNOD7MnK7a%3DYwURuK9ZQmDfca%3DRnKkbAmr7Fib-A%40mail.gmail.com.

[Dave Tapuska]

Like I indicated before calling dispatchEvent in javascript is not the same
as the UA calling dispatchEvent.

The former does not invoke the default event handlers. It will call
javascript that has handlers for keydown, but if the UA does something in
response to a key (such as 'Tab' moving focus) it won't perform it in that
way. Also you won't have the context of a user gesture which is required
for some APIs to function.

You'll need to decide if that does cover your problem space or not.

dave.

Post by Luigi Semenzato
It turns out that there are drawbacks to using evemu-play to emulate
key presses (it's racy).
Going back to the element.dispatchEvent approach, I am now wondering
if and how it can be made to work. I was planning to use
https://developer.chrome.com/extensions/content_scripts#execution-environment
the API available to content scripts seems quite limited.
new Promise((resolve, reject) => {
chrome.tabs.executeScript(1732467094, {code: "
{
var kbEvent = document.createEvent('KeyboardEvent');
var initKind = typeof kbEvent.initKeyboardEvent !== 'undefined' ?
'initKeyboardEvent' : 'initKeyEvent';
kbEvent[initKind]('keydown',
true,
true,
window,
false,
false,
false,
false,
42, // keycode
0);
document.dispatchEvent(keyboardEvent);
}"}, () => { resolve() }) });
Am I delusional?

Post by Luigi Semenzato
Thank you for the clarification and the suggestion. Given that this
is a chromeos-specific application, evemu-play (used by
desktopui_ScreenLocker) is a very appealing option. :)

Post by Jacob Dufault
Have you considered instead generating events as-if they come from

hardware? desktopui_ScreenLocker does that here.

Post by Luigi Semenzato

Post by Jacob Dufault
- it will catch focus regressions
- does not depend on UI being implemented as a webui
- more similar to how a user actually uses the device since it uses

more of the input stack

Post by Luigi Semenzato

Post by Jacob Dufault

Post by Dave Tapuska
No they are not equivalent. Injecting events via the debugging

protocol (devtools) causes the events to appear from the browser.

Post by Luigi Semenzato

Post by Jacob Dufault

Post by Dave Tapuska
1) Avoids dispatching the event to the window (if you are calling

document.dispatchEvent)).

Post by Luigi Semenzato

Post by Jacob Dufault

Post by Dave Tapuska
2) The events are untrusted so they don't create user gestures nor

perform the default action.

Post by Luigi Semenzato

Post by Jacob Dufault

Post by Dave Tapuska
3) They avoid the browser processing; things like keyboard

accelerators (ie; think Ctrl-N for new tab, etc).

Post by Luigi Semenzato

Post by Jacob Dufault

Post by Dave Tapuska
dave.
On Wed, Nov 21, 2018 at 4:10 PM Luigi Semenzato <

Post by Luigi Semenzato
he telemetry code simulates key presses (for instance, for typing
name and password at login prompts) by sending a json dictionary to a
debug socket with "method": "Input.dispatchKeyEvent". It appears

that

Post by Luigi Semenzato

Post by Jacob Dufault

Post by Dave Tapuska

Post by Luigi Semenzato
it would also be possible to do the same thing by injecting some
Javascript that calls document.dispatchEvent(keyboardEvent). I am
wondering if these are exactly equivalent or if there is some benefit
for one vs. the other.
(I am implementing this functionality in Go and I slightly favor the
second choice, mostly because the existing API uses a single devtools
socket.)
Thanks!
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google

Groups "Chromium-dev" group.

Post by Luigi Semenzato

Post by Jacob Dufault

Post by Dave Tapuska

Post by Luigi Semenzato
To view this discussion on the web visit

https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAA25o9SKg3P0YGYAHPVPmgu9Jj8ueyBq8F%3DB6m1Bk3F9TrQULQ%40mail.gmail.com
.

Post by Luigi Semenzato

Post by Jacob Dufault

Post by Dave Tapuska
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google

Groups "Chromium-dev" group.

Post by Luigi Semenzato

Post by Jacob Dufault

Post by Dave Tapuska
To unsubscribe from this group and stop receiving emails from it,
To view this discussion on the web visit

https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHgVhZWAVWdNOD7MnK7a%3DYwURuK9ZQmDfca%3DRnKkbAmr7Fib-A%40mail.gmail.com
.

[Luigi Semenzato]

My problem space is just entering login name and password for various sites.

So I gather that this won't work in general then? I would imagine
that a number of login entry widgets should be able to use the default
handlers. (Maybe not so for passwords)

Like I indicated before calling dispatchEvent in javascript is not the same as the UA calling dispatchEvent.
The former does not invoke the default event handlers. It will call javascript that has handlers for keydown, but if the UA does something in response to a key (such as 'Tab' moving focus) it won't perform it in that way. Also you won't have the context of a user gesture which is required for some APIs to function.
You'll need to decide if that does cover your problem space or not.
dave.

Post by Luigi Semenzato
It turns out that there are drawbacks to using evemu-play to emulate
key presses (it's racy).
Going back to the element.dispatchEvent approach, I am now wondering
if and how it can be made to work. I was planning to use
https://developer.chrome.com/extensions/content_scripts#execution-environment
the API available to content scripts seems quite limited.
new Promise((resolve, reject) => {
chrome.tabs.executeScript(1732467094, {code: "
{
var kbEvent = document.createEvent('KeyboardEvent');
var initKind = typeof kbEvent.initKeyboardEvent !== 'undefined' ?
'initKeyboardEvent' : 'initKeyEvent';
kbEvent[initKind]('keydown',
true,
true,
window,
false,
false,
false,
false,
42, // keycode
0);
document.dispatchEvent(keyboardEvent);
}"}, () => { resolve() }) });
Am I delusional?

Post by Luigi Semenzato
Thank you for the clarification and the suggestion. Given that this
is a chromeos-specific application, evemu-play (used by
desktopui_ScreenLocker) is a very appealing option. :)

Have you considered instead generating events as-if they come from hardware? desktopui_ScreenLocker does that here.
- it will catch focus regressions
- does not depend on UI being implemented as a webui
- more similar to how a user actually uses the device since it uses more of the input stack

No they are not equivalent. Injecting events via the debugging protocol (devtools) causes the events to appear from the browser.
1) Avoids dispatching the event to the window (if you are calling document.dispatchEvent)).
2) The events are untrusted so they don't create user gestures nor perform the default action.
3) They avoid the browser processing; things like keyboard accelerators (ie; think Ctrl-N for new tab, etc).
dave.

Post by Luigi Semenzato
he telemetry code simulates key presses (for instance, for typing
name and password at login prompts) by sending a json dictionary to a
debug socket with "method": "Input.dispatchKeyEvent". It appears that
it would also be possible to do the same thing by injecting some
Javascript that calls document.dispatchEvent(keyboardEvent). I am
wondering if these are exactly equivalent or if there is some benefit
for one vs. the other.
(I am implementing this functionality in Go and I slightly favor the
second choice, mostly because the existing API uses a single devtools
socket.)
Thanks!
--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAA25o9SKg3P0YGYAHPVPmgu9Jj8ueyBq8F%3DB6m1Bk3F9TrQULQ%40mail.gmail.com.

--
--
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAHgVhZWAVWdNOD7MnK7a%3DYwURuK9ZQmDfca%3DRnKkbAmr7Fib-A%40mail.gmail.com.