UiAutomation  |  API reference  |  Android Developers

Class for interacting with the device's UI by simulation user actions and introspection of the screen content. It relies on the platform accessibility APIs to introspect the screen and to perform some actions on the remote view tree. It also allows injecting of arbitrary raw input events simulating user interaction with keyboards and touch devices. One can think of a UiAutomation as a special type of AccessibilityService which does not provide hooks for the service life cycle and exposes other APIs that are useful for UI test automation.

The APIs exposed by this class are low-level to maximize flexibility when developing UI test automation tools and libraries. Generally, a UiAutomation client should be using a higher-level library or implement high-level functions. For example, performing a tap on the screen requires construction and injecting of a touch down and up events which have to be delivered to the system by a call to injectInputEvent(InputEvent,boolean).

The APIs exposed by this class operate across applications enabling a client to write tests that cover use cases spanning over multiple applications. For example, going to the settings application to change a setting and then interacting with another application whose behavior depends on that setting.

Nested classes

interface UiAutomation.AccessibilityEventFilter

Listener for filtering accessibility events. 

interface UiAutomation.OnAccessibilityEventListener

Listener for observing the AccessibilityEvent stream. 

Constants

int FLAG_DONT_SUPPRESS_ACCESSIBILITY_SERVICES

UiAutomation suppresses accessibility services by default.

int FLAG_DONT_USE_ACCESSIBILITY

UiAutomation uses the accessibility subsystem by default.

int ROTATION_FREEZE_0

Rotation constant: Freeze rotation to 0 degrees (natural orientation)

int ROTATION_FREEZE_180

Rotation constant: Freeze rotation to 180 degrees .

int ROTATION_FREEZE_270

Rotation constant: Freeze rotation to 270 degrees .

int ROTATION_FREEZE_90

Rotation constant: Freeze rotation to 90 degrees .

int ROTATION_FREEZE_CURRENT

Rotation constant: Freeze rotation to its current state.

int ROTATION_UNFREEZE

Rotation constant: Unfreeze rotation (rotating the device changes its rotation state).

Public methods

void adoptShellPermissionIdentity(String... permissions)

Adopt the permission identity of the shell UID only for the provided permissions.

void adoptShellPermissionIdentity()

Adopt the permission identity of the shell UID for all permissions.

boolean clearCache()

Clears the accessibility cache.

void clearWindowAnimationFrameStats()

This method was deprecated in API level 33. animation-frames are no-longer used. Use Shared FrameTimeline jank metrics instead.

boolean clearWindowContentFrameStats(int windowId)

Clears the frame statistics for the content of a given window.

void dropShellPermissionIdentity()

Drop the shell permission identity adopted by a previous call to adoptShellPermissionIdentity().

AccessibilityEvent executeAndWaitForEvent(Runnable command, UiAutomation.AccessibilityEventFilter filter, long timeoutMillis)

Executes a command and waits for a specific accessibility event up to a given wait timeout.

ParcelFileDescriptor executeShellCommand(String command)

Executes a shell command.

ParcelFileDescriptor[] executeShellCommandRw(String command)

Executes a shell command.

ParcelFileDescriptor[] executeShellCommandRwe(String command)

Executes a shell command.

AccessibilityNodeInfo findFocus(int focus)

Find the view that has the specified focus type.

AccessibilityNodeInfo getRootInActiveWindow()

Gets the root AccessibilityNodeInfo in the active window.

AccessibilityServiceInfo getServiceInfo()

Gets the an AccessibilityServiceInfo describing this UiAutomation.

WindowAnimationFrameStats getWindowAnimationFrameStats()

This method was deprecated in API level 33. animation-frames are no-longer used.

WindowContentFrameStats getWindowContentFrameStats(int windowId)

Gets the frame statistics for a given window.

List<AccessibilityWindowInfo> getWindows()

Gets the windows on the screen associated with the UiAutomation context (usually the default display.

SparseArray<List<AccessibilityWindowInfo>> getWindowsOnAllDisplays()

Gets the windows on the screen of all displays.

void grantRuntimePermission(String packageName, String permission)

Grants a runtime permission to a package.

void grantRuntimePermissionAsUser(String packageName, String permission, UserHandle userHandle)

Grants a runtime permission to a package for a user.

boolean injectInputEvent(InputEvent event, boolean sync)

A method for injecting an arbitrary input event.

boolean performGlobalAction(int action)

Performs a global action.

void revokeRuntimePermission(String packageName, String permission)

Revokes a runtime permission from a package.

void revokeRuntimePermissionAsUser(String packageName, String permission, UserHandle userHandle)

Revokes a runtime permission from a package.

void setAnimationScale(float scale)

Sets the system settings values that control the scaling factor for animations.

void setOnAccessibilityEventListener(UiAutomation.OnAccessibilityEventListener listener)

Sets a callback for observing the stream of AccessibilityEvents.

boolean setRotation(int rotation)

Sets the device rotation.

void setRunAsMonkey(boolean enable)

Sets whether this UiAutomation to run in a "monkey" mode.

void setServiceInfo(AccessibilityServiceInfo info)

Sets the AccessibilityServiceInfo that describes how this UiAutomation will be handled by the platform accessibility layer.

Bitmap takeScreenshot()

Takes a screenshot.

Bitmap takeScreenshot(Window window)

Used to capture a screenshot of a Window.

String toString()

Returns a string representation of the object.

void waitForIdle(long idleTimeoutMillis, long globalTimeoutMillis)

Waits for the accessibility event stream to become idle, which is not to have received an accessibility event within idleTimeoutMillis.

Inherited methods

From class java.lang.Object

Object clone()

Creates and returns a copy of this object.

boolean equals(Object obj)

Indicates whether some other object is "equal to" this one.

void finalize()

Called by the garbage collector on an object when garbage collection determines that there are no more references to the object.

final Class<?> getClass()

Returns the runtime class of this Object.

int hashCode()

Returns a hash code value for the object.

final void notify()

Wakes up a single thread that is waiting on this object's monitor.

final void notifyAll()

Wakes up all threads that are waiting on this object's monitor.

String toString()

Returns a string representation of the object.

final void wait(long timeoutMillis, int nanos)

Causes the current thread to wait until it is awakened, typically by being notified or interrupted, or until a certain amount of real time has elapsed.

final void wait(long timeoutMillis)

Causes the current thread to wait until it is awakened, typically by being notified or interrupted, or until a certain amount of real time has elapsed.

final void wait()

Causes the current thread to wait until it is awakened, typically by being notified or interrupted.

Constants

FLAG_DONT_SUPPRESS_ACCESSIBILITY_SERVICES

public static final int FLAG_DONT_SUPPRESS_ACCESSIBILITY_SERVICES

UiAutomation suppresses accessibility services by default. This flag specifies that existing accessibility services should continue to run, and that new ones may start. This flag is set when obtaining the UiAutomation from Instrumentation.getUiAutomation(int).

Constant Value: 1 (0x00000001)

FLAG_DONT_USE_ACCESSIBILITY

public static final int FLAG_DONT_USE_ACCESSIBILITY

UiAutomation uses the accessibility subsystem by default. This flag provides an option to eliminate the overhead of engaging the accessibility subsystem for tests that do not need to interact with the user interface. Setting this flag disables methods that rely on accessibility. This flag is set when obtaining the UiAutomation from Instrumentation.getUiAutomation(int).

Constant Value: 2 (0x00000002)

ROTATION_FREEZE_0

public static final int ROTATION_FREEZE_0

Rotation constant: Freeze rotation to 0 degrees (natural orientation)

Constant Value: 0 (0x00000000)

ROTATION_FREEZE_180

public static final int ROTATION_FREEZE_180

Rotation constant: Freeze rotation to 180 degrees .

Constant Value: 2 (0x00000002)

ROTATION_FREEZE_270

public static final int ROTATION_FREEZE_270

Rotation constant: Freeze rotation to 270 degrees .

Constant Value: 3 (0x00000003)

ROTATION_FREEZE_90

public static final int ROTATION_FREEZE_90

Rotation constant: Freeze rotation to 90 degrees .

Constant Value: 1 (0x00000001)

ROTATION_FREEZE_CURRENT

public static final int ROTATION_FREEZE_CURRENT

Rotation constant: Freeze rotation to its current state.

Constant Value: -1 (0xffffffff)

ROTATION_UNFREEZE

public static final int ROTATION_UNFREEZE

Rotation constant: Unfreeze rotation (rotating the device changes its rotation state).

Constant Value: -2 (0xfffffffe)

Public methods

adoptShellPermissionIdentity

public void adoptShellPermissionIdentity (String... permissions)

Adopt the permission identity of the shell UID only for the provided permissions. This allows you to call APIs protected permissions which normal apps cannot hold but are granted to the shell UID. If you already adopted shell permissions by calling this method, or adoptShellPermissionIdentity() a subsequent call will replace any previous adoption.

Note: This method behave differently from adoptShellPermissionIdentity(). Only the listed permissions will use the shell identity and other permissions will still check against the original UID

Parameters
permissions String: The permissions to adopt or null to adopt all.

adoptShellPermissionIdentity

public void adoptShellPermissionIdentity ()

Adopt the permission identity of the shell UID for all permissions. This allows you to call APIs protected permissions which normal apps cannot hold but are granted to the shell UID. If you already adopted all shell permissions by calling this method or adoptShellPermissionIdentity(String...) a subsequent call will replace any previous adoption. Note that your permission state becomes that of the shell UID and it is not a combination of your and the shell UID permissions.

Note: Calling this method adopts all shell permissions and overrides any subset of adopted permissions via adoptShellPermissionIdentity(String...).

clearCache

public boolean clearCache ()

Clears the accessibility cache.

Returns
boolean true if the cache was cleared

clearWindowAnimationFrameStats

public void clearWindowAnimationFrameStats ()

This method was deprecated in API level 33.
animation-frames are no-longer used. Use Shared FrameTimeline jank metrics instead.

Clears the window animation rendering statistics. These statistics contain information about the most recently rendered window animation frames, i.e. for window transition animations.

dropShellPermissionIdentity

public void dropShellPermissionIdentity ()

Drop the shell permission identity adopted by a previous call to adoptShellPermissionIdentity(). If you did not adopt the shell permission identity this method would be a no-op.

executeAndWaitForEvent

public AccessibilityEvent executeAndWaitForEvent (Runnable command, 
                UiAutomation.AccessibilityEventFilter filter, 
                long timeoutMillis)

Executes a command and waits for a specific accessibility event up to a given wait timeout. To detect a sequence of events one can implement a filter that keeps track of seen events of the expected sequence and returns true after the last event of that sequence is received.

Note: It is caller's responsibility to recycle the returned event.

Parameters
command Runnable: The command to execute.
filter UiAutomation.AccessibilityEventFilter: Filter that recognizes the expected event.
timeoutMillis long: The wait timeout in milliseconds.
Returns
AccessibilityEvent
Throws
IllegalStateException If the connection to the accessibility subsystem is not established.
TimeoutException If the expected event is not received within the timeout.

executeShellCommand

public ParcelFileDescriptor executeShellCommand (String command)

Executes a shell command. This method returns a file descriptor that points to the standard output stream. The command execution is similar to running "adb shell " from a host connected to the device.

Note: It is your responsibility to close the returned file descriptor once you are done reading.

Parameters
command String: The command to execute.
Returns
ParcelFileDescriptor A file descriptor to the standard output stream.

executeShellCommandRw

public ParcelFileDescriptor[] executeShellCommandRw (String command)

Executes a shell command. This method returns two file descriptors, one that points to the standard output stream (element at index 0), and one that points to the standard input stream (element at index 1). The command execution is similar to running "adb shell " from a host connected to the device.

Note: It is your responsibility to close the returned file descriptors once you are done reading/writing.

Parameters
command String: The command to execute.
This value cannot be null.
Returns
ParcelFileDescriptor[] File descriptors (out, in) to the standard output/input streams.
This value cannot be null.

executeShellCommandRwe

public ParcelFileDescriptor[] executeShellCommandRwe (String command)

Executes a shell command. This method returns three file descriptors, one that points to the standard output stream (element at index 0), one that points to the standard input stream (element at index 1), and one points to standard error stream (element at index 2). The command execution is similar to running "adb shell " from a host connected to the device.

Note: It is your responsibility to close the returned file descriptors once you are done reading/writing.

Parameters
command String: The command to execute.
This value cannot be null.
Returns
ParcelFileDescriptor[] File descriptors (out, in, err) to the standard output/input/error streams.
This value cannot be null.

getWindowAnimationFrameStats

public WindowAnimationFrameStats getWindowAnimationFrameStats ()

This method was deprecated in API level 33.
animation-frames are no-longer used.

Gets the window animation frame statistics. These statistics contain information about the most recently rendered window animation frames, i.e. for window transition animations.

A typical usage requires clearing the window animation frame statistics via clearWindowAnimationFrameStats() followed by an interaction that causes a window transition which uses a window animation and finally getting the window animation frame statistics by calling this method.

 // Start with a clean slate.
 uiAutimation.clearWindowAnimationFrameStats();

 // Do stuff to trigger a window transition.

 // Get the frame statistics.
 WindowAnimationFrameStats stats = uiAutomation.getWindowAnimationFrameStats();
 
Returns
WindowAnimationFrameStats The window animation frame statistics.

getWindowContentFrameStats

public WindowContentFrameStats getWindowContentFrameStats (int windowId)

Gets the frame statistics for a given window. These statistics contain information about the most recently rendered content frames.

A typical usage requires clearing the window frame statistics via clearWindowContentFrameStats(int) followed by an interaction with the UI and finally getting the window frame statistics via calling this method.

 // Assume we have at least one window.
 final int windowId = getWindows().get(0).getId();

 // Start with a clean slate.
 uiAutimation.clearWindowContentFrameStats(windowId);

 // Do stuff with the UI.

 // Get the frame statistics.
 WindowContentFrameStats stats = uiAutomation.getWindowContentFrameStats(windowId);
 
Parameters
windowId int: The window id.
Returns
WindowContentFrameStats The window frame statistics, or null if the window is not present.
Throws
IllegalStateException If the connection to the accessibility subsystem is not established.

See also:

getWindows

public List<AccessibilityWindowInfo> getWindows ()

Gets the windows on the screen associated with the UiAutomation context (usually the default display.

This method returns only the windows that a sighted user can interact with, as opposed to all windows.

For example, if there is a modal dialog shown and the user cannot touch anything behind it, then only the modal window will be reported (assuming it is the top one). For convenience the returned windows are ordered in a descending layer order, which is the windows that are higher in the Z-order are reported first.

Note: In order to access the windows you have to opt-in to retrieve the interactive windows by setting the AccessibilityServiceInfo.FLAG_RETRIEVE_INTERACTIVE_WINDOWS flag.

Returns
List<AccessibilityWindowInfo> The windows if there are windows such, otherwise an empty list.
Throws
IllegalStateException If the connection to the accessibility subsystem is not established.

getWindowsOnAllDisplays

public SparseArray<List<AccessibilityWindowInfo>> getWindowsOnAllDisplays ()

Gets the windows on the screen of all displays. This method returns only the windows that a sighted user can interact with, as opposed to all windows. For example, if there is a modal dialog shown and the user cannot touch anything behind it, then only the modal window will be reported (assuming it is the top one). For convenience the returned windows are ordered in a descending layer order, which is the windows that are higher in the Z-order are reported first.

Note: In order to access the windows you have to opt-in to retrieve the interactive windows by setting the AccessibilityServiceInfo.FLAG_RETRIEVE_INTERACTIVE_WINDOWS flag.

Returns
SparseArray<List<AccessibilityWindowInfo>> The windows of all displays if there are windows and the service is can retrieve them, otherwise an empty list. The key of SparseArray is display ID.
This value cannot be null.
Throws
IllegalStateException If the connection to the accessibility subsystem is not established.

grantRuntimePermission

public void grantRuntimePermission (String packageName, 
                String permission)

Grants a runtime permission to a package.

Parameters
packageName String: The package to which to grant.
permission String: The permission to grant.
Throws
SecurityException if unable to grant the permission.

grantRuntimePermissionAsUser

public void grantRuntimePermissionAsUser (String packageName, 
                String permission, 
                UserHandle userHandle)

Grants a runtime permission to a package for a user.

Parameters
packageName String: The package to which to grant.
permission String: The permission to grant.
userHandle UserHandle
Throws
SecurityException if unable to grant the permission.

injectInputEvent

public boolean injectInputEvent (InputEvent event, 
                boolean sync)

A method for injecting an arbitrary input event. This method waits for all window container animations and surface operations to complete.

Note: It is caller's responsibility to recycle the event.

Parameters
event InputEvent: the event to inject
sync boolean: whether to inject the event synchronously
Returns
boolean true if event injection succeeded

revokeRuntimePermission

public void revokeRuntimePermission (String packageName, 
                String permission)

Revokes a runtime permission from a package.

Parameters
packageName String: The package to which to grant.
permission String: The permission to grant.
Throws
SecurityException if unable to revoke the permission.

revokeRuntimePermissionAsUser

public void revokeRuntimePermissionAsUser (String packageName, 
                String permission, 
                UserHandle userHandle)

Revokes a runtime permission from a package.

Parameters
packageName String: The package to which to grant.
permission String: The permission to grant.
userHandle UserHandle
Throws
SecurityException if unable to revoke the permission.

setOnAccessibilityEventListener

public void setOnAccessibilityEventListener (UiAutomation.OnAccessibilityEventListener listener)

Sets a callback for observing the stream of AccessibilityEvents. The callbacks are delivered on the main application thread.

Parameters
listener UiAutomation.OnAccessibilityEventListener: The callback.
Throws
IllegalStateException If the connection to the accessibility subsystem is not established.

setRunAsMonkey

public void setRunAsMonkey (boolean enable)

Sets whether this UiAutomation to run in a "monkey" mode. Applications can query whether they are executed in a "monkey" mode, i.e. run by a test framework, and avoid doing potentially undesirable actions such as calling 911 or posting on public forums etc.

Parameters
enable boolean: whether to run in a "monkey" mode or not. Default is not.

setServiceInfo

public void setServiceInfo (AccessibilityServiceInfo info)

Sets the AccessibilityServiceInfo that describes how this UiAutomation will be handled by the platform accessibility layer.

Parameters
info AccessibilityServiceInfo: The info.
Throws
IllegalStateException If the connection to the accessibility subsystem is not established.

takeScreenshot

public Bitmap takeScreenshot ()

Takes a screenshot.

Returns
Bitmap The screenshot bitmap on success, null otherwise.

takeScreenshot

public Bitmap takeScreenshot (Window window)

Used to capture a screenshot of a Window. This can return null in the following cases: 1. Window content hasn't been layed out. 2. Window doesn't have a valid SurfaceControl 3. An error occurred in SurfaceFlinger when trying to take the screenshot.

Parameters
window Window: Window to take a screenshot of.
This value cannot be null.
Returns
Bitmap The screenshot bitmap on success, null otherwise.

toString

public String toString ()

Returns a string representation of the object.

Returns
String a string representation of the object.

waitForIdle

public void waitForIdle (long idleTimeoutMillis, 
                long globalTimeoutMillis)

Waits for the accessibility event stream to become idle, which is not to have received an accessibility event within idleTimeoutMillis. The total time spent to wait for an idle accessibility event stream is bounded by the globalTimeoutMillis.

Parameters
idleTimeoutMillis long: The timeout in milliseconds between two events to consider the device idle.
globalTimeoutMillis long: The maximal global timeout in milliseconds in which to wait for an idle state.
Throws
IllegalStateException If the connection to the accessibility subsystem is not established.
TimeoutException If no idle state was detected within globalTimeoutMillis.