Developer Information


Developer Guide

Writing Plugins

Workload Automation offers several plugin points (or plugin types). The most interesting of these are

workloads

These are the tasks that get executed and measured on the device. These can be benchmarks, high-level use cases, or pretty much anything else.

targets

These are interfaces to the physical devices (development boards or end-user devices, such as smartphones) that use cases run on. Typically each model of a physical device would require its own interface class (though some functionality may be reused by subclassing from an existing base).

instruments

Instruments allow collecting additional data from workload execution (e.g. system traces). Instruments are not specific to a particular workload. Instruments can hook into any stage of workload execution.

output processors

These are used to format the results of workload execution once they have been collected. Depending on the callback used, these will run either after each iteration and/or at the end of the run, after all of the results have been collected.

You can create a plugin by subclassing the appropriate base class, defining appropriate methods and attributes, and putting the .py file containing the class into the “plugins” subdirectory under ~/.workload_automation (or equivalent) where it will be automatically picked up by WA.

Plugin Basics

This sub-section covers things common to implementing plugins of all types. It is recommended you familiarize yourself with the information here before proceeding onto guidance for specific plugin types.

Dynamic Resource Resolution

The idea is to decouple resource identification from resource discovery. Workloads/instruments/devices/etc state what resources they need, and not where to look for them – this instead is left to the resource resolver that is part of the execution context. The actual discovery of resources is performed by resource getters that are registered with the resolver.

A resource type is defined by a subclass of wa.framework.resource.Resource. An instance of this class describes a resource that is to be obtained. At minimum, a Resource instance has an owner (which is typically the object that is looking for the resource), but specific resource types may define other parameters that describe an instance of that resource (such as file names, URLs, etc).

An object looking for a resource invokes a resource resolver with an instance of Resource describing the resource it is after. The resolver goes through the getters registered for that resource type in priority order attempting to obtain the resource; once the resource is obtained, it is returned to the calling object. If none of the registered getters could find the resource, NotFoundError is raised (or None is returned instead, if invoked with strict=False).

The most common kind of object looking for resources is a Workload, and the Workload class defines wa.framework.workload.Workload.init_resources() method, which may be overridden by subclasses to perform resource resolution. For example, a workload looking for an executable file would do so like this:

from wa import Workload
from wa.import Executable

class MyBenchmark(Workload):

    # ...

    def init_resources(self, resolver):
        resource = Executable(self, self.target.abi, 'my_benchmark')
        host_exe = resolver.get(resource)

    # ...

Currently available resource types are defined in wa.framework.resources.

Deploying executables to a target

Some targets may have certain restrictions on where executable binaries may be placed and how they should be invoked. To ensure your plugin works with as wide a range of targets as possible, you should use WA APIs for deploying and invoking executables on a target, as outlined below.

As with other resources, host-side paths to the executable binary to be deployed should be obtained via the resource resolver. A special resource type, Executable is used to identify a binary to be deployed. This is similar to the regular File resource, however it takes an additional parameter that specifies the ABI for which the executable was compiled for.

In order for the binary to be obtained in this way, it must be stored in one of the locations scanned by the resource resolver in a directory structure <root>/bin/<abi>/<binary> (where root is the base resource location to be searched, e.g. ~/.workload_automation/dependencies/<plugin name>, and <abi> is the ABI for which the executable has been compiled, as returned by self.target.abi).

Once the path to the host-side binary has been obtained, it may be deployed using one of two methods from a Target instance – install or install_if_needed. The latter will check a version of that binary has been previously deployed by WA and will not try to re-install.

from wa import Executable

host_binary = context.get(Executable(self, self.target.abi, 'some_binary'))
target_binary = self.target.install_if_needed(host_binary)

Note

Please also note that the check is done based solely on the binary name. For more information please see the devlib documentation.

Both of the above methods will return the path to the installed binary on the target. The executable should be invoked only via that path; do not assume that it will be in PATH on the target (or that the executable with the same name in PATH is the version deployed by WA.

For more information on how to implement this, please see the how to guide.

Deploying assets

WA provides a generic mechanism for deploying assets during workload initialization. WA will automatically try to retrieve and deploy each asset to the target’s working directory that is contained in a workloads deployable_assets attribute stored as a list.

If the parameter cleanup_assets is set then any asset deployed will be removed again and the end of the run.

If the workload requires a custom deployment mechanism the deploy_assets method can be overridden for that particular workload, in which case, either additional assets should have their on target paths added to the workload’s deployed_assests attribute or the corresponding remove_assets method should also be implemented.

Adding an Instrument

Instruments can be used to collect additional measurements during workload execution (e.g. collect power readings). An instrument can hook into almost any stage of workload execution. Any new instrument should be a subclass of Instrument and it must have a name. When a new instrument is added to Workload Automation, the methods of the new instrument will be found automatically and hooked up to the supported signals. Once a signal is broadcasted, the corresponding registered method is invoked.

Each method in Instrument must take two arguments, which are self and context. Supported methods and their corresponding signals can be found in the Signals Documentation. To make implementations easier and common, the basic steps to add new instrument is similar to the steps to add new workload and an example can be found in the How To section.

To implement your own instrument the relevant methods of the interface shown below should be implemented:

name

The name of the instrument, this must be unique to WA.

description

A description of what the instrument can be used for.

parameters

A list of additional Parameters the instrument can take.

initialize(context)

This method will only be called once during the workload run therefore operations that only need to be performed initially should be performed here for example pushing the files to the target device, installing them.

setup(context)

This method is invoked after the workload is setup. All the necessary setup should go inside this method. Setup, includes operations like clearing logs, additional configuration etc.

start(context)

It is invoked just before the workload start execution. Here is where instrument measurement start being registered/taken.

stop(context)

It is invoked just after the workload execution stops and where the measurements should stop being taken/registered.

update_output(context)

This method is invoked after the workload updated its result and where the taken measures should be added to the result so it can be processed by WA.

teardown(context)

It is invoked after the workload is torn down. It is a good place to clean any logs generated by the instrument.

finalize(context)

This method is the complement to the initialize method and will also only be called once so should be used to deleting/uninstalling files pushed to the device.

This is similar to a Workload, except all methods are optional. In addition to the workload-like methods, instruments can define a number of other methods that will get invoked at various points during run execution. The most useful of which is perhaps initialize that gets invoked after the device has been initialised for the first time, and can be used to perform one-time setup (e.g. copying files to the device – there is no point in doing that for each iteration). The full list of available methods can be found in Signals Documentation.

Prioritization

Callbacks (e.g. setup() methods) for all instruments get executed at the same point during workload execution, one after another. The order in which the callbacks get invoked should be considered arbitrary and should not be relied on (e.g. you cannot expect that just because instrument A is listed before instrument B in the config, instrument A’s callbacks will run first).

In some cases (e.g. in start() and stop() methods), it is important to ensure that a particular instrument’s callbacks run a closely as possible to the workload’s invocations in order to maintain accuracy of readings; or, conversely, that a callback is executed after the others, because it takes a long time and may throw off the accuracy of other instruments. You can do this by using decorators on the appropriate methods. The available decorators are: very_slow, slow, normal, fast, very_fast, with very_fast running closest to the workload invocation and very_slow running furtherest away. For example:

from wa import very_fast
# ..

class PreciseInstrument(Instrument)

    # ...
    @very_fast
    def start(self, context):
        pass

    @very_fast
    def stop(self, context):
        pass

    # ...

PreciseInstrument will be started after all other instruments (i.e. just before the workload runs), and it will stopped before all other instruments (i.e. just after the workload runs).

If more than one active instrument has specified fast (or slow) callbacks, then their execution order with respect to each other is not guaranteed. In general, having a lot of instruments enabled is going to negatively affect the readings. The best way to ensure accuracy of measurements is to minimize the number of active instruments (perhaps doing several identical runs with different instruments enabled).

Example

Below is a simple instrument that measures the execution time of a workload:

class ExecutionTimeInstrument(Instrument):
    """
    Measure how long it took to execute the run() methods of a Workload.

    """

    name = 'execution_time'

    def initialize(self, context):
        self.start_time = None
        self.end_time = None

    @very_fast
    def start(self, context):
        self.start_time = time.time()

    @very_fast
    def stop(self, context):
        self.end_time = time.time()

    def update_output(self, context):
        execution_time = self.end_time - self.start_time
        context.add_metric('execution_time', execution_time, 'seconds')
Instrumentation Signal-Method Mapping

Instrument methods get automatically hooked up to signals based on their names. Mostly, the method name corresponds to the name of the signal, however there are a few convenience aliases defined (listed first) to make easier to relate instrumentation code to the workload execution model. For an overview on when these signals are dispatched during execution please see the Developer Reference.

method name

signal

initialize

run-initialized

setup

before-workload-setup

start

before-workload-execution

stop

after-workload-execution

process_workload_output

successful-workload-output-update

update_output

after-workload-output-update

teardown

after-workload-teardown

finalize

run-finalized

on_run_start

run-started

on_run_end

run-completed

on_job_start

job-started

on_job_restart

job-restarted

on_job_end

job-completed

on_job_failure

job-failed

on_job_abort

job-aborted

before_job_queue_execution

before-job-queue-execution

on_successful_job_queue_exection

successful-job-queue-execution

after_job_queue_execution

after-job-queue-execution

before_job

before-job

on_successful_job

successful-job

after_job

after-job

before_processing_job_output

before-job-output-processed

on_successfully_processing_job

successful-job-output-processed

after_processing_job_output

after-job-output-processed

before_reboot

before-reboot

on_successful_reboot

successful-reboot

after_reboot

after-reboot

on_error

error-logged

on_warning

warning-logged

The methods above may be decorated with on the listed decorators to set the priority (a value in the wa.framework.signal.CallbackPriority enum) of the Instrument method relative to other callbacks registered for the signal (within the same priority level, callbacks are invoked in the order they were registered). The table below shows the mapping of the decorator to the corresponding priority name and level:

decorator

CallbackPriority name

CallbackPriority value

extremely_slow

extremely_low

-30

very_slow

very_low

-20

slow

low

-10

normal

normal

0

fast

high

10

very_fast

very_high

20

extremely_fast

extremely_high

30

Unresponsive Targets

If a target is believed to be unresponsive, instrument callbacks will be disabled to prevent a cascade of errors and potential corruptions of state, as it is generally assumed that instrument callbacks will want to do something with the target.

If your callback only does something with the host, and does not require an active target connection, you can decorate it with @hostside decorator to ensure it gets invoked even if the target becomes unresponsive.

Adding an Output processor

A output processor is responsible for processing the results. This may involve formatting and writing them to a file, uploading them to a database, generating plots, etc. WA comes with a few output processors that output results in a few common formats (such as csv or JSON).

You can add your own output processors by creating a Python file in ~/.workload_automation/plugins with a class that derives from wa.OutputProcessor, and should implement the relevant methods shown below, for more information and please see the Adding an Output Processor section.

name

The name of the output processor, this must be unique to WA.

description

A description of what the output processor can be used for.

parameters

A list of additional Parameters the output processor can take.

initialize(context)

This method will only be called once during the workload run therefore operations that only need to be performed initially should be performed here.

process_job_output(output, target_info, run_ouput)

This method should be used to perform the processing of the output from an individual job output. This is where any additional artifacts should be generated if applicable.

export_job_output(output, target_info, run_ouput)

This method should be used to perform the exportation of the existing data collected/generated for an individual job. E.g. uploading them to a database etc.

process_run_output(output, target_info)

This method should be used to perform the processing of the output from the run as a whole. This is where any additional artifacts should be generated if applicable.

export_run_output(output, target_info)

This method should be used to perform the exportation of the existing data collected/generated for the run as a whole. E.g. uploading them to a database etc.

finalize(context)

This method is the complement to the initialize method and will also only be called once.

The method names should be fairly self-explanatory. The difference between “process” and “export” methods is that export methods will be invoked after process methods for all output processors have been generated. Process methods may generate additional artifacts (metrics, files, etc.), while export methods should not – they should only handle existing results (upload them to a database, archive on a filer, etc).

The output object passed to job methods is an instance of wa.framework.output.JobOutput, the output object passed to run methods is an instance of wa.RunOutput.

Adding a Resource Getter

A resource getter is a plugin that is designed to retrieve a resource (binaries, APK files or additional workload assets). Resource getters are invoked in priority order until one returns the desired resource.

If you want WA to look for resources somewhere it doesn’t by default (e.g. you have a repository of APK files), you can implement a getter for the resource and register it with a higher priority than the standard WA getters, so that it gets invoked first.

Instances of a resource getter should implement the following interface:

class ResourceGetter(Plugin):

    name = None

    def register(self, resolver):
        raise NotImplementedError()

The getter should define a name for itself (as with all plugins), in addition it should implement the register method. This involves registering a method with the resolver that should used to be called when trying to retrieve a resource (typically get) along with it’s priority (see Getter Prioritization below. That method should return an instance of the resource that has been discovered (what “instance” means depends on the resource, e.g. it could be a file path), or None if this getter was unable to discover that resource.

Getter Prioritization

A priority is an integer with higher numeric values indicating a higher priority. The following standard priority aliases are defined for getters:

preferred

Take this resource in favour of the environment resource.

local

Found somewhere under ~/.workload_automation/ or equivalent, or from environment variables, external configuration files, etc. These will override resource supplied with the package.

lan

Resource will be retrieved from a locally mounted remote location (such as samba share)

remote

Resource will be downloaded from a remote location (such as an HTTP server)

package

Resource provided with the package.

These priorities are defined as class members of wa.framework.resource.SourcePriority, e.g. SourcePriority.preferred.

Most getters in WA will be registered with either local or package priorities. So if you want your getter to override the default, it should typically be registered as preferred.

You don’t have to stick to standard priority levels (though you should, unless there is a good reason). Any integer is a valid priority. The standard priorities range from 0 to 40 in increments of 10.

Example

The following is an implementation of a getter that searches for files in the users dependencies directory, typically ~/.workload_automation/dependencies/<workload_name> It uses the get_from_location method to filter the available files in the provided directory appropriately:

import sys

from wa import settings,
from wa.framework.resource import ResourceGetter, SourcePriority
from wa.framework.getters import get_from_location
from wa.utils.misc import ensure_directory_exists as _d

class UserDirectory(ResourceGetter):

    name = 'user'

    def register(self, resolver):
        resolver.register(self.get, SourcePriority.local)

    def get(self, resource):
        basepath = settings.dependencies_directory
        directory = _d(os.path.join(basepath, resource.owner.name))
        return get_from_location(directory, resource)

Adding a Target

In WA3, a ‘target’ consists of a platform and a devlib target. The implementations of the targets are located in devlib. WA3 will instantiate a devlib target passing relevant parameters parsed from the configuration. For more information about devlib targets please see the documentation.

The currently available platforms are:
generic

The ‘standard’ platform implementation of the target, this should work for the majority of use cases.

juno

A platform implementation specifically for the juno.

tc2

A platform implementation specifically for the tc2.

gem5

A platform implementation to interact with a gem5 simulation.

The currently available targets from devlib are:
linux

A device running a Linux based OS.

android

A device running Android OS.

local

Used to run locally on a linux based host.

chromeos

A device running ChromeOS, supporting an android container if available.

For an example of adding you own customized version of an existing devlib target, please see the how to section Adding a Custom Target.

Other Plugin Types

In addition to plugin types covered above, there are few other, more specialized ones. They will not be covered in as much detail. Most of them expose relatively simple interfaces with only a couple of methods and it is expected that if the need arises to extend them, the API-level documentation that accompanies them, in addition to what has been outlined here, should provide enough guidance.

commands

This allows extending WA with additional sub-commands (to supplement exiting ones outlined in the Commands section).

modules

Modules are “plugins for plugins”. They can be loaded by other plugins to expand their functionality (for example, a flashing module maybe loaded by a device in order to support flashing).

Packaging Your Plugins

If your have written a bunch of plugins, and you want to make it easy to deploy them to new systems and/or to update them on existing systems, you can wrap them in a Python package. You can use wa create package command to generate appropriate boiler plate. This will create a setup.py and a directory for your package that you can place your plugins into.

For example, if you have a workload inside my_workload.py and an output processor in my_output_processor.py, and you want to package them as my_wa_exts package, first run the create command

wa create package my_wa_exts

This will create a my_wa_exts directory which contains a my_wa_exts/setup.py and a subdirectory my_wa_exts/my_wa_exts which is the package directory for your plugins (you can rename the top-level my_wa_exts directory to anything you like – it’s just a “container” for the setup.py and the package directory). Once you have that, you can then copy your plugins into the package directory, creating my_wa_exts/my_wa_exts/my_workload.py and my_wa_exts/my_wa_exts/my_output_processor.py. If you have a lot of plugins, you might want to organize them into subpackages, but only the top-level package directory is created by default, and it is OK to have everything in there.

Note

When discovering plugins through this mechanism, WA traverses the Python module/submodule tree, not the directory structure, therefore, if you are going to create subdirectories under the top level directory created for you, it is important that your make sure they are valid Python packages; i.e. each subdirectory must contain a __init__.py (even if blank) in order for the code in that directory and its subdirectories to be discoverable.

At this stage, you may want to edit params structure near the bottom of the setup.py to add correct author, license and contact information (see “Writing the Setup Script” section in standard Python documentation for details). You may also want to add a README and/or a COPYING file at the same level as the setup.py. Once you have the contents of your package sorted, you can generate the package by running

cd my_wa_exts
python setup.py sdist

This will generate my_wa_exts/dist/my_wa_exts-0.0.1.tar.gz package which can then be deployed on the target system with standard Python package management tools, e.g.

sudo pip install my_wa_exts-0.0.1.tar.gz

As part of the installation process, the setup.py in the package, will write the package’s name into ~/.workoad_automation/packages. This will tell WA that the package contains plugin and it will load them next time it runs.

Note

There are no uninstall hooks in setuputils, so if you ever uninstall your WA plugins package, you will have to manually remove it from ~/.workload_automation/packages otherwise WA will complain about a missing package next time you try to run it.


How Tos

Deploying Executables

Installing binaries for a particular plugin should generally only be performed once during a run. This should typically be done in the initialize method, if the only functionality performed in the method is to install the required binaries then the initialize method should be decorated with the @once decorator otherwise this should be placed into a dedicated method which is decorated instead. Please note if doing this then any installed paths should be added as class attributes rather than instance variables. As a general rule if binaries are installed as part of initialize then they should be uninstalled in the complementary finalize method.

Part of an example workload demonstrating this is shown below:

class MyWorkload(Workload):
      #..
      @once
      def initialize(self, context):
          resource = Executable(self, self.target.abi, 'my_executable')
          host_binary = context.resolver.get(resource)
          MyWorkload.target_binary = self.target.install(host_binary)
      #..

      def setup(self, context):
          self.command = "{} -a -b -c".format(self.target_binary)
          self.target.execute(self.command)
      #..

      @once
      def finalize(self, context):
          self.target.uninstall('my_executable')

Adding a Workload

The easiest way to create a new workload is to use the create command. wa create workload <args>. This will use predefined templates to create a workload based on the options that are supplied to be used as a starting point for the workload. For more information on using the create workload command see wa create workload -h

The first thing to decide is the type of workload you want to create depending on the OS you will be using and the aim of the workload. The are currently 6 available workload types to choose as detailed in the Developer Reference.

Once you have decided what type of workload you wish to choose this can be specified with -k <workload_kind> followed by the workload name. This will automatically generate a workload in the your WA_CONFIG_DIR/plugins. If you wish to specify a custom location this can be provided with -p <path>

A typical invocation of the create command would be in the form:

wa create workload -k <workload_kind> <workload_name>

Adding a Basic Workload

To add a basic workload template for our example workload we can simply use the command:

wa create workload -k basic ziptest

This will generate a very basic workload with dummy methods for the each method in the workload interface and it is left to the developer to add any required functionality.

Not all the methods from the interface are required to be implemented, this example shows how a subset might be used to implement a simple workload that times how long it takes to compress a file of a particular size on the device.

Note

This is intended as an example of how to implement the Workload interface. The methodology used to perform the actual measurement is not necessarily sound, and this Workload should not be used to collect real measurements.

The first step is to subclass our desired workload type depending on the purpose of our workload, in this example we are implementing a very simple workload and do not require any additional feature so shall inherit directly from the the base Workload class. We then need to provide a name for our workload which is what will be used to identify your workload for example in an agenda or via the show command, if you used the create command this will already be populated for you.

import os
from wa import Workload, Parameter

class ZipTest(Workload):

    name = 'ziptest'

The description attribute should be a string in the structure of a short summary of the purpose of the workload, and will be shown when using the list command, followed by a more in- depth explanation separated by a new line.

description = '''
              Times how long it takes to gzip a file of a particular size on a device.

              This workload was created for illustration purposes only. It should not be
              used to collect actual measurements.
              '''

In order to allow for additional configuration of the workload from a user a list of parameters can be supplied. These can be configured in a variety of different ways. For example here we are ensuring that the value of the parameter is an integer and larger than 0 using the kind and constraint options, also if no value is provided we are providing a default value of 2000000. These parameters will automatically have their value set as an attribute of the workload so later on we will be able to use the value provided here as self.file_size.

parameters = [
        Parameter('file_size', kind=int, default=2000000,
                  constraint=lambda x: 0 < x,
                  description='Size of the file (in bytes) to be gzipped.')
]

Next we will implement our setup method. This is where we do any preparation that is required before the workload is ran, this is usually things like setting up required files on the device and generating commands from user input. In this case we will generate our input file on the host system and then push it to a known location on the target for use in the ‘run’ stage.

def setup(self, context):
    super(ZipTestWorkload, self).setup(context)
    # Generate a file of the specified size containing random garbage.
    host_infile = os.path.join(context.output_directory, 'infile')
    command = 'openssl rand -base64 {} > {}'.format(self.file_size, host_infile)
    os.system(command)
    # Set up on-device paths
    devpath = self.target.path  # os.path equivalent for the target
    self.target_infile = devpath.join(self.target.working_directory, 'infile')
    self.target_outfile = devpath.join(self.target.working_directory, 'outfile')
    # Push the file to the target
    self.target.push(host_infile, self.target_infile)

The run method is where the actual ‘work’ of the workload takes place and is what is measured by any instrumentation. So for this example this is the execution of creating the zip file on the target.

def run(self, context):
    cmd = 'cd {} && (time gzip {}) &>> {}'
    self.target.execute(cmd.format(self.target.working_directory,
                                   self.target_infile,
                                   self.target_outfile))

The extract_results method is used to extract any results from the target for example we want to pull the file containing the timing information that we will use to generate metrics for our workload and then we add this file as an artifact with a ‘raw’ kind, which means once WA has finished processing it will allow it to decide whether to keep the file or not.

def extract_results(self, context):
    super(ZipTestWorkload, self).extract_results(context)
    # Pull the results file to the host
    self.host_outfile = os.path.join(context.output_directory, 'timing_results')
    self.target.pull(self.target_outfile, self.host_outfile)
    context.add_artifact('ziptest-results', self.host_outfile, kind='raw')

The update_output method we can do any generation of metrics that we wish to for our workload. In this case we are going to simply convert the times reported into seconds and add them as ‘metrics’ to WA which can then be displayed to the user along with any others in a format dependant on which output processors they have enabled for the run.

def update_output(self, context):
    super(ZipTestWorkload, self).update_output(context)
    # Extract metrics form the file's contents and update the result
    # with them.
    content = iter(open(self.host_outfile).read().strip().split())
    for value, metric in zip(content, content):
        mins, secs = map(float, value[:-1].split('m'))
        context.add_metric(metric, secs + 60 * mins, 'seconds')

Finally in the teardown method we will perform any required clean up for the workload so we will delete the input and output files from the device.

def teardown(self, context):
    super(ZipTestWorkload, self).teardown(context)
    self.target.remove(self.target_infile)
    self.target.remove(self.target_outfile)

The full implementation of this workload would look something like:

import os
from wa import Workload, Parameter

class ZipTestWorkload(Workload):

    name = 'ziptest'

    description = '''
                  Times how long it takes to gzip a file of a particular size on a device.

                  This workload was created for illustration purposes only. It should not be
                  used to collect actual measurements.
                  '''

    parameters = [
            Parameter('file_size', kind=int, default=2000000,
                      constraint=lambda x: 0 < x,
                      description='Size of the file (in bytes) to be gzipped.')
    ]

    def setup(self, context):
        super(ZipTestWorkload, self).setup(context)
        # Generate a file of the specified size containing random garbage.
        host_infile = os.path.join(context.output_directory, 'infile')
        command = 'openssl rand -base64 {} > {}'.format(self.file_size, host_infile)
        os.system(command)
        # Set up on-device paths
        devpath = self.target.path  # os.path equivalent for the target
        self.target_infile = devpath.join(self.target.working_directory, 'infile')
        self.target_outfile = devpath.join(self.target.working_directory, 'outfile')
        # Push the file to the target
        self.target.push(host_infile, self.target_infile)

    def run(self, context):
        cmd = 'cd {} && (time gzip {}) &>> {}'
        self.target.execute(cmd.format(self.target.working_directory,
                                       self.target_infile,
                                       self.target_outfile))
    def extract_results(self, context):
        super(ZipTestWorkload, self).extract_results(context)
        # Pull the results file to the host
        self.host_outfile = os.path.join(context.output_directory, 'timing_results')
        self.target.pull(self.target_outfile, self.host_outfile)
        context.add_artifact('ziptest-results', self.host_outfile, kind='raw')

    def update_output(self, context):
        super(ZipTestWorkload, self).update_output(context)
        # Extract metrics form the file's contents and update the result
        # with them.
        content = iter(open(self.host_outfile).read().strip().split())
        for value, metric in zip(content, content):
            mins, secs = map(float, value[:-1].split('m'))
            context.add_metric(metric, secs + 60 * mins, 'seconds')

    def teardown(self, context):
        super(ZipTestWorkload, self).teardown(context)
        self.target.remove(self.target_infile)
        self.target.remove(self.target_outfile)

Adding a ApkUiAutomator Workload

If we wish to create a workload to automate the testing of the Google Docs android app, we would choose to perform the automation using UIAutomator and we would want to automatically deploy and install the apk file to the target, therefore we would choose the ApkUiAutomator workload type with the following command:

$ wa create workload -k apkuiauto google_docs
Workload created in $WA_USER_DIRECTORY/plugins/google_docs

From here you can navigate to the displayed directory and you will find your __init__.py and a uiauto directory. The former is your python WA workload and will look something like this. For an example of what should be done in each of the main method please see adding a basic example above.

from wa import Parameter, ApkUiautoWorkload
class GoogleDocs(ApkUiautoWorkload):
    name = 'google_docs'
    description = "This is an placeholder description"
    # Replace with a list of supported package names in the APK file(s).
    package_names = ['package_name']

    parameters = [
     # Workload parameters go here e.g.
     Parameter('example_parameter', kind=int, allowed_values=[1,2,3],
               default=1, override=True, mandatory=False,
               description='This is an example parameter')
    ]

    def __init__(self, target, **kwargs):
     super(GoogleDocs, self).__init__(target, **kwargs)
     # Define any additional attributes required for the workload

    def init_resources(self, resolver):
     super(GoogleDocs, self).init_resources(resolver)
     # This method may be used to perform early resource discovery and
     # initialization. This is invoked during the initial loading stage and
     # before the device is ready, so cannot be used for any device-dependent
     # initialization. This method is invoked before the workload instance is
     # validated.

    def initialize(self, context):
     super(GoogleDocs, self).initialize(context)
     # This method should be used to perform once-per-run initialization of a
     # workload instance.

    def validate(self):
     super(GoogleDocs, self).validate()
     # Validate inter-parameter assumptions etc

    def setup(self, context):
     super(GoogleDocs, self).setup(context)
     # Perform any necessary setup before starting the UI automation

    def extract_results(self, context):
     super(GoogleDocs, self).extract_results(context)
     # Extract results on the target

    def update_output(self, context):
     super(GoogleDocs, self).update_output(context)
     # Update the output within the specified execution context with the
     # metrics and artifacts form this workload iteration.

    def teardown(self, context):
     super(GoogleDocs, self).teardown(context)
     # Perform any final clean up for the Workload.

Depending on the purpose of your workload you can choose to implement which methods you require. The main things that need setting are the list of package_names which must be a list of strings containing the android package name that will be used during resource resolution to locate the relevant apk file for the workload. Additionally the the workload parameters will need to updating to any relevant parameters required by the workload as well as the description.

The latter will contain a framework for performing the UI automation on the target, the files you will be most interested in will be uiauto/app/src/main/java/arm/wa/uiauto/UiAutomation.java which will contain the actual code of the automation and will look something like:

package com.arm.wa.uiauto.google_docs;

import android.app.Activity;
import android.os.Bundle;
import org.junit.Test;
import org.junit.runner.RunWith;
import android.support.test.runner.AndroidJUnit4;

import android.util.Log;
import android.view.KeyEvent;

// Import the uiautomator libraries
import android.support.test.uiautomator.UiObject;
import android.support.test.uiautomator.UiObjectNotFoundException;
import android.support.test.uiautomator.UiScrollable;
import android.support.test.uiautomator.UiSelector;

import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;

import com.arm.wa.uiauto.BaseUiAutomation;

@RunWith(AndroidJUnit4.class)
public class UiAutomation extends BaseUiAutomation {

    protected Bundle parameters;
    protected int example_parameter;

    public static String TAG = "google_docs";

    @Before
    public void initilize() throws Exception {
        // Perform any parameter initialization here
        parameters = getParams(); // Required to decode passed parameters.
        packageID = getPackageID(parameters);
        example_parameter = parameters.getInt("example_parameter");
    }

    @Test
    public void setup() throws Exception {
        // Optional: Perform any setup required before the main workload
        // is ran, e.g. dismissing welcome screens
    }

    @Test
    public void runWorkload() throws Exception {
           // The main UI Automation code goes here
    }

    @Test
    public void extractResults() throws Exception {
        // Optional: Extract any relevant results from the workload,
    }

    @Test
    public void teardown() throws Exception {
        // Optional: Perform any clean up for the workload
    }
}
A few items to note from the template:
  • Each of the stages of execution for example setup, runWorkload etc are decorated with the @Test decorator, this is important to allow these methods to be called at the appropriate time however any additional methods you may add do not require this decorator.

  • The initialize method has the @Before decorator, this is there to ensure that this method is called before executing any of the workload stages and therefore is used to decode and initialize any parameters that are passed in.

  • The code currently retrieves the example_parameter that was provided to the python workload as an Integer, there are similar calls to retrieve parameters of different types e.g. getString, getBoolean, getDouble etc.

Once you have implemented your java workload you can use the file uiauto/build.sh to compile your automation into an apk file to perform the automation. The generated apk will be generated with the package name com.arm.wa.uiauto.<workload_name> which when running your workload will be automatically detected by the resource getters and deployed to the device.

Adding a ReventApk Workload

If we wish to create a workload to automate the testing of a UI based workload that we cannot / do not wish to use UiAutomator then we can perform the automation using revent. In this example we would want to automatically deploy and install an apk file to the target, therefore we would choose the ApkRevent workload type with the following command:

$ wa create workload -k apkrevent my_game
Workload created in $WA_USER_DIRECTORY/plugins/my_game

This will generate a revent based workload you will end up with a very similar python file as to the one outlined in generating a UiAutomator based workload however without the accompanying java automation files.

The main difference between the two is that this workload will subclass ApkReventWorkload instead of ApkUiautomatorWorkload as shown below.

from wa import ApkReventWorkload

class MyGame(ApkReventWorkload):

    name = 'mygame'
    package_names = ['com.mylogo.mygame']

    # ..

Adding an Instrument

This is an example of how we would create a instrument which will trace device errors using a custom “trace” binary file. For more detailed information please see the Instrument Reference. The first thing to do is to create a new file under $WA_USER_DIRECTORY/plugins/ and subclass Instrument. Make sure to overwrite the variable name with what we want our instrument to be called and then locate our binary for the instrument.

class TraceErrorsInstrument(Instrument):

    name = 'trace-errors'

    def __init__(self, target, **kwargs):
        super(TraceErrorsInstrument, self).__init__(target, **kwargs)
        self.binary_name = 'trace'
        self.binary_file = os.path.join(os.path.dirname(__file__), self.binary_name)
        self.trace_on_target = None

We then declare and implement the required methods as detailed in the Instrument API. For the initialize method, we want to install the executable file to the target so we can use the target’s install method which will try to copy the file to a location on the device that supports execution, change the file mode appropriately and return the file path on the target.

def initialize(self, context):
    self.trace_on_target = self.target.install(self.binary_file)

Then we implemented the start method, which will simply run the file to start tracing. Supposing that the call to this binary requires some overhead to begin collecting errors we might want to decorate the method with the @slow decorator to try and reduce the impact on other running instruments. For more information on prioritization please see the Developer Reference.

@slow
def start(self, context):
    self.target.execute('{} start'.format(self.trace_on_target))

Lastly, we need to stop tracing once the workload stops and this happens in the stop method, assuming stopping the collection also require some overhead we have again decorated the method.

@slow
def stop(self, context):
    self.target.execute('{} stop'.format(self.trace_on_target))

Once we have generated our result data we need to retrieve it from the device for further processing or adding directly to WA’s output for that job. For example for trace data we will want to pull it to the device and add it as a artifact to WA’s context. Once we have retrieved the data, we can now do any further processing and add any relevant Metrics to the context. For this we will use the the add_metric method to add the results to the final output for that workload. The method can be passed 4 params, which are the metric key, value, unit and lower_is_better.

def update_output(self, context):
    # pull the trace file from the target
    self.result = os.path.join(self.target.working_directory, 'trace.txt')
    self.outfile = os.path.join(context.output_directory, 'trace.txt')
    self.target.pull(self.result, self.outfile)
    context.add_artifact('error_trace', self.outfile, kind='export')

    # parse the file if needs to be parsed, or add result directly to
    # context.

    metric = # ..
    context.add_metric('number_of_errors', metric, lower_is_better=True

At the end of each job we might want to delete any files generated by the instruments and the code to clear these file goes in teardown method.

def teardown(self, context):
    self.target.remove(os.path.join(self.target.working_directory, 'trace.txt'))

At the very end of the run we would want to uninstall the binary we deployed earlier.

def finalize(self, context):
    self.target.uninstall(self.binary_name)

So the full example would look something like:

from wa import Instrument

class TraceErrorsInstrument(Instrument):

    name = 'trace-errors'

    def __init__(self, target, **kwargs):
        super(TraceErrorsInstrument, self).__init__(target, **kwargs)
        self.binary_name = 'trace'
        self.binary_file = os.path.join(os.path.dirname(__file__), self.binary_name)
        self.trace_on_target = None

    def initialize(self, context):
        self.trace_on_target = self.target.install(self.binary_file)

    @slow
    def start(self, context):
        self.target.execute('{} start'.format(self.trace_on_target))

    @slow
    def stop(self, context):
        self.target.execute('{} stop'.format(self.trace_on_target))

    def update_output(self, context):
        self.result = os.path.join(self.target.working_directory, 'trace.txt')
        self.outfile = os.path.join(context.output_directory, 'trace.txt')
        self.target.pull(self.result, self.outfile)
        context.add_artifact('error_trace', self.outfile, kind='export')

        metric = # ..
        context.add_metric('number_of_errors', metric, lower_is_better=True

    def teardown(self, context):
        self.target.remove(os.path.join(self.target.working_directory, 'trace.txt'))

    def finalize(self, context):
        self.target.uninstall(self.binary_name)

Adding an Output Processor

This is an example of how we would create an output processor which will format the run metrics as a column-aligned table. The first thing to do is to create a new file under $WA_USER_DIRECTORY/plugins/ and subclass OutputProcessor. Make sure to overwrite the variable name with what we want our processor to be called and provide a short description.

Next we need to implement any relevant methods, (please see adding an output processor for all the available methods). In this case we only want to implement the export_run_output method as we are not generating any new artifacts and we only care about the overall output rather than the individual job outputs. The implementation is very simple, it just loops through all the available metrics for all the available jobs and adds them to a list which is written to file and then added as an artifact to the context.

import os
from wa import OutputProcessor
from wa.utils.misc import write_table


class Table(OutputProcessor):

    name = 'table'
    description = 'Generates a text file containing a column-aligned table of run results.'

    def export_run_output(self, output, target_info):
        rows = []

        for job in output.jobs:
            for metric in job.metrics:
                rows.append([metric.name, str(metric.value), metric.units or '',
                             metric.lower_is_better  and '-' or '+'])

        outfile =  output.get_path('table.txt')
        with open(outfile, 'w') as wfh:
            write_table(rows, wfh)
        output.add_artifact('results_table', 'table.txt', 'export')

Adding a Custom Target

This is an example of how we would create a customised target, this is typically used where we would need to augment the existing functionality for example on development boards where we need to perform additional actions to implement some functionality. In this example we are going to assume that this particular device is running Android and requires a special “wakeup” command to be sent before it can execute any other command.

To add a new target to WA we will first create a new file in $WA_USER_DIRECTORY/plugins/example_target.py. In order to facilitate with creating a new target WA provides a helper function to create a description for the specified target class, and specified components. For components that are not explicitly specified it will attempt to guess sensible defaults based on the target class’ bases.

# Import our helper function
from wa import add_description_for_target

# Import the Target that our custom implementation will be based on
from devlib import AndroidTarget

class ExampleTarget(AndroidTarget):
    # Provide the name that will be used to identify your custom target
    name = 'example_target'

    # Override our custom method(s)
    def execute(self, *args, **kwargs):
        super(ExampleTarget, self).execute('wakeup', check_exit_code=False)
        return super(ExampleTarget, self).execute(*args, **kwargs)


description = '''An Android target which requires an explicit "wakeup" command
                  to be sent before accepting any other command'''
# Call the helper function with our newly created function and its description.
add_description_for_target(ExampleTarget, description)

Processing WA Output

This section will illustrate the use of WA’s output processing API by creating a simple ASCII report generator. To make things concrete, this how-to will be processing the output from running the following agenda:

sections:
    - runtime_params:
        frequency: min
      classifiers:
          frequency: min
    - runtime_params:
        frequency: max
      classifiers:
          frequency: max
workloads:
    - sysbench
    - deepbench

This runs two workloads under two different configurations each – once with CPU frequency fixed to max, and once with CPU frequency fixed to min. Classifiers are used to indicate the configuration in the output.

First, create the RunOutput object, which is the main interface for interacting with WA outputs. Or alternatively a RunDatabaseOutput if storing your results in a postgres database.

import sys

from wa import RunOutput

# Path to the output directory specified in the first argument
ro = RunOutput(sys.argv[1])

Run Info

Next, we’re going to print out an overall summary of the run.

from __future__ import print_function   # for Python 2 compat.

from wa.utils.misc import format_duration

print('-'*20)
print('Run ID:', ro.info.uuid)
print('Run status:', ro.status)
print('Run started at:', ro.info.start_time.isoformat())
print('Run completed at:', ro.info.end_time.isoformat())
print('Run duration:', format_duration(ro.info.duration))
print('Ran', len(ro.jobs), 'jobs')
print('-'*20)
print()

RunOutput.info is an instance of RunInfo which encapsulates Overall-run metadata, such as the duration.

Target Info

Next, some information about the device the results where collected on.

print('    Target Information     ')
print('    -------------------    ')
print('hostname:', ro.target_info.hostname)
if ro.target_info.os == 'android':
    print('Android ID:', ro.target_info.android_id)
else:
    print('host ID:', ro.target_info.hostid)
print('CPUs:', ', '.join(cpu.name for cpu in ro.target_info.cpus))
print()

print('OS:', ro.target_info.os)
print('ABI:', ro.target_info.abi)
print('rooted:', ro.target_info.is_rooted)
print('kernel version:', ro.target_info.kernel_version)
print('os version:')
for k, v in ro.target_info.os_version.items():
    print('\t', k+':', v)
print()
print('-'*27)
print()

RunOutput.target_info is an instance of TargetInfo that contains information collected from the target during the run.

Jobs Summary

Next, show a summary of executed jobs.

from wa.utils.misc import write_table

print('           Jobs            ')
print('           ----            ')
print()
rows = []
for job in ro.jobs:
    rows.append([job.id, job.label, job.iteration, job.status])
write_table(rows, sys.stdout, align='<<><',
        headers=['ID', 'LABEL', 'ITER.', 'STATUS'])
print()
print('-'*27)
print()

RunOutput.jobs is a list of JobOutput objects. These contain information about that particular job, including its execution status, and Metrics and Artifacts generated by the job.

Compare Metrics

Finally, collect metrics, sort them by the “frequency” classifier. Classifiers that are present in the metric but not its job have been added by the workload. For the purposes of this report, they will be used to augment the metric’s name.

from collections import defaultdict

print()
print('    Metrics Comparison     ')
print('    ------------------     ')
print()
scores = defaultdict(lambda: defaultdict(lambda: defaultdict()))
for job in ro.jobs:
    for metric in job.metrics:
        workload = job.label
        name = metric.name
        freq = job.classifiers['frequency']
        for cname, cval in sorted(metric.classifiers.items()):
            if cname not in job.classifiers:
                # was not propagated from the job, therefore was
                # added by the workload
                name += '/{}={}'.format(cname, cval)

        scores[workload][name][freq] = metric

Once the metrics have been sorted, generate the report showing the delta between the two configurations (indicated by the “frequency” classifier) and highlight any unexpected deltas (based on the lower_is_better attribute of the metric). (In practice, you will want to run multiple iterations of each configuration, calculate averages and standard deviations, and only highlight statically significant deltas.)

rows = []
for workload in sorted(scores.keys()):
    wldata = scores[workload]

    for name in sorted(wldata.keys()):
        min_score = wldata[name]['min'].value
        max_score = wldata[name]['max'].value
        delta =  max_score - min_score
        units = wldata[name]['min'].units or ''
        lib = wldata[name]['min'].lower_is_better

        warn = ''
        if (lib and delta > 0) or (not lib and delta < 0):
            warn = '!!!'

        rows.append([workload, name,
        '{:.3f}'.format(min_score), '{:.3f}'.format(max_score),
        '{:.3f}'.format(delta), units, warn])

    # separate workloads with a blank row
    rows.append(['', '', '', '', '', '', ''])


write_table(rows, sys.stdout, align='<<>>><<',
        headers=['WORKLOAD', 'METRIC', 'MIN.', 'MAX', 'DELTA', 'UNITS', ''])
print()
print('-'*27)

This concludes this how-to. For more information, please see output processing API documentation.

Complete Example

Below is the complete example code, and a report it generated for a sample run.

from __future__ import print_function   # for Python 2 compat.
import sys
from collections import defaultdict

from wa import RunOutput
from wa.utils.misc import format_duration, write_table



# Path to the output directory specified in the first argument
ro = RunOutput(sys.argv[1])

print('-'*27)
print('Run ID:', ro.info.uuid)
print('Run status:', ro.status)
print('Run started at:', ro.info.start_time.isoformat())
print('Run completed at:', ro.info.end_time.isoformat())
print('Run duration:', format_duration(ro.info.duration))
print('Ran', len(ro.jobs), 'jobs')
print('-'*27)
print()

print('    Target Information     ')
print('    -------------------    ')
print('hostname:', ro.target_info.hostname)
if ro.target_info.os == 'android':
    print('Android ID:', ro.target_info.android_id)
else:
    print('host ID:', ro.target_info.hostid)
print('CPUs:', ', '.join(cpu.name for cpu in ro.target_info.cpus))
print()

print('OS:', ro.target_info.os)
print('ABI:', ro.target_info.abi)
print('rooted:', ro.target_info.is_rooted)
print('kernel version:', ro.target_info.kernel_version)
print('OS version:')
for k, v in ro.target_info.os_version.items():
    print('\t', k+':', v)
print()
print('-'*27)
print()

print('           Jobs            ')
print('           ----            ')
print()
rows = []
for job in ro.jobs:
    rows.append([job.id, job.label, job.iteration, job.status])
write_table(rows, sys.stdout, align='<<><',
        headers=['ID', 'LABEL', 'ITER.', 'STATUS'])
print()
print('-'*27)

print()
print('    Metrics Comparison     ')
print('    ------------------     ')
print()
scores = defaultdict(lambda: defaultdict(lambda: defaultdict()))
for job in ro.jobs:
    for metric in job.metrics:
        workload = job.label
        name = metric.name
        freq = job.classifiers['frequency']
        for cname, cval in sorted(metric.classifiers.items()):
            if cname not in job.classifiers:
                # was not propagated from the job, therefore was
                # added by the workload
                name += '/{}={}'.format(cname, cval)

    scores[workload][name][freq] = metric

rows = []
for workload in sorted(scores.keys()):
    wldata = scores[workload]

    for name in sorted(wldata.keys()):
        min_score = wldata[name]['min'].value
        max_score = wldata[name]['max'].value
        delta =  max_score - min_score
        units = wldata[name]['min'].units or ''
        lib = wldata[name]['min'].lower_is_better

        warn = ''
        if (lib and delta > 0) or (not lib and delta < 0):
            warn = '!!!'

        rows.append([workload, name,
        '{:.3f}'.format(min_score), '{:.3f}'.format(max_score),
        '{:.3f}'.format(delta), units, warn])

    # separate workloads with a blank row
    rows.append(['', '', '', '', '', '', ''])


write_table(rows, sys.stdout, align='<<>>><<',
        headers=['WORKLOAD', 'METRIC', 'MIN.', 'MAX', 'DELTA', 'UNITS', ''])
print()
print('-'*27)

Sample output:

---------------------------
Run ID: 78aef931-cd4c-429b-ac9f-61f6893312e6
Run status: OK
Run started at: 2018-06-27T12:55:23.746941
Run completed at: 2018-06-27T13:04:51.067309
Run duration: 9 minutes 27 seconds
Ran 4 jobs
---------------------------

Target Information
-------------------
hostname: localhost
Android ID: b9d1d8b48cfba007
CPUs: A53, A53, A53, A53, A73, A73, A73, A73

OS: android
ABI: arm64
rooted: True
kernel version: 4.9.75-04208-g2c913991a83d-dirty 114 SMP PREEMPT Wed May 9 10:33:36 BST 2018
OS version:
        all_codenames: O
        base_os:
        codename: O
        incremental: eng.valsch.20170517.180115
        preview_sdk: 0
        release: O
        sdk: 25
        security_patch: 2017-04-05

---------------------------

        Jobs
        ----

ID     LABEL     ITER. STATUS
--     -----     ----- ------
s1-wk1 sysbench      1 OK
s1-wk2 deepbench     1 OK
s2-wk1 sysbench      1 OK
s2-wk2 deepbench     1 OK

---------------------------

Metrics Comparison
------------------

WORKLOAD  METRIC                                            MIN.       MAX    DELTA UNITS
--------  ------                                            ----       ---    ----- -----
deepbench GOPS/a_t=n/b_t=n/k=1024/m=128/n=1                0.699     0.696   -0.003         !!!
deepbench GOPS/a_t=n/b_t=n/k=1024/m=3072/n=1               0.471     0.715    0.244
deepbench GOPS/a_t=n/b_t=n/k=1024/m=3072/n=1500           23.514    36.432   12.918
deepbench GOPS/a_t=n/b_t=n/k=1216/m=64/n=1                 0.333     0.333   -0.000         !!!
deepbench GOPS/a_t=n/b_t=n/k=128/m=3072/n=1                0.405     1.073    0.668
deepbench GOPS/a_t=n/b_t=n/k=128/m=3072/n=1500            19.914    34.966   15.052
deepbench GOPS/a_t=n/b_t=n/k=128/m=4224/n=1                0.232     0.486    0.255
deepbench GOPS/a_t=n/b_t=n/k=1280/m=128/n=1500            20.721    31.654   10.933
deepbench GOPS/a_t=n/b_t=n/k=1408/m=128/n=1                0.701     0.702    0.001
deepbench GOPS/a_t=n/b_t=n/k=1408/m=176/n=1500            19.902    29.116    9.214
deepbench GOPS/a_t=n/b_t=n/k=176/m=4224/n=1500            26.030    39.550   13.519
deepbench GOPS/a_t=n/b_t=n/k=2048/m=35/n=700              10.884    23.615   12.731
deepbench GOPS/a_t=n/b_t=n/k=2048/m=5124/n=700            26.740    37.334   10.593
deepbench execution_time                                 318.758   220.629  -98.129 seconds !!!
deepbench time (msec)/a_t=n/b_t=n/k=1024/m=128/n=1         0.375     0.377    0.002         !!!
deepbench time (msec)/a_t=n/b_t=n/k=1024/m=3072/n=1       13.358     8.793   -4.565
deepbench time (msec)/a_t=n/b_t=n/k=1024/m=3072/n=1500   401.338   259.036 -142.302
deepbench time (msec)/a_t=n/b_t=n/k=1216/m=64/n=1          0.467     0.467    0.000         !!!
deepbench time (msec)/a_t=n/b_t=n/k=128/m=3072/n=1         1.943     0.733   -1.210
deepbench time (msec)/a_t=n/b_t=n/k=128/m=3072/n=1500     59.237    33.737  -25.500
deepbench time (msec)/a_t=n/b_t=n/k=128/m=4224/n=1         4.666     2.224   -2.442
deepbench time (msec)/a_t=n/b_t=n/k=1280/m=128/n=1500     23.721    15.528   -8.193
deepbench time (msec)/a_t=n/b_t=n/k=1408/m=128/n=1         0.514     0.513   -0.001
deepbench time (msec)/a_t=n/b_t=n/k=1408/m=176/n=1500     37.354    25.533  -11.821
deepbench time (msec)/a_t=n/b_t=n/k=176/m=4224/n=1500     85.679    56.391  -29.288
deepbench time (msec)/a_t=n/b_t=n/k=2048/m=35/n=700        9.220     4.249   -4.970
deepbench time (msec)/a_t=n/b_t=n/k=2048/m=5124/n=700    549.413   393.517 -155.896

sysbench  approx.  95 percentile                           3.800     1.450   -2.350 ms
sysbench  execution_time                                   1.790     1.437   -0.353 seconds !!!
sysbench  response time avg                                1.400     1.120   -0.280 ms
sysbench  response time max                               40.740    42.760    2.020 ms      !!!
sysbench  response time min                                0.710     0.710    0.000 ms
sysbench  thread fairness events avg                    1250.000  1250.000    0.000
sysbench  thread fairness events stddev                  772.650   213.040 -559.610
sysbench  thread fairness execution time avg               1.753     1.401   -0.352         !!!
sysbench  thread fairness execution time stddev            0.000     0.000    0.000
sysbench  total number of events                       10000.000 10000.000    0.000
sysbench  total time                                       1.761     1.409   -0.352 s


---------------------------

Developer Reference

Framework Overview

Execution Model

At the high level, the execution model looks as follows:

_images/WA_Execution.svg

After some initial setup, the framework initializes the device, loads and initialized instruments and output processors and begins executing jobs defined by the workload specs in the agenda. Each job executes in basic stages:

initialize

Perform any once-per-run initialization of a workload instance, i.e. binary resource resolution.

setup

Initial setup for the workload is performed. E.g. required assets are deployed to the devices, required services or applications are launched, etc. Run time configuration of the device for the workload is also performed at this time.

setup_rerun (apk based workloads only)

For some apk based workloads the application is required to be started twice. If the requires_rerun attribute of the workload is set to True then after the first setup method is called the application will be killed and then restarted. This method can then be used to perform any additional setup required.

run

This is when the workload actually runs. This is defined as the part of the workload that is to be measured. Exactly what happens at this stage depends entirely on the workload.

extract results

Extract any results that have been generated during the execution of the workload from the device and back to that target. Any files pulled from the devices should be added as artifacts to the run context.

update output

Perform any required parsing and processing of any collected results and add any generated metrics to the run context.

teardown

Final clean up is performed, e.g. applications may closed, files generated during execution deleted, etc.

Signals are dispatched (see below) at each stage of workload execution, which installed instruments can hook into in order to collect measurements, alter workload execution, etc. Instruments implementation usually mirrors that of workloads, defining initialization, setup, teardown and output processing stages for a particular instrument. Instead of a run method instruments usually implement start and stop methods instead which triggered just before and just after a workload run. However, the signal dispatch mechanism gives a high degree of flexibility to instruments allowing them to hook into almost any stage of a WA run (apart from the very early initialization).

Metrics and artifacts generated by workloads and instruments are accumulated by the framework and are then passed to active output processors. This happens after each individual workload execution and at the end of the run. A output processor may chose to act at either or both of these points.

Control Flow

This section goes into more detail explaining the relationship between the major components of the framework and how control passes between them during a run. It will only go through the major transitions and interactions and will not attempt to describe every single thing that happens.

Note

This is the control flow for the wa run command which is the main functionality of WA. Other commands are much simpler and most of what is described below does not apply to them.

  1. wa.framework.entrypoint parses the command from the arguments, creates a wa.framework.configuration.execution.ConfigManager and executes the run command (wa.commands.run.RunCommand) passing it the ConfigManger.

  2. Run command initializes the output directory and creates a wa.framework.configuration.parsers.AgendaParser and will parser an agenda and populate the ConfigManger based on the command line arguments. Finally it instantiates a wa.framework.execution.Executor and passes it the completed ConfigManager.

  3. The Executor uses the ConfigManager to create a wa.framework.configuration.core.RunConfiguration and fully defines the configuration for the run (which will be serialised into __meta subdirectory under the output directory).

  4. The Executor proceeds to instantiate a TargetManager, used to handle the device connection and configuration, and a wa.framework.execution.ExecutionContext which is used to track the current state of the run execution and also serves as a means of communication between the core framework and plugins. After this any required instruments and output processors are initialized and installed.

  5. Finally, the Executor instantiates a wa.framework.execution.Runner, initializes its job queue with workload specs from the RunConfiguration, and kicks it off.

  6. The Runner performs the run time configuration of the device and goes through the workload specs (in the order defined by execution_order setting), running each spec according to the execution model described in the previous section and sending signals (see below) at appropriate points during execution.

  7. At the end of the run, the control is briefly passed back to the Executor, which outputs a summary for the run.

Signal Dispatch

WA uses the louie (formerly, pydispatcher) library for signal dispatch. Callbacks can be registered for signals emitted during the run. WA uses a version of louie that has been modified to introduce priority to registered callbacks (so that callbacks that are know to be slow can be registered with a lower priority and therefore do not interfere with other callbacks).

This mechanism is abstracted for instruments. Methods of an wa.framework.Instrument subclass automatically get hooked to appropriate signals based on their names when the instrument is “installed” for the run. Priority can then be specified by adding extremely_fast, very_fast, fast , slow, very_slow or extremely_slow decorators to the method definitions.

The full list of method names and the signals they map to may be seen at the instrument method map.

Signal dispatching mechanism may also be used directly, for example to dynamically register callbacks at runtime or allow plugins other than Instruments to access stages of the run they are normally not aware of.

Signals can be either paired or non paired signals. Non paired signals are one off signals that are sent to indicate special events or transitions in execution stages have occurred for example TARGET_CONNECTED. Paired signals are used to signify the start and end of a particular event. If the start signal has been sent the end signal is guaranteed to also be sent, whether the operation was a successes or not, however in the case of correct operation an additional success signal will also be sent. For example in the event of a successful reboot of the the device, the following signals will be sent BEFORE_REBOOT, SUCCESSFUL_REBOOT and AFTER_REBOOT.

An overview of what signals are sent at which point during execution can be seen below. Most of the paired signals have been removed from the diagram for clarity and shown as being dispatched from a particular stage of execution, however in reality these signals will be sent just before and just after these stages are executed. As mentioned above for each of these signals there will be at least 2 and up to 3 signals sent. If the “BEFORE_X” signal (sent just before the stage is ran) is sent then the “AFTER_X” (sent just after the stage is ran) signal is guaranteed to also be sent, and under normal operation a “SUCCESSFUL_X” signal is also sent just after stage has been completed. The diagram also lists the conditional signals that can be sent at any time during execution if something unexpected happens, for example an error occurs or the user aborts the run.

_images/WA_Signal_Dispatch.svg

For more information see Instrumentation Signal-Method Mapping.


Plugins

Workload Automation offers several plugin points (or plugin types). The most interesting of these are

workloads

These are the tasks that get executed and measured on the device. These can be benchmarks, high-level use cases, or pretty much anything else.

targets

These are interfaces to the physical devices (development boards or end-user devices, such as smartphones) that use cases run on. Typically each model of a physical device would require its own interface class (though some functionality may be reused by subclassing from an existing base).

instruments

Instruments allow collecting additional data from workload execution (e.g. system traces). Instruments are not specific to a particular workload. Instruments can hook into any stage of workload execution.

output processors

These are used to format the results of workload execution once they have been collected. Depending on the callback used, these will run either after each iteration and/or at the end of the run, after all of the results have been collected.

You can create a plugin by subclassing the appropriate base class, defining appropriate methods and attributes, and putting the .py file containing the class into the “plugins” subdirectory under ~/.workload_automation (or equivalent) where it will be automatically picked up by WA.

Plugin Basics

This section contains reference information common to plugins of all types.

The Context

Note

For clarification on the meaning of “workload specification” “spec”, “job” and “workload” and the distinction between them, please see the glossary.

The majority of methods in plugins accept a context argument. This is an instance of wa.framework.execution.ExecutionContext. It contains information about the current state of execution of WA and keeps track of things like which workload is currently running.

Notable methods of the context are:

context.get_resource(resource, strict=True)

This method should be used to retrieve a resource using the resource getters rather than using the ResourceResolver directly as this method additionally record any found resources hash in the output metadata.

context.add_artifact(name, host_file_path, kind, description=None, classifier=None)

Plugins can add artifacts of various kinds to the run output directory for WA and associate them with a description and/or classifier.

context.add_metric(name, value, units=None, lower_is_better=False, classifiers=None)

This method should be used to add metrics that have been generated from a workload, this will allow WA to process the results accordingly depending on which output processors are enabled.

Notable attributes of the context are:

context.workload

wa.framework.workload object that is currently being executed.

context.tm

This is the target manager that can be used to access various information about the target including initialization parameters.

context.current_job

This is an instance of wa.framework.job.Job and contains all the information relevant to the workload job currently being executed.

context.current_job.spec

The current workload specification being executed. This is an instance of wa.framework.configuration.core.JobSpec and defines the workload and the parameters under which it is being executed.

context.current_job.current_iteration

The current iteration of the spec that is being executed. Note that this is the iteration for that spec, i.e. the number of times that spec has been run, not the total number of all iterations have been executed so far.

context.job_output

This is the output object for the current iteration which is an instance of wa.framework.output.JobOutput. It contains the status of the iteration as well as the metrics and artifacts generated by the job.

In addition to these, context also defines a few useful paths (see below).

Paths

You should avoid using hard-coded absolute paths in your plugins whenever possible, as they make your code too dependent on a particular environment and may mean having to make adjustments when moving to new (host and/or device) platforms. To help avoid hard-coded absolute paths, WA defines a number of standard locations. You should strive to define your paths relative to one of these.

On the host

Host paths are available through the context object, which is passed to most plugin methods.

context.run_output_directory

This is the top-level output directory for all WA results (by default, this will be “wa_output” in the directory in which WA was invoked.

context.output_directory

This is the output directory for the current iteration. This will an iteration-specific subdirectory under the main results location. If there is no current iteration (e.g. when processing overall run results) this will point to the same location as run_output_directory.

Additionally, the global wa.settings object exposes on other location:

settings.dependency_directory

this is the root directory for all plugin dependencies (e.g. media files, assets etc) that are not included within the plugin itself.

As per Python best practice, it is recommended that methods and values in os.path standard library module are used for host path manipulation.

On the target

Workloads and instruments have a target attribute, which is an interface to the target used by WA. It defines the following location:

target.working_directory

This is the directory for all WA-related files on the target. All files deployed to the target should be pushed to somewhere under this location (the only exception being executables installed with target.install method).

Since there could be a mismatch between path notation used by the host and the target, the os.path modules should not be used for on-target path manipulation. Instead target has an equipment module exposed through target.path attribute. This has all the same attributes and behaves the same way as os.path, but is guaranteed to produce valid paths for the target, irrespective of the host’s path notation. For example:

result_file = self.target.path.join(self.target.working_directory, "result.txt")
self.command = "{} -a -b -c {}".format(target_binary, result_file)

Note

Output processors, unlike workloads and instruments, do not have their own target attribute as they are designed to be able to be run offline.

Parameters

All plugins can be parametrized. Parameters are specified using parameters class attribute. This should be a list of wa.framework.plugin.Parameter instances. The following attributes can be specified on parameter creation:

name

This is the only mandatory argument. The name will be used to create a corresponding attribute in the plugin instance, so it must be a valid Python identifier.

kind

This is the type of the value of the parameter. This must be an callable. Normally this should be a standard Python type, e.g. int or float, or one the types defined in wa.utils.types. If not explicitly specified, this will default to str.

Note

Irrespective of the kind specified, None is always a valid value for a parameter. If you don’t want to allow None, then set mandatory (see below) to True.

allowed_values

A list of the only allowed values for this parameter.

Note

For composite types, such as list_of_strings or list_of_ints in wa.utils.types, each element of the value will be checked against allowed_values rather than the composite value itself.

default

The default value to be used for this parameter if one has not been specified by the user. Defaults to None.

mandatory

A bool indicating whether this parameter is mandatory. Setting this to True will make None an illegal value for the parameter. Defaults to False.

Note

Specifying a default will mean that this parameter will, effectively, be ignored (unless the user sets the param to None).

Note

Mandatory parameters are bad. If at all possible, you should strive to provide a sensible default or to make do without the parameter. Only when the param is absolutely necessary, and there really is no sensible default that could be given (e.g. something like login credentials), should you consider making it mandatory.

constraint

This is an additional constraint to be enforced on the parameter beyond its type or fixed allowed values set. This should be a predicate (a function that takes a single argument – the user-supplied value – and returns a bool indicating whether the constraint has been satisfied).

override

A parameter name must be unique not only within an plugin but also with that plugin’s class hierarchy. If you try to declare a parameter with the same name as already exists, you will get an error. If you do want to override a parameter from further up in the inheritance hierarchy, you can indicate that by setting override attribute to True.

When overriding, you do not need to specify every other attribute of the parameter, just the ones you what to override. Values for the rest will be taken from the parameter in the base class.

Validation and cross-parameter constraints

A plugin will get validated at some point after construction. When exactly this occurs depends on the plugin type, but it will be validated before it is used.

You can implement validate method in your plugin (that takes no arguments beyond the self) to perform any additional internal validation in your plugin. By “internal”, I mean that you cannot make assumptions about the surrounding environment (e.g. that the device has been initialized).

The contract for validate method is that it should raise an exception (either wa.framework.exception.ConfigError or plugin-specific exception type – see further on this page) if some validation condition has not, and cannot, been met. If the method returns without raising an exception, then the plugin is in a valid internal state.

Note that validate can be used not only to verify, but also to impose a valid internal state. In particular, this where cross-parameter constraints can be resolved. If the default or allowed_values of one parameter depend on another parameter, there is no way to express that declaratively when specifying the parameters. In that case the dependent attribute should be left unspecified on creation and should instead be set inside validate.

Logging

Every plugin class has it’s own logger that you can access through self.logger inside the plugin’s methods. Generally, a Target will log everything it is doing, so you shouldn’t need to add much additional logging for device actions. However you might what to log additional information, e.g. what settings your plugin is using, what it is doing on the host, etc. (Operations on the host will not normally be logged, so your plugin should definitely log what it is doing on the host). One situation in particular where you should add logging is before doing something that might take a significant amount of time, such as downloading a file.

Documenting

All plugins and their parameter should be documented. For plugins themselves, this is done through description class attribute. The convention for an plugin description is that the first paragraph should be a short summary description of what the plugin does and why one would want to use it (among other things, this will get extracted and used by wa list command). Subsequent paragraphs (separated by blank lines) can then provide a more detailed description, including any limitations and setup instructions.

For parameters, the description is passed as an argument on creation. Please note that if default, allowed_values, or constraint, are set in the parameter, they do not need to be explicitly mentioned in the description (wa documentation utilities will automatically pull those). If the default is set in validate or additional cross-parameter constraints exist, this should be documented in the parameter description.

Both plugins and their parameters should be documented using reStructureText markup (standard markup for Python documentation). See:

http://docutils.sourceforge.net/rst.html

Aside from that, it is up to you how you document your plugin. You should try to provide enough information so that someone unfamiliar with your plugin is able to use it, e.g. you should document all settings and parameters your plugin expects (including what the valid values are).

Error Notification

When you detect an error condition, you should raise an appropriate exception to notify the user. The exception would typically be ConfigError or (depending the type of the plugin) WorkloadError/DeviceError/InstrumentError/OutputProcessorError. All these errors are defined in wa.framework.exception module.

A ConfigError should be raised where there is a problem in configuration specified by the user (either through the agenda or config files). These errors are meant to be resolvable by simple adjustments to the configuration (and the error message should suggest what adjustments need to be made. For all other errors, such as missing dependencies, mis-configured environment, problems performing operations, etc., the plugin type-specific exceptions should be used.

If the plugin itself is capable of recovering from the error and carrying on, it may make more sense to log an ERROR or WARNING level message using the plugin’s logger and to continue operation.

Metrics

This is what WA uses to store a single metric collected from executing a workload.

name

the name of the metric. Uniquely identifies the metric within the results.

value

The numerical value of the metric for this execution of a workload. This can be either an int or a float.

units

Units for the collected value. Can be None if the value has no units (e.g. it’s a count or a standardised score).

lower_is_better

Boolean flag indicating where lower values are better than higher ones. Defaults to False.

classifiers

A set of key-value pairs to further classify this metric beyond current iteration (e.g. this can be used to identify sub-tests).

Metrics can be added to WA output via the context:

context.add_metric("score", 9001)
context.add_metric("time", 2.35, "seconds", lower_is_better=True)

You only need to specify the name and the value for the metric. Units and classifiers are optional, and, if not specified otherwise, it will be assumed that higher values are better (lower_is_better=False).

The metric will be added to the result for the current job, if there is one; otherwise, it will be added to the overall run result.

Artifacts

This is an artifact generated during execution/post-processing of a workload. Unlike metrics, this represents an actual artifact, such as a file, generated. This may be “output”, such as trace, or it could be “meta data” such as logs. These are distinguished using the kind attribute, which also helps WA decide how it should be handled. Currently supported kinds are:

log

A log file. Not part of the “output” as such but contains information about the run/workload execution that be useful for diagnostics/meta analysis.

meta

A file containing metadata. This is not part of the “output”, but contains information that may be necessary to reproduce the results (contrast with log artifacts which are not necessary).

data

This file contains new data, not available otherwise and should be considered part of the “output” generated by WA. Most traces would fall into this category.

export

Exported version of results or some other artifact. This signifies that this artifact does not contain any new data that is not available elsewhere and that it may be safely discarded without losing information.

raw

Signifies that this is a raw dump/log that is normally processed to extract useful information and is then discarded. In a sense, it is the opposite of export, but in general may also be discarded.

Note

whether a file is marked as log/data or raw depends on how important it is to preserve this file, e.g. when archiving, vs how much space it takes up. Unlike export artifacts which are (almost) always ignored by other exporters as that would never result in data loss, raw files may be processed by exporters if they decided that the risk of losing potentially (though unlikely) useful data is greater than the time/space cost of handling the artifact (e.g. a database uploader may choose to ignore raw artifacts, whereas a network filer archiver may choose to archive them).

As with Metrics, artifacts are added via the context:

context.add_artifact("benchmark-output", "bech-out.txt", kind="raw",
                     description="stdout from running the benchmark")

Note

The file must exist on the host by the point at which the artifact is added, otherwise an error will be raised.

The artifact will be added to the result of the current job, if there is one; otherwise, it will be added to the overall run result. In some situations, you may wish to add an artifact to the overall run while being inside a job context, this can be done with add_run_artifact:

context.add_run_artifact("score-summary", "scores.txt", kind="export",
       description="""
       Summary of the scores so far. Updated after
       every job.
       """)

In this case, you also need to make sure that the file represented by the artifact is written to the output directory for the run and not the current job.

Metadata

There may be additional data collected by your plugin that you want to record as part of the result, but that does not fall under the definition of a “metric”. For example, you may want to record the version of the binary you’re executing. You can do this by adding a metadata entry:

context.add_metadata("exe-version", 1.3)

Metadata will be added either to the current job result, or to the run result, depending on the current context. Metadata values can be scalars or nested structures of dicts/sequences; the only constraint is that all constituent objects of the value must be POD (Plain Old Data) types – see WA POD types.

There is special support for handling metadata entries that are dicts of values. The following call adds a metadata entry "versions" who’s value is {"my_exe": 1.3}:

context.add_metadata("versions", "my_exe", 1.3)

If you attempt to add a metadata entry that already exists, an error will be raised, unless force=True is specified, in which case, it will be overwritten.

Updating an existing entry whose value is a collection can be done with update_metadata:

context.update_metadata("ran_apps", "my_exe")
context.update_metadata("versions", "my_other_exe", "2.3.0")

The first call appends "my_exe" to the list at metadata entry "ran_apps". The second call updates the "versions" dict in the metadata with an entry for "my_other_exe".

If an entry does not exit, update_metadata will create it, so it’s recommended to always use that for non-scalar entries, unless the intention is specifically to ensure that the entry does not exist at the time of the call.

Classifiers

Classifiers are key-value pairs of tags that can be attached to metrics, artifacts, jobs, or the entire run. Run and job classifiers get propagated to metrics and artifacts. Classifier keys should be strings, and their values should be simple scalars (i.e. strings, numbers, or bools).

Classifiers can be thought of as “tags” that are used to annotate metrics and artifacts, in order to make it easier to sort through them later. WA itself does not do anything with them, however output processors will augment the output they generate with them (for example, csv processor can add additional columns for classifier keys).

Classifiers are typically added by the user to attach some domain-specific information (e.g. experiment configuration identifier) to the results, see using classifiers. However, plugins can also attach additional classifiers, by specifying them in add_metric() and add_artifacts() calls.

Metadata vs Classifiers

Both metadata and classifiers are sets of essentially opaque key-value pairs that get included in WA output. While they may seem somewhat similar and interchangeable, they serve different purposes and are handled differently by the framework.

Classifiers are used to annotate generated metrics and artifacts in order to assist post-processing tools in sorting through them. Metadata is used to record additional information that is not necessary for processing the results, but that may be needed in order to reproduce them or to make sense of them in a grander context.

These are specific differences in how they are handled:

  • Classifiers are often provided by the user via the agenda (though can also be added by plugins). Metadata in only created by the framework and plugins.

  • Classifier values must be simple scalars; metadata values can be nested collections, such as lists or dicts.

  • Classifiers are used by output processors to augment the output the latter generated; metadata typically isn’t.

  • Classifiers are essentially associated with the individual metrics and artifacts (though in the agenda they’re specified at workload, section, or global run levels); metadata is associated with a particular job or run, and not with metrics or artifacts.


Execution Decorators

The following decorators are available for use in order to control how often a method should be able to be executed.

For example, if we want to ensure that no matter how many iterations of a particular workload are ran, we only execute the initialize method for that instance once, we would use the decorator as follows:

from wa.utils.exec_control import once

@once
def initialize(self, context):
    # Perform one time initialization e.g. installing a binary to target
    # ..
@once_per_instance

The specified method will be invoked only once for every bound instance within the environment.

@once_per_class

The specified method will be invoked only once for all instances of a class within the environment.

@once

The specified method will be invoked only once within the environment.

Warning

If a method containing a super call is decorated, this will also cause stop propagation up the hierarchy, unless this is the desired effect, additional functionality should be implemented in a separate decorated method which can then be called allowing for normal propagation to be retained.


Utils

Workload Automation defines a number of utilities collected under wa.utils subpackage. These utilities were created to help with the implementation of the framework itself, but may be also be useful when implementing plugins.


Workloads

All of the type inherit from the same base Workload and its API can be seen in the API section.

Workload methods (except for validate) take a single argument that is a wa.framework.execution.ExecutionContext instance. This object keeps track of the current execution state (such as the current workload, iteration number, etc), and contains, among other things, a wa.framework.output.JobOutput instance that should be populated from the update_output method with the results of the execution. For more information please see the context documentation.

# ...

def update_output(self, context):
   # ...
   context.add_metric('energy', 23.6, 'Joules', lower_is_better=True)

# ...
Workload Types

There are multiple workload types that you can inherit from depending on the purpose of your workload, the different types along with an output of their intended use cases are outlined below.

Basic (wa.Workload)

This type of the workload is the simplest type of workload and is left the to developer to implement its full functionality.

Apk (wa.ApkWorkload)

This workload will simply deploy and launch an android app in its basic form with no UI interaction.

UiAuto (wa.UiautoWorkload)

This workload is for android targets which will use UiAutomator to interact with UI elements without a specific android app, for example performing manipulation of android itself. This is the preferred type of automation as the results are more portable and reproducible due to being able to wait for UI elements to appear rather than having to rely on human recordings.

ApkUiAuto (wa.ApkUiautoWorkload)

The is the same as the UiAuto workload however it is also associated with an android app e.g. AdobeReader and will automatically deploy and launch the android app before running the automation.

Revent (wa.ReventWorkload)

Revent workloads are designed primarily for games as these are unable to be automated with UiAutomator due to the fact that they are rendered within a single UI element. They require a recording to be performed manually and currently will need re-recording for each different device. For more information on revent workloads been please see Automating GUI Interactions With Revent

APKRevent (wa.ApkReventWorkload)

The is the same as the Revent workload however it is also associated with an android app e.g. AngryBirds and will automatically deploy and launch the android app before running the automation.


Revent Recordings

Convention for Naming revent Files for Revent Workloads

There is a convention for naming revent files which you should follow if you want to record your own revent files. Each revent file must be called (case sensitive) <device name>.<stage>.revent, where <device name> is the name of your device (as defined by the model name of your device which can be retrieved with adb shell getprop ro.product.model or by the name attribute of your customized device class), and <stage> is one of the following currently supported stages:

setup

This stage is where the application is loaded (if present). It is a good place to record an revent here to perform any tasks to get ready for the main part of the workload to start.

run

This stage is where the main work of the workload should be performed. This will allow for more accurate results if the revent file for this stage only records the main actions under test.

extract_results

This stage is used after the workload has been completed to retrieve any metrics from the workload e.g. a score.

teardown

This stage is where any final actions should be performed to clean up the workload.

Only the run stage is mandatory, the remaining stages will be replayed if a recording is present otherwise no actions will be performed for that particular stage.

All your custom revent files should reside at '$WA_USER_DIRECTORY/dependencies/WORKLOAD NAME/'. So typically to add a custom revent files for a device named “mydevice” and a workload name “myworkload”, you would need to add the revent files to the directory ~/.workload_automation/dependencies/myworkload/revent_files creating the directory structure if necessary.

mydevice.setup.revent
mydevice.run.revent
mydevice.extract_results.revent
mydevice.teardown.revent

Any revent file in the dependencies will always overwrite the revent file in the workload directory. So for example it is possible to just provide one revent for setup in the dependencies and use the run.revent that is in the workload directory.

File format of revent recordings

You do not need to understand recording format in order to use revent. This section is intended for those looking to extend revent in some way, or to utilize revent recordings for other purposes.

Format Overview

Recordings are stored in a binary format. A recording consists of three sections:

+-+-+-+-+-+-+-+-+-+-+-+
|       Header        |
+-+-+-+-+-+-+-+-+-+-+-+
|                     |
|  Device Description |
|                     |
+-+-+-+-+-+-+-+-+-+-+-+
|                     |
|                     |
|     Event Stream    |
|                     |
|                     |
+-+-+-+-+-+-+-+-+-+-+-+

The header contains metadata describing the recording. The device description contains information about input devices involved in this recording. Finally, the event stream contains the recorded input events.

All fields are either fixed size or prefixed with their length or the number of (fixed-sized) elements.

Note

All values below are little endian

Recording Header

An revent recoding header has the following structure

  • It starts with the “magic” string REVENT to indicate that this is an revent recording.

  • The magic is followed by a 16 bit version number. This indicates the format version of the recording that follows. Current version is 2.

  • The next 16 bits indicate the type of the recording. This dictates the structure of the Device Description section. Valid values are:

    0

    This is a general input event recording. The device description contains a list of paths from which the events where recorded.

    1

    This a gamepad recording. The device description contains the description of the gamepad used to create the recording.

  • The header is zero-padded to 128 bits.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      'R'      |      'E'      |      'V'      |      'E'      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      'N'      |      'T'      |            Version            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             Mode              |            PADDING            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            PADDING                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Device Description

This section describes the input devices used in the recording. Its structure is determined by the value of Mode field in the header.

General Recording

Note

This is the only format supported prior to version 2.

The recording has been made from all available input devices. This section contains the list of /dev/input paths for the devices, prefixed with total number of the devices recorded.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Number of devices                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|             Device paths              +-+-+-+-+-+-+-+-+-+-+-+-+
|                                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Similarly, each device path is a length-prefixed string. Unlike C strings, the path is not NULL-terminated.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Length of device path                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                          Device path                          |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Gamepad Recording

The recording has been made from a specific gamepad. All events in the stream will be for that device only. The section describes the device properties that will be used to create a virtual input device using /dev/uinput. Please see linux/input.h header in the Linux kernel source for more information about the fields in this section.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            bustype            |             vendor            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            product            |            version            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         name_length                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                             name                              |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            ev_bits                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                                                               |
|                       key_bits (96 bytes)                     |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                                                               |
|                       rel_bits (96 bytes)                     |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                                                               |
|                       abs_bits (96 bytes)                     |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          num_absinfo                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                                                               |
|                                                               |
|                                                               |
|                        absinfo entries                        |
|                                                               |
|                                                               |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Each absinfo entry consists of six 32 bit values. The number of entries is determined by the abs_bits field.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            value                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           minimum                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           maximum                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             fuzz                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             flat                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          resolution                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Event Stream

The majority of an revent recording will be made up of the input events that were recorded. The event stream is prefixed with the number of events in the stream, and start and end times for the recording.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Number of events                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Number of events (cont.)                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Start Time Seconds                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Start Time Seconds (cont.)                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Start Time Microseconds                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|              Start Time Microseconds (cont.)                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        End Time Seconds                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    End Time Seconds (cont.)                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      End Time Microseconds                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                End Time Microseconds (cont.)                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                                                               |
|             Events                                            |
|                                                               |
|                                                               |
|                                       +-+-+-+-+-+-+-+-+-+-+-+-+
|                                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Event Structure

Each event entry structured as follows:

  • An unsigned short integer representing which device from the list of device paths this event is for (zero indexed). E.g. Device ID = 3 would be the 4th device in the list of device paths.

  • A unsigned long integer representing the number of seconds since “epoch” when the event was recorded.

  • A unsigned long integer representing the microseconds part of the timestamp.

  • An unsigned integer representing the event type

  • An unsigned integer representing the event code

  • An unsigned integer representing the event value

For more information about the event type, code and value please read: https://www.kernel.org/doc/Documentation/input/event-codes.txt

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Device ID           |        Timestamp Seconds      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Timestamp Seconds (cont.)               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Timestamp Seconds (cont.)   |        stamp Micoseconds      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|              Timestamp Micoseconds (cont.)                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Micoseconds (cont.) |          Event Type           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Event Code           |          Event Value          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Event Value (cont.)     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Parser

WA has a parser for revent recordings. This can be used to work with revent recordings in scripts. Here is an example:

from wa.utils.revent import ReventRecording

with ReventRecording('/path/to/recording.revent') as recording:
    print("Recording: {}".format(recording.filepath))
    print("There are {} input events".format(recording.num_events))
    print("Over a total of {} seconds".format(recording.duration))

Serialization

Overview of Serialization

WA employs a serialization mechanism in order to store some of its internal structures inside the output directory. Serialization is performed in two stages:

  1. A serializable object is converted into a POD (Plain Old Data) structure consisting of primitive Python types, and a few additional types (see WA POD Types below).

  2. The POD structure is serialized into a particular format by a generic parser for that format. Currently, yaml and json are supported.

Deserialization works in reverse order – first the serialized text is parsed into a POD, which is then converted to the appropriate object.

Implementing Serializable Objects

In order to be considered serializable, an object must either be a POD, or it must implement the to_pod() method and from_pod static/class method, which will perform the conversion to/form pod.

As an example, below as a (somewhat trimmed) implementation of the Event class:

class Event(object):

    @staticmethod
    def from_pod(pod):
        instance = Event(pod['message'])
        instance.timestamp = pod['timestamp']
        return instance

    def __init__(self, message):
        self.timestamp = datetime.utcnow()
        self.message = message

    def to_pod(self):
        return dict(
            timestamp=self.timestamp,
            message=self.message,
        )

Serialization API

read_pod(source, fmt=None)
write_pod(pod, dest, fmt=None)

These read and write PODs from a file. The format will be inferred, if possible, from the extension of the file, or it may be specified explicitly with fmt. source and dest can be either strings, in which case they will be interpreted as paths, or they can be file-like objects.

is_pod(obj)

Returns True if obj is a POD, and False otherwise.

dump(o, wfh, fmt='json', \*args, \*\*kwargs)
load(s, fmt='json', \*args, \*\*kwargs)

These implment an altenative serialization interface, which matches the interface exposed by the parsers for the supported formats.

WA POD Types

POD types are types that can be handled by a serializer directly, without a need for any additional information. These consist of the build-in python types

list
tuple
dict
set
str
unicode
int
float
bool

…the standard library types

OrderedDict
datetime

…and the WA-defined types

regex_type
none_type
level
cpu_mask

Any structure consisting entirely of these types is a POD and can be serialized and then deserialized without losing information. It is important to note that only these specific types are considered POD, their subclasses are not.

Note

dicts get deserialized as OrderedDicts.

Serialization Formats

WA utilizes two serialization formats: YAML and JSON. YAML is used for files intended to be primarily written and/or read by humans; JSON is used for files intended to be primarily written and/or read by WA and other programs.

The parsers and serializers for these formats used by WA have been modified to handle additional types (e.g. regular expressions) that are typically not supported by the formats. This was done in such a way that the resulting files are still valid and can be parsed by any parser for that format.


Contributing

Code

We welcome code contributions via GitHub pull requests. To help with maintainability of the code line we ask that the code uses a coding style consistent with the rest of WA code. Briefly, it is

  • PEP8 with line length and block comment rules relaxed (the wrapper for PEP8 checker inside dev_scripts will run it with appropriate configuration).

  • Four-space indentation (no tabs!).

  • Title-case for class names, underscore-delimited lower case for functions, methods, and variables.

  • Use descriptive variable names. Delimit words with '_' for readability. Avoid shortening words, skipping vowels, etc (common abbreviations such as “stats” for “statistics”, “config” for “configuration”, etc are OK). Do not use Hungarian notation (so prefer birth_date over dtBirth).

New extensions should also follow implementation guidelines specified in the Writing Plugins section of the documentation.

We ask that the following checks are performed on the modified code prior to submitting a pull request:

Note

You will need pylint and pep8 static checkers installed:

pip install pep8
pip install pylint

It is recommended that you install via pip rather than through your distribution’s package manager because the latter is likely to contain out-of-date version of these tools.

  • ./dev_scripts/pylint should be run without arguments and should produce no output (any output should be addressed by making appropriate changes in the code or adding a pylint ignore directive, if there is a good reason for keeping the code as is).

  • ./dev_scripts/pep8 should be run without arguments and should produce no output (any output should be addressed by making appropriate changes in the code).

  • If the modifications touch core framework (anything under wa/framework), unit tests should be run using nosetests, and they should all pass.

    • If significant additions have been made to the framework, unit tests should be added to cover the new functionality.

  • If modifications have been made to the UI Automation source of a workload, the corresponding APK should be rebuilt and submitted as part of the same pull request. This can be done via the build.sh script in the relevant uiauto subdirectory.

  • If modifications have been made to documentation (this includes description attributes for Parameters and Extensions), documentation should be built to make sure no errors or warning during build process, and a visual inspection of new/updated sections in resulting HTML should be performed to ensure everything renders as expected.

Once you have your contribution is ready, please follow instructions in GitHub documentation to create a pull request.


Documentation

Headings

To allow for consistent headings to be used through out the document the following character sequences should be used when creating headings

=========
Heading 1
=========

Only used for top level headings which should also have an entry in the
navigational side bar.

*********
Heading 2
*********

Main page heading used for page title, should not have a top level entry in the
side bar.

Heading 3
==========

Regular section heading.

Heading 4
---------

Sub-heading.

Heading 5
~~~~~~~~~

Heading 6
^^^^^^^^^

Heading 7
"""""""""

Configuration Listings

To keep a consistent style for presenting configuration options, the preferred style is to use a Field List.

(See: http://docutils.sourceforge.net/docs/user/rst/quickref.html#field-lists)

Example:

:parameter: My Description

Will render as:

parameter

My Description


API Style

When documenting an API the currently preferred style is to provide a short description of the class, followed by the attributes of the class in a Definition List followed by the methods using the method directive.

(See: http://docutils.sourceforge.net/docs/user/rst/quickref.html#definition-lists)

Example:

API
===

:class:`MyClass`
----------------

:class:`MyClass` is an example class to demonstrate API documentation.

``attribute1``
    The first attribute of the example class.

``attribute2``
    Another attribute example.

methods
"""""""

.. method:: MyClass.retrieve_output(name)

    Retrieve the output for ``name``.

    :param name:  The output that should be returned.
    :return: An :class:`Output` object for ``name``.
    :raises NotFoundError: If no output can be found.

Will render as:

MyClass is an example class to demonstrate API documentation.

attribute1

The first attribute of the example class.

attribute2

Another attribute example.

methods
MyClass.retrieve_output(name)

Retrieve the output for name.

Parameters

name – The output that should be returned.

Returns

An Output object for name.

Raises

NotFoundError – If no output can be found.