Developer Information¶
Contents
Developer Guide¶
Writing Plugins¶
Workload Automation offers several plugin points (or plugin types). The most interesting of these are
workloads: | These are the tasks that get executed and measured on the device. These can be benchmarks, high-level use cases, or pretty much anything else. |
---|---|
targets: | These are interfaces to the physical devices (development boards or end-user devices, such as smartphones) that use cases run on. Typically each model of a physical device would require its own interface class (though some functionality may be reused by subclassing from an existing base). |
instruments: | Instruments allow collecting additional data from workload execution (e.g. system traces). Instruments are not specific to a particular workload. Instruments can hook into any stage of workload execution. |
output processors: | |
These are used to format the results of workload execution once they have been collected. Depending on the callback used, these will run either after each iteration and/or at the end of the run, after all of the results have been collected. |
You can create a plugin by subclassing the appropriate base class, defining
appropriate methods and attributes, and putting the .py file containing the
class into the “plugins” subdirectory under ~/.workload_automation
(or
equivalent) where it will be automatically picked up by WA.
Plugin Basics¶
This sub-section covers things common to implementing plugins of all types. It is recommended you familiarize yourself with the information here before proceeding onto guidance for specific plugin types.
Dynamic Resource Resolution¶
The idea is to decouple resource identification from resource discovery. Workloads/instruments/devices/etc state what resources they need, and not where to look for them – this instead is left to the resource resolver that is part of the execution context. The actual discovery of resources is performed by resource getters that are registered with the resolver.
A resource type is defined by a subclass of
wa.framework.resource.Resource
. An instance of this class describes a
resource that is to be obtained. At minimum, a Resource
instance has an
owner (which is typically the object that is looking for the resource), but
specific resource types may define other parameters that describe an instance of
that resource (such as file names, URLs, etc).
An object looking for a resource invokes a resource resolver with an instance of
Resource
describing the resource it is after. The resolver goes through the
getters registered for that resource type in priority order attempting to obtain
the resource; once the resource is obtained, it is returned to the calling
object. If none of the registered getters could find the resource,
NotFoundError
is raised (or None
is returned instead, if invoked with
strict=False
).
The most common kind of object looking for resources is a Workload
, and the
Workload
class defines
wa.framework.workload.Workload.init_resources()
method, which may be
overridden by subclasses to perform resource resolution. For example, a workload
looking for an executable file would do so like this:
from wa import Workload
from wa.import Executable
class MyBenchmark(Workload):
# ...
def init_resources(self, resolver):
resource = Executable(self, self.target.abi, 'my_benchmark')
host_exe = resolver.get(resource)
# ...
Currently available resource types are defined in wa.framework.resources
.
Deploying executables to a target¶
Some targets may have certain restrictions on where executable binaries may be placed and how they should be invoked. To ensure your plugin works with as wide a range of targets as possible, you should use WA APIs for deploying and invoking executables on a target, as outlined below.
As with other resources, host-side paths to the executable binary to be deployed
should be obtained via the resource resolver. A
special resource type, Executable
is used to identify a binary to be
deployed. This is similar to the regular File
resource, however it takes an
additional parameter that specifies the ABI for which the executable was
compiled for.
In order for the binary to be obtained in this way, it must be stored in one of
the locations scanned by the resource resolver in a directory structure
<root>/bin/<abi>/<binary>
(where root
is the base resource location to
be searched, e.g. ~/.workload_automation/dependencies/<plugin name>
, and
<abi>
is the ABI for which the executable has been compiled, as returned by
self.target.abi
).
Once the path to the host-side binary has been obtained, it may be deployed
using one of two methods from a
Target instance –
install
or install_if_needed
. The latter will check a version of that
binary has been previously deployed by WA and will not try to re-install.
from wa import Executable
host_binary = context.get(Executable(self, self.target.abi, 'some_binary'))
target_binary = self.target.install_if_needed(host_binary)
Note
Please also note that the check is done based solely on the binary name. For more information please see the devlib documentation.
Both of the above methods will return the path to the installed binary on the
target. The executable should be invoked only via that path; do not assume
that it will be in PATH
on the target (or that the executable with the same
name in PATH
is the version deployed by WA.
For more information on how to implement this, please see the how to guide.
Deploying assets¶
WA provides a generic mechanism for deploying assets during workload initialization.
WA will automatically try to retrieve and deploy each asset to the target’s working directory
that is contained in a workloads deployable_assets
attribute stored as a list.
If the parameter cleanup_assets
is set then any asset deployed will be removed
again and the end of the run.
If the workload requires a custom deployment mechanism the deploy_assets
method can be overridden for that particular workload, in which case, either
additional assets should have their on target paths added to the workload’s
deployed_assests
attribute or the corresponding remove_assets
method
should also be implemented.
Adding an Instrument¶
Instruments can be used to collect additional measurements during workload execution (e.g. collect power readings). An instrument can hook into almost any stage of workload execution. Any new instrument should be a subclass of Instrument and it must have a name. When a new instrument is added to Workload Automation, the methods of the new instrument will be found automatically and hooked up to the supported signals. Once a signal is broadcasted, the corresponding registered method is invoked.
Each method in Instrument
must take two arguments, which are self
and
context
. Supported methods and their corresponding signals can be found in
the Signals Documentation. To make
implementations easier and common, the basic steps to add new instrument is
similar to the steps to add new workload and an example can be found in the
How To section.
To implement your own instrument the relevant methods of the interface shown below should be implemented:
name: The name of the instrument, this must be unique to WA. description: A description of what the instrument can be used for. parameters: A list of additional Parameters
the instrument can take.initialize(context): This method will only be called once during the workload run therefore operations that only need to be performed initially should be performed here for example pushing the files to the target device, installing them. setup(context): This method is invoked after the workload is setup. All the necessary setup should go inside this method. Setup, includes operations like clearing logs, additional configuration etc. start(context): It is invoked just before the workload start execution. Here is where instrument measurement start being registered/taken. stop(context): It is invoked just after the workload execution stops and where the measurements should stop being taken/registered. update_output(context): This method is invoked after the workload updated its result and where the taken measures should be added to the result so it can be processed by WA. teardown(context): It is invoked after the workload is torn down. It is a good place to clean any logs generated by the instrument. finalize(context): This method is the complement to the initialize method and will also only be called once so should be used to deleting/uninstalling files pushed to the device.
This is similar to a Workload
, except all methods are optional. In addition to
the workload-like methods, instruments can define a number of other methods that
will get invoked at various points during run execution. The most useful of
which is perhaps initialize
that gets invoked after the device has been
initialised for the first time, and can be used to perform one-time setup (e.g.
copying files to the device – there is no point in doing that for each
iteration). The full list of available methods can be found in
Signals Documentation.
Prioritization¶
Callbacks (e.g. setup()
methods) for all instruments get executed at the
same point during workload execution, one after another. The order in which the
callbacks get invoked should be considered arbitrary and should not be relied
on (e.g. you cannot expect that just because instrument A is listed before
instrument B in the config, instrument A’s callbacks will run first).
In some cases (e.g. in start()
and stop()
methods), it is important to
ensure that a particular instrument’s callbacks run a closely as possible to the
workload’s invocations in order to maintain accuracy of readings; or,
conversely, that a callback is executed after the others, because it takes a
long time and may throw off the accuracy of other instruments. You can do
this by using decorators on the appropriate methods. The available decorators are:
very_slow
, slow
, normal
, fast
, very_fast
, with very_fast
running closest to the workload invocation and very_slow
running furtherest
away. For example:
from wa import very_fast
# ..
class PreciseInstrument(Instrument)
# ...
@very_fast
def start(self, context):
pass
@very_fast
def stop(self, context):
pass
# ...
PreciseInstrument
will be started after all other instruments (i.e.
just before the workload runs), and it will stopped before all other
instruments (i.e. just after the workload runs).
If more than one active instrument has specified fast (or slow) callbacks, then their execution order with respect to each other is not guaranteed. In general, having a lot of instruments enabled is going to negatively affect the readings. The best way to ensure accuracy of measurements is to minimize the number of active instruments (perhaps doing several identical runs with different instruments enabled).
Example¶
Below is a simple instrument that measures the execution time of a workload:
class ExecutionTimeInstrument(Instrument):
"""
Measure how long it took to execute the run() methods of a Workload.
"""
name = 'execution_time'
def initialize(self, context):
self.start_time = None
self.end_time = None
@very_fast
def start(self, context):
self.start_time = time.time()
@very_fast
def stop(self, context):
self.end_time = time.time()
def update_output(self, context):
execution_time = self.end_time - self.start_time
context.add_metric('execution_time', execution_time, 'seconds')
Instrumentation Signal-Method Mapping¶
Instrument methods get automatically hooked up to signals based on their names. Mostly, the method name corresponds to the name of the signal, however there are a few convenience aliases defined (listed first) to make easier to relate instrumentation code to the workload execution model. For an overview on when these signals are dispatched during execution please see the Developer Reference.
method name | signal |
---|---|
initialize | run-initialized |
setup | before-workload-setup |
start | before-workload-execution |
stop | after-workload-execution |
process_workload_output | successful-workload-output-update |
update_output | after-workload-output-update |
teardown | after-workload-teardown |
finalize | run-finalized |
on_run_start | run-started |
on_run_end | run-completed |
on_job_start | job-started |
on_job_restart | job-restarted |
on_job_end | job-completed |
on_job_failure | job-failed |
on_job_abort | job-aborted |
before_job_queue_execution | before-job-queue-execution |
on_successful_job_queue_exection | successful-job-queue-execution |
after_job_queue_execution | after-job-queue-execution |
before_job | before-job |
on_successful_job | successful-job |
after_job | after-job |
before_processing_job_output | before-job-output-processed |
on_successfully_processing_job | successful-job-output-processed |
after_processing_job_output | after-job-output-processed |
before_reboot | before-reboot |
on_successful_reboot | successful-reboot |
after_reboot | after-reboot |
on_error | error-logged |
on_warning | warning-logged |
The methods above may be decorated with on the listed decorators to set the priority of the Instrument method relative to other callbacks registered for the signal (within the same priority level, callbacks are invoked in the order they were registered). The table below shows the mapping of the decorator to the corresponding priority:
decorator | priority |
---|---|
extremely_low | -30 |
very_low | -20 |
low | -10 |
normal | 0 |
high | 10 |
very_high | 20 |
extremely_high | 30 |
Unresponsive Targets¶
If a target is believed to be unresponsive, instrument callbacks will be disabled to prevent a cascade of errors and potential corruptions of state, as it is generally assumed that instrument callbacks will want to do something with the target.
If your callback only does something with the host, and does not require an
active target connection, you can decorate it with @hostside
decorator to
ensure it gets invoked even if the target becomes unresponsive.
Adding an Output processor¶
A output processor is responsible for processing the results. This may involve formatting and writing them to a file, uploading them to a database, generating plots, etc. WA comes with a few output processors that output results in a few common formats (such as csv or JSON).
You can add your own output processors by creating a Python file in
~/.workload_automation/plugins
with a class that derives from
wa.OutputProcessor
, and should
implement the relevant methods shown below, for more information and please
see the
Adding an Output Processor section.
name: The name of the output processor, this must be unique to WA. description: A description of what the output processor can be used for. parameters: A list of additional Parameters
the output processor can take.initialize(context): This method will only be called once during the workload run therefore operations that only need to be performed initially should be performed here. process_job_output(output, target_info, run_ouput): This method should be used to perform the processing of the output from an individual job output. This is where any additional artifacts should be generated if applicable. export_job_output(output, target_info, run_ouput): This method should be used to perform the exportation of the existing data collected/generated for an individual job. E.g. uploading them to a database etc. process_run_output(output, target_info): This method should be used to perform the processing of the output from the run as a whole. This is where any additional artifacts should be generated if applicable. export_run_output(output, target_info): This method should be used to perform the exportation of the existing data collected/generated for the run as a whole. E.g. uploading them to a database etc. finalize(context): This method is the complement to the initialize method and will also only be called once.
The method names should be fairly self-explanatory. The difference between “process” and “export” methods is that export methods will be invoked after process methods for all output processors have been generated. Process methods may generate additional artifacts (metrics, files, etc.), while export methods should not – they should only handle existing results (upload them to a database, archive on a filer, etc).
The output object passed to job methods is an instance of
wa.framework.output.JobOutput
, the output object passed to run methods
is an instance of wa.RunOutput
.
Adding a Resource Getter¶
A resource getter is a plugin that is designed to retrieve a resource (binaries, APK files or additional workload assets). Resource getters are invoked in priority order until one returns the desired resource.
If you want WA to look for resources somewhere it doesn’t by default (e.g. you have a repository of APK files), you can implement a getter for the resource and register it with a higher priority than the standard WA getters, so that it gets invoked first.
Instances of a resource getter should implement the following interface:
class ResourceGetter(Plugin):
name = None
def register(self, resolver):
raise NotImplementedError()
The getter should define a name for itself (as with all plugins), in addition it
should implement the register
method. This involves registering a method
with the resolver that should used to be called when trying to retrieve a resource
(typically get
) along with it’s priority (see Getter Prioritization
below. That method should return an instance of the resource that
has been discovered (what “instance” means depends on the resource, e.g. it
could be a file path), or None
if this getter was unable to discover
that resource.
Getter Prioritization¶
A priority is an integer with higher numeric values indicating a higher priority. The following standard priority aliases are defined for getters:
preferred: Take this resource in favour of the environment resource. local: Found somewhere under ~/.workload_automation/ or equivalent, or from environment variables, external configuration files, etc. These will override resource supplied with the package. lan: Resource will be retrieved from a locally mounted remote location (such as samba share) remote: Resource will be downloaded from a remote location (such as an HTTP server) package: Resource provided with the package.
These priorities are defined as class members of
wa.framework.resource.SourcePriority
, e.g. SourcePriority.preferred
.
Most getters in WA will be registered with either local
or
package
priorities. So if you want your getter to override the default, it
should typically be registered as preferred
.
You don’t have to stick to standard priority levels (though you should, unless there is a good reason). Any integer is a valid priority. The standard priorities range from 0 to 40 in increments of 10.
Example¶
The following is an implementation of a getter that searches for files in the
users dependencies directory, typically
~/.workload_automation/dependencies/<workload_name>
It uses the
get_from_location
method to filter the available files in the provided
directory appropriately:
import sys
from wa import settings,
from wa.framework.resource import ResourceGetter, SourcePriority
from wa.framework.getters import get_from_location
from wa.utils.misc import ensure_directory_exists as _d
class UserDirectory(ResourceGetter):
name = 'user'
def register(self, resolver):
resolver.register(self.get, SourcePriority.local)
def get(self, resource):
basepath = settings.dependencies_directory
directory = _d(os.path.join(basepath, resource.owner.name))
return get_from_location(directory, resource)
Adding a Target¶
In WA3, a ‘target’ consists of a platform and a devlib target. The
implementations of the targets are located in devlib
. WA3 will instantiate a
devlib target passing relevant parameters parsed from the configuration. For
more information about devlib targets please see the documentation.
- The currently available platforms are:
generic: The ‘standard’ platform implementation of the target, this should work for the majority of use cases. juno: A platform implementation specifically for the juno. tc2: A platform implementation specifically for the tc2. gem5: A platform implementation to interact with a gem5 simulation. - The currently available targets from devlib are:
linux: A device running a Linux based OS. android: A device running Android OS. local: Used to run locally on a linux based host. chromeos: A device running ChromeOS, supporting an android container if available.
For an example of adding you own customized version of an existing devlib target, please see the how to section Adding a Custom Target.
Other Plugin Types¶
In addition to plugin types covered above, there are few other, more specialized ones. They will not be covered in as much detail. Most of them expose relatively simple interfaces with only a couple of methods and it is expected that if the need arises to extend them, the API-level documentation that accompanies them, in addition to what has been outlined here, should provide enough guidance.
commands: | This allows extending WA with additional sub-commands (to supplement exiting ones outlined in the Commands section). |
---|---|
modules: | Modules are “plugins for plugins”. They can be loaded by other plugins to expand their functionality (for example, a flashing module maybe loaded by a device in order to support flashing). |
Packaging Your Plugins¶
If your have written a bunch of plugins, and you want to make it easy to
deploy them to new systems and/or to update them on existing systems, you can
wrap them in a Python package. You can use wa create package
command to
generate appropriate boiler plate. This will create a setup.py
and a
directory for your package that you can place your plugins into.
For example, if you have a workload inside my_workload.py
and an output
processor in my_output_processor.py
, and you want to package them as
my_wa_exts
package, first run the create command
wa create package my_wa_exts
This will create a my_wa_exts
directory which contains a
my_wa_exts/setup.py
and a subdirectory my_wa_exts/my_wa_exts
which is
the package directory for your plugins (you can rename the top-level
my_wa_exts
directory to anything you like – it’s just a “container” for the
setup.py and the package directory). Once you have that, you can then copy your
plugins into the package directory, creating
my_wa_exts/my_wa_exts/my_workload.py
and
my_wa_exts/my_wa_exts/my_output_processor.py
. If you have a lot of
plugins, you might want to organize them into subpackages, but only the
top-level package directory is created by default, and it is OK to have
everything in there.
Note
When discovering plugins through this mechanism, WA traverses the Python module/submodule tree, not the directory structure, therefore, if you are going to create subdirectories under the top level directory created for you, it is important that your make sure they are valid Python packages; i.e. each subdirectory must contain a __init__.py (even if blank) in order for the code in that directory and its subdirectories to be discoverable.
At this stage, you may want to edit params
structure near the bottom of
the setup.py
to add correct author, license and contact information (see
“Writing the Setup Script” section in standard Python documentation for
details). You may also want to add a README and/or a COPYING file at the same
level as the setup.py. Once you have the contents of your package sorted,
you can generate the package by running
cd my_wa_exts
python setup.py sdist
This will generate my_wa_exts/dist/my_wa_exts-0.0.1.tar.gz
package which
can then be deployed on the target system with standard Python package
management tools, e.g.
sudo pip install my_wa_exts-0.0.1.tar.gz
As part of the installation process, the setup.py in the package, will write the
package’s name into ~/.workoad_automation/packages
. This will tell WA that
the package contains plugin and it will load them next time it runs.
Note
There are no uninstall hooks in setuputils
, so if you ever
uninstall your WA plugins package, you will have to manually remove
it from ~/.workload_automation/packages
otherwise WA will complain
about a missing package next time you try to run it.
How Tos¶
Contents
Deploying Executables¶
Installing binaries for a particular plugin should generally only be performed
once during a run. This should typically be done in the initialize
method,
if the only functionality performed in the method is to install the required binaries
then the initialize
method should be decorated with the @once
decorator otherwise this should be placed into a dedicated
method which is decorated instead. Please note if doing this then any installed
paths should be added as class attributes rather than instance variables. As a
general rule if binaries are installed as part of initialize
then they
should be uninstalled in the complementary finalize
method.
Part of an example workload demonstrating this is shown below:
class MyWorkload(Workload):
#..
@once
def initialize(self, context):
resource = Executable(self, self.target.abi, 'my_executable')
host_binary = context.resolver.get(resource)
MyWorkload.target_binary = self.target.install(host_binary)
#..
def setup(self, context):
self.command = "{} -a -b -c".format(self.target_binary)
self.target.execute(self.command)
#..
@once
def finalize(self, context):
self.target.uninstall('my_executable')
Adding a Workload¶
The easiest way to create a new workload is to use the
create command. wa create workload <args>
. This
will use predefined templates to create a workload based on the options that are
supplied to be used as a starting point for the workload. For more information
on using the create workload command see wa create workload -h
The first thing to decide is the type of workload you want to create depending on the OS you will be using and the aim of the workload. The are currently 6 available workload types to choose as detailed in the Developer Reference.
Once you have decided what type of workload you wish to choose this can be
specified with -k <workload_kind>
followed by the workload name. This
will automatically generate a workload in the your WA_CONFIG_DIR/plugins
. If
you wish to specify a custom location this can be provided with -p
<path>
Adding a Basic Workload¶
To add a basic workload you can simply use the command:
wa create workload basic
This will generate a very basic workload with dummy methods for the workload interface and it is left to the developer to add any required functionality to the workload.
Not all the methods are required to be implemented, this example shows how a subset might be used to implement a simple workload that times how long it takes to compress a file of a particular size on the device.
Note
This is intended as an example of how to implement the Workload interface. The methodology used to perform the actual measurement is not necessarily sound, and this Workload should not be used to collect real measurements.
The first step is to subclass our desired
workload type depending on the purpose of our workload,
in this example we are implementing a very simple workload and do not
require any additional feature so shall inherit directly from the the base
Workload
class. We then need to provide a name
for our workload
which is what will be used to identify your workload for example in an
agenda or via the show command.
import os
from wa import Workload, Parameter
class ZipTestWorkload(Workload):
name = 'ziptest'
The description
attribute should be a string in the structure of a short
summary of the purpose of the workload, and will be shown when using the
list command, followed by a more in- depth explanation
separated by a new line.
description = '''
Times how long it takes to gzip a file of a particular size on a device.
This workload was created for illustration purposes only. It should not be
used to collect actual measurements.
'''
In order to allow for additional configuration of the workload from a user a
list of parameters can be supplied. These can be
configured in a variety of different ways. For example here we are ensuring that
the value of the parameter is an integer and larger than 0 using the kind
and constraint
options, also if no value is provided we are providing a
default
value of 2000000. These parameters will automatically have their
value set as an attribute of the workload so later on we will be able to use the
value provided here as self.file_size
.
parameters = [
Parameter('file_size', kind=int, default=2000000,
constraint=lambda x: 0 < x,
description='Size of the file (in bytes) to be gzipped.')
]
Next we will implement our setup
method. This is where we do any preparation
that is required before the workload is ran, this is usually things like setting
up required files on the device and generating commands from user input. In this
case we will generate our input file on the host system and then push it to a
known location on the target for use in the ‘run’ stage.
def setup(self, context):
super(ZipTestWorkload, self).setup(context)
# Generate a file of the specified size containing random garbage.
host_infile = os.path.join(context.output_directory, 'infile')
command = 'openssl rand -base64 {} > {}'.format(self.file_size, host_infile)
os.system(command)
# Set up on-device paths
devpath = self.target.path # os.path equivalent for the target
self.target_infile = devpath.join(self.target.working_directory, 'infile')
self.target_outfile = devpath.join(self.target.working_directory, 'outfile')
# Push the file to the target
self.target.push(host_infile, self.target_infile)
The run
method is where the actual ‘work’ of the workload takes place and is
what is measured by any instrumentation. So for this example this is the
execution of creating the zip file on the target.
def run(self, context):
cmd = 'cd {} && (time gzip {}) &>> {}'
self.target.execute(cmd.format(self.target.working_directory,
self.target_infile,
self.target_outfile))
The extract_results
method is used to extract any results from the target
for example we want to pull the file containing the timing information that we
will use to generate metrics for our workload and then we add this file as an
artifact with a ‘raw’ kind, which means once WA has finished processing it will
allow it to decide whether to keep the file or not.
def extract_results(self, context):
super(ZipTestWorkload, self).extract_results(context)
# Pull the results file to the host
self.host_outfile = os.path.join(context.output_directory, 'timing_results')
self.target.pull(self.target_outfile, self.host_outfile)
context.add_artifact('ziptest-results', host_output_file, kind='raw')
The update_output
method we can do any generation of metrics that we wish to
for our workload. In this case we are going to simply convert the times reported
into seconds and add them as ‘metrics’ to WA which can then be displayed to the
user along with any others in a format dependant on which output processors they
have enabled for the run.
def update_output(self, context):
super(ZipTestWorkload, self).update_output(context)
# Extract metrics form the file's contents and update the result
# with them.
content = iter(open(self.host_outfile).read().strip().split())
for value, metric in zip(content, content):
mins, secs = map(float, value[:-1].split('m'))
context.add_metric(metric, secs + 60 * mins, 'seconds')
Finally in the teardown
method we will perform any required clean up for the
workload so we will delete the input and output files from the device.
def teardown(self, context):
super(ZipTestWorkload, self).teardown(context)
self.target.remove(self.target_infile)
self.target.remove(self.target_outfile)
The full implementation of this workload would look something like:
import os
from wa import Workload, Parameter
class ZipTestWorkload(Workload):
name = 'ziptest'
description = '''
Times how long it takes to gzip a file of a particular size on a device.
This workload was created for illustration purposes only. It should not be
used to collect actual measurements.
'''
parameters = [
Parameter('file_size', kind=int, default=2000000,
constraint=lambda x: 0 < x,
description='Size of the file (in bytes) to be gzipped.')
]
def setup(self, context):
super(ZipTestWorkload, self).setup(context)
# Generate a file of the specified size containing random garbage.
host_infile = os.path.join(context.output_directory, 'infile')
command = 'openssl rand -base64 {} > {}'.format(self.file_size, host_infile)
os.system(command)
# Set up on-device paths
devpath = self.target.path # os.path equivalent for the target
self.target_infile = devpath.join(self.target.working_directory, 'infile')
self.target_outfile = devpath.join(self.target.working_directory, 'outfile')
# Push the file to the target
self.target.push(host_infile, self.target_infile)
def run(self, context):
cmd = 'cd {} && (time gzip {}) &>> {}'
self.target.execute(cmd.format(self.target.working_directory,
self.target_infile,
self.target_outfile))
def extract_results(self, context):
super(ZipTestWorkload, self).extract_results(context)
# Pull the results file to the host
self.host_outfile = os.path.join(context.output_directory, 'timing_results')
self.target.pull(self.target_outfile, self.host_outfile)
context.add_artifact('ziptest-results', host_output_file, kind='raw')
def update_output(self, context):
super(ZipTestWorkload, self).update_output(context)
# Extract metrics form the file's contents and update the result
# with them.
content = iter(open(self.host_outfile).read().strip().split())
for value, metric in zip(content, content):
mins, secs = map(float, value[:-1].split('m'))
context.add_metric(metric, secs + 60 * mins, 'seconds')
def teardown(self, context):
super(ZipTestWorkload, self).teardown(context)
self.target.remove(self.target_infile)
self.target.remove(self.target_outfile)
Adding a ApkUiAutomator Workload¶
If we wish to create a workload to automate the testing of the Google Docs android app, we would choose to perform the automation using UIAutomator and we would want to automatically deploy and install the apk file to the target, therefore we would choose the ApkUiAutomator workload type with the following command:
$ wa create workload -k apkuiauto google_docs
Workload created in $WA_USER_DIRECTORY/plugins/google_docs
From here you can navigate to the displayed directory and you will find your
__init__.py
and a uiauto
directory. The former is your python WA
workload and will look something like this. For an example of what should be
done in each of the main method please see
adding a basic example above.
from wa import Parameter, ApkUiautoWorkload
class GoogleDocs(ApkUiautoWorkload):
name = 'google_docs'
description = "This is an placeholder description"
# Replace with a list of supported package names in the APK file(s).
package_names = ['package_name']
parameters = [
# Workload parameters go here e.g.
Parameter('example_parameter', kind=int, allowed_values=[1,2,3],
default=1, override=True, mandatory=False,
description='This is an example parameter')
]
def __init__(self, target, **kwargs):
super(GoogleDocs, self).__init__(target, **kwargs)
# Define any additional attributes required for the workload
def init_resources(self, resolver):
super(GoogleDocs, self).init_resources(resolver)
# This method may be used to perform early resource discovery and
# initialization. This is invoked during the initial loading stage and
# before the device is ready, so cannot be used for any device-dependent
# initialization. This method is invoked before the workload instance is
# validated.
def initialize(self, context):
super(GoogleDocs, self).initialize(context)
# This method should be used to perform once-per-run initialization of a
# workload instance.
def validate(self):
super(GoogleDocs, self).validate()
# Validate inter-parameter assumptions etc
def setup(self, context):
super(GoogleDocs, self).setup(context)
# Perform any necessary setup before starting the UI automation
def extract_results(self, context):
super(GoogleDocs, self).extract_results(context)
# Extract results on the target
def update_output(self, context):
super(GoogleDocs, self).update_output(context)
# Update the output within the specified execution context with the
# metrics and artifacts form this workload iteration.
def teardown(self, context):
super(GoogleDocs, self).teardown(context)
# Perform any final clean up for the Workload.
Depending on the purpose of your workload you can choose to implement which
methods you require. The main things that need setting are the list of
package_names
which must be a list of strings containing the android package
name that will be used during resource resolution to locate the relevant apk
file for the workload. Additionally the the workload parameters will need to
updating to any relevant parameters required by the workload as well as the
description.
The latter will contain a framework for performing the UI automation on the
target, the files you will be most interested in will be
uiauto/app/src/main/java/arm/wa/uiauto/UiAutomation.java
which will contain
the actual code of the automation and will look something like:
package com.arm.wa.uiauto.google_docs;
import android.app.Activity;
import android.os.Bundle;
import org.junit.Test;
import org.junit.runner.RunWith;
import android.support.test.runner.AndroidJUnit4;
import android.util.Log;
import android.view.KeyEvent;
// Import the uiautomator libraries
import android.support.test.uiautomator.UiObject;
import android.support.test.uiautomator.UiObjectNotFoundException;
import android.support.test.uiautomator.UiScrollable;
import android.support.test.uiautomator.UiSelector;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import com.arm.wa.uiauto.BaseUiAutomation;
@RunWith(AndroidJUnit4.class)
public class UiAutomation extends BaseUiAutomation {
protected Bundle parameters;
protected int example_parameter;
public static String TAG = "google_docs";
@Before
public void initilize() throws Exception {
// Perform any parameter initialization here
parameters = getParams(); // Required to decode passed parameters.
packageID = getPackageID(parameters);
example_parameter = parameters.getInt("example_parameter");
}
@Test
public void setup() throws Exception {
// Optional: Perform any setup required before the main workload
// is ran, e.g. dismissing welcome screens
}
@Test
public void runWorkload() throws Exception {
// The main UI Automation code goes here
}
@Test
public void extractResults() throws Exception {
// Optional: Extract any relevant results from the workload,
}
@Test
public void teardown() throws Exception {
// Optional: Perform any clean up for the workload
}
}
- A few items to note from the template:
- Each of the stages of execution for example
setup
,runWorkload
etc are decorated with the@Test
decorator, this is important to allow these methods to be called at the appropriate time however any additional methods you may add do not require this decorator. - The
initialize
method has the@Before
decorator, this is there to ensure that this method is called before executing any of the workload stages and therefore is used to decode and initialize any parameters that are passed in. - The code currently retrieves the
example_parameter
that was provided to the python workload as an Integer, there are similar calls to retrieve parameters of different types e.g.getString
,getBoolean
,getDouble
etc.
- Each of the stages of execution for example
Once you have implemented your java workload you can use the file
uiauto/build.sh
to compile your automation into an apk file to perform the
automation. The generated apk will be generated with the package name
com.arm.wa.uiauto.<workload_name>
which when running your workload will be
automatically detected by the resource getters and deployed to the device.
Adding a ReventApk Workload¶
If we wish to create a workload to automate the testing of a UI based workload that we cannot / do not wish to use UiAutomator then we can perform the automation using revent. In this example we would want to automatically deploy and install an apk file to the target, therefore we would choose the ApkRevent workload type with the following command:
$ wa create workload -k apkrevent my_game
Workload created in $WA_USER_DIRECTORY/plugins/my_game
This will generate a revent based workload you will end up with a very similar python file as to the one outlined in generating a UiAutomator based workload however without the accompanying java automation files.
The main difference between the two is that this workload will subclass
ApkReventWorkload
instead of ApkUiautomatorWorkload
as shown below.
from wa import ApkReventWorkload
class MyGame(ApkReventWorkload):
name = 'mygame'
package_names = ['com.mylogo.mygame']
# ..
Adding an Instrument¶
This is an example of how we would create a instrument which will trace device
errors using a custom “trace” binary file. For more detailed information please see the
Instrument Reference. The first thing to do is to subclass
Instrument
, overwrite the variable name with what we want our instrument
to be called and locate our binary for our instrument.
class TraceErrorsInstrument(Instrument):
name = 'trace-errors'
def __init__(self, target):
super(TraceErrorsInstrument, self).__init__(target)
self.binary_name = 'trace'
self.binary_file = os.path.join(os.path.dirname(__file__), self.binary_name)
self.trace_on_target = None
We then declare and implement the required methods as detailed in the
Instrument API. For the initialize
method, we want to install
the executable file to the target so we can use the target’s install
method which will try to copy the file to a location on the device that
supports execution, change the file mode appropriately and return the
file path on the target.
def initialize(self, context):
self.trace_on_target = self.target.install(self.binary_file)
Then we implemented the start method, which will simply run the file to start
tracing. Supposing that the call to this binary requires some overhead to begin
collecting errors we might want to decorate the method with the @slow
decorator to try and reduce the impact on other running instruments. For more
information on prioritization please see the
Developer Reference.
@slow
def start(self, context):
self.target.execute('{} start'.format(self.trace_on_target))
Lastly, we need to stop tracing once the workload stops and this happens in the stop method, assuming stopping the collection also require some overhead we have again decorated the method.
@slow
def stop(self, context):
self.target.execute('{} stop'.format(self.trace_on_target))
Once we have generated our result data we need to retrieve it from the device for further processing or adding directly to WA’s output for that job. For example for trace data we will want to pull it to the device and add it as a artifact to WA’s context as shown below:
def extract_results(self, context):
# pull the trace file from the target
self.result = os.path.join(self.target.working_directory, 'trace.txt')
self.target.pull(self.result, context.working_directory)
context.add_artifact('error_trace', self.result, kind='export')
Once we have retrieved the data we can now do any further processing and add any
relevant Metrics to the context. For this we
will use the the add_metric
method to add the results to the final output
for that workload. The method can be passed 4 params, which are the metric
key, value, unit and lower_is_better.
def update_output(self, context):
# parse the file if needs to be parsed, or add result directly to
# context.
metric = # ..
context.add_metric('number_of_errors', metric, lower_is_better=True
At the end of each job we might want to delete any files generated by the instruments and the code to clear these file goes in teardown method.
def teardown(self, context):
self.target.remove(os.path.join(self.target.working_directory, 'trace.txt'))
At the very end of the run we would want to uninstall the binary we deployed earlier.
def finalize(self, context):
self.target.uninstall(self.binary_name)
So the full example would look something like:
class TraceErrorsInstrument(Instrument):
name = 'trace-errors'
def __init__(self, target):
super(TraceErrorsInstrument, self).__init__(target)
self.binary_name = 'trace'
self.binary_file = os.path.join(os.path.dirname(__file__), self.binary_name)
self.trace_on_target = None
def initialize(self, context):
self.trace_on_target = self.target.install(self.binary_file)
@slow
def start(self, context):
self.target.execute('{} start'.format(self.trace_on_target))
@slow
def stop(self, context):
self.target.execute('{} stop'.format(self.trace_on_target))
def extract_results(self, context):
self.result = os.path.join(self.target.working_directory, 'trace.txt')
self.target.pull(self.result, context.working_directory)
context.add_artifact('error_trace', self.result, kind='export')
def update_output(self, context):
metric = # ..
context.add_metric('number_of_errors', metric, lower_is_better=True
def teardown(self, context):
self.target.remove(os.path.join(self.target.working_directory, 'trace.txt'))
def finalize(self, context):
self.target.uninstall(self.binary_name)
Adding an Output Processor¶
This is an example of how we would create an output processor which will format
the run metrics as a column-aligned table. The first thing to do is to subclass
OutputProcessor
and overwrite the variable name with what we want our
processor to be called and provide a short description.
Next we need to implement any relevant methods, (please see
adding an output processor for all the
available methods). In this case we only want to implement the
export_run_output
method as we are not generating any new artifacts and
we only care about the overall output rather than the individual job
outputs. The implementation is very simple, it just loops through all
the available metrics for all the available jobs and adds them to a list
which is written to file and then added as an artifact to
the context.
import os
from wa import OutputProcessor
from wa.utils.misc import write_table
class Table(OutputProcessor):
name = 'table'
description = 'Generates a text file containing a column-aligned table of run results.'
def export_run_output(self, output, target_info):
rows = []
for job in output.jobs:
for metric in job.metrics:
rows.append([metric.name, str(metric.value), metric.units or '',
metric.lower_is_better and '-' or '+'])
outfile = output.get_path('table.txt')
with open(outfile, 'w') as wfh:
write_table(rows, wfh)
output.add_artifact('results_table', 'table.txt', 'export')
Adding a Custom Target¶
This is an example of how we would create a customised target, this is typically used where we would need to augment the existing functionality for example on development boards where we need to perform additional actions to implement some functionality. In this example we are going to assume that this particular device is running Android and requires a special “wakeup” command to be sent before it can execute any other command.
To add a new target to WA we will first create a new file in
$WA_USER_DIRECTORY/plugins/example_target.py
. In order to facilitate with
creating a new target WA provides a helper function to create a description for
the specified target class, and specified components. For components that are
not explicitly specified it will attempt to guess sensible defaults based on the target
class’ bases.
# Import our helper function
from wa import add_description_for_target
# Import the Target that our custom implementation will be based on
from devlib import AndroidTarget
class ExampleTarget(AndroidTarget):
# Provide the name that will be used to identify your custom target
name = 'example_target'
# Override our custom method(s)
def execute(self, *args, **kwargs):
super(ExampleTarget, self).execute('wakeup', check_exit_code=False)
return super(ExampleTarget, self).execute(*args, **kwargs)
description = '''An Android target which requires an explicit "wakeup" command
to be sent before accepting any other command'''
# Call the helper function with our newly created function and its description.
add_description_for_target(ExampleTarget, description)
Processing WA Output¶
This section will illustrate the use of WA’s output processing API by creating a simple ASCII report generator. To make things concrete, this how-to will be processing the output from running the following agenda:
sections:
- runtime_params:
frequency: min
classifiers:
frequency: min
- runtime_params:
frequency: max
classifiers:
frequency: max
workloads:
- sysbench
- deepbench
This runs two workloads under two different configurations each – once with CPU frequency fixed to max, and once with CPU frequency fixed to min. Classifiers are used to indicate the configuration in the output.
First, create the RunOutput
object, which is the main interface for
interacting with WA outputs. Or alternatively a RunDatabaseOutput
if storing your results in a postgres database.
import sys
from wa import RunOutput
# Path to the output directory specified in the first argument
ro = RunOutput(sys.argv[1])
Run Info¶
Next, we’re going to print out an overall summary of the run.
from __future__ import print_function # for Python 2 compat.
from wa.utils.misc import format_duration
print('-'*20)
print('Run ID:', ro.info.uuid)
print('Run status:', ro.status)
print('Run started at:', ro.info.start_time.isoformat())
print('Run completed at:', ro.info.end_time.isoformat())
print('Run duration:', format_duration(ro.info.duration))
print('Ran', len(ro.jobs), 'jobs')
print('-'*20)
print()
RunOutput.info
is an instance of RunInfo
which encapsulates
Overall-run metadata, such as the duration.
Target Info¶
Next, some information about the device the results where collected on.
print(' Target Information ')
print(' ------------------- ')
print('hostname:', ro.target_info.hostname)
if ro.target_info.os == 'android':
print('Android ID:', ro.target_info.android_id)
else:
print('host ID:', ro.target_info.hostid)
print('CPUs:', ', '.join(cpu.name for cpu in ro.target_info.cpus))
print()
print('OS:', ro.target_info.os)
print('ABI:', ro.target_info.abi)
print('rooted:', ro.target_info.is_rooted)
print('kernel version:', ro.target_info.kernel_version)
print('os version:')
for k, v in ro.target_info.os_version.items():
print('\t', k+':', v)
print()
print('-'*27)
print()
RunOutput.target_info
is an instance of TargetInfo
that contains
information collected from the target during the run.
Jobs Summary¶
Next, show a summary of executed jobs.
from wa.utils.misc import write_table
print(' Jobs ')
print(' ---- ')
print()
rows = []
for job in ro.jobs:
rows.append([job.id, job.label, job.iteration, job.status])
write_table(rows, sys.stdout, align='<<><',
headers=['ID', 'LABEL', 'ITER.', 'STATUS'])
print()
print('-'*27)
print()
RunOutput.jobs
is a list of JobOutput
objects. These contain
information about that particular job, including its execution status, and
Metrics and Artifacts generated by the job.
Compare Metrics¶
Finally, collect metrics, sort them by the “frequency” classifier. Classifiers that are present in the metric but not its job have been added by the workload. For the purposes of this report, they will be used to augment the metric’s name.
from collections import defaultdict
print()
print(' Metrics Comparison ')
print(' ------------------ ')
print()
scores = defaultdict(lambda: defaultdict(lambda: defaultdict()))
for job in ro.jobs:
for metric in job.metrics:
workload = job.label
name = metric.name
freq = job.classifiers['frequency']
for cname, cval in sorted(metric.classifiers.items()):
if cname not in job.classifiers:
# was not propagated from the job, therefore was
# added by the workload
name += '/{}={}'.format(cname, cval)
scores[workload][name][freq] = metric
Once the metrics have been sorted, generate the report showing the delta
between the two configurations (indicated by the “frequency” classifier) and
highlight any unexpected deltas (based on the lower_is_better
attribute of
the metric). (In practice, you will want to run multiple iterations of each
configuration, calculate averages and standard deviations, and only highlight
statically significant deltas.)
rows = []
for workload in sorted(scores.keys()):
wldata = scores[workload]
for name in sorted(wldata.keys()):
min_score = wldata[name]['min'].value
max_score = wldata[name]['max'].value
delta = max_score - min_score
units = wldata[name]['min'].units or ''
lib = wldata[name]['min'].lower_is_better
warn = ''
if (lib and delta > 0) or (not lib and delta < 0):
warn = '!!!'
rows.append([workload, name,
'{:.3f}'.format(min_score), '{:.3f}'.format(max_score),
'{:.3f}'.format(delta), units, warn])
# separate workloads with a blank row
rows.append(['', '', '', '', '', '', ''])
write_table(rows, sys.stdout, align='<<>>><<',
headers=['WORKLOAD', 'METRIC', 'MIN.', 'MAX', 'DELTA', 'UNITS', ''])
print()
print('-'*27)
This concludes this how-to. For more information, please see output processing API documentation.
Complete Example¶
Below is the complete example code, and a report it generated for a sample run.
from __future__ import print_function # for Python 2 compat.
import sys
from collections import defaultdict
from wa import RunOutput
from wa.utils.misc import format_duration, write_table
# Path to the output directory specified in the first argument
ro = RunOutput(sys.argv[1])
print('-'*27)
print('Run ID:', ro.info.uuid)
print('Run status:', ro.status)
print('Run started at:', ro.info.start_time.isoformat())
print('Run completed at:', ro.info.end_time.isoformat())
print('Run duration:', format_duration(ro.info.duration))
print('Ran', len(ro.jobs), 'jobs')
print('-'*27)
print()
print(' Target Information ')
print(' ------------------- ')
print('hostname:', ro.target_info.hostname)
if ro.target_info.os == 'android':
print('Android ID:', ro.target_info.android_id)
else:
print('host ID:', ro.target_info.hostid)
print('CPUs:', ', '.join(cpu.name for cpu in ro.target_info.cpus))
print()
print('OS:', ro.target_info.os)
print('ABI:', ro.target_info.abi)
print('rooted:', ro.target_info.is_rooted)
print('kernel version:', ro.target_info.kernel_version)
print('OS version:')
for k, v in ro.target_info.os_version.items():
print('\t', k+':', v)
print()
print('-'*27)
print()
print(' Jobs ')
print(' ---- ')
print()
rows = []
for job in ro.jobs:
rows.append([job.id, job.label, job.iteration, job.status])
write_table(rows, sys.stdout, align='<<><',
headers=['ID', 'LABEL', 'ITER.', 'STATUS'])
print()
print('-'*27)
print()
print(' Metrics Comparison ')
print(' ------------------ ')
print()
scores = defaultdict(lambda: defaultdict(lambda: defaultdict()))
for job in ro.jobs:
for metric in job.metrics:
workload = job.label
name = metric.name
freq = job.classifiers['frequency']
for cname, cval in sorted(metric.classifiers.items()):
if cname not in job.classifiers:
# was not propagated from the job, therefore was
# added by the workload
name += '/{}={}'.format(cname, cval)
scores[workload][name][freq] = metric
rows = []
for workload in sorted(scores.keys()):
wldata = scores[workload]
for name in sorted(wldata.keys()):
min_score = wldata[name]['min'].value
max_score = wldata[name]['max'].value
delta = max_score - min_score
units = wldata[name]['min'].units or ''
lib = wldata[name]['min'].lower_is_better
warn = ''
if (lib and delta > 0) or (not lib and delta < 0):
warn = '!!!'
rows.append([workload, name,
'{:.3f}'.format(min_score), '{:.3f}'.format(max_score),
'{:.3f}'.format(delta), units, warn])
# separate workloads with a blank row
rows.append(['', '', '', '', '', '', ''])
write_table(rows, sys.stdout, align='<<>>><<',
headers=['WORKLOAD', 'METRIC', 'MIN.', 'MAX', 'DELTA', 'UNITS', ''])
print()
print('-'*27)
Sample output:
---------------------------
Run ID: 78aef931-cd4c-429b-ac9f-61f6893312e6
Run status: OK
Run started at: 2018-06-27T12:55:23.746941
Run completed at: 2018-06-27T13:04:51.067309
Run duration: 9 minutes 27 seconds
Ran 4 jobs
---------------------------
Target Information
-------------------
hostname: localhost
Android ID: b9d1d8b48cfba007
CPUs: A53, A53, A53, A53, A73, A73, A73, A73
OS: android
ABI: arm64
rooted: True
kernel version: 4.9.75-04208-g2c913991a83d-dirty 114 SMP PREEMPT Wed May 9 10:33:36 BST 2018
OS version:
all_codenames: O
base_os:
codename: O
incremental: eng.valsch.20170517.180115
preview_sdk: 0
release: O
sdk: 25
security_patch: 2017-04-05
---------------------------
Jobs
----
ID LABEL ITER. STATUS
-- ----- ----- ------
s1-wk1 sysbench 1 OK
s1-wk2 deepbench 1 OK
s2-wk1 sysbench 1 OK
s2-wk2 deepbench 1 OK
---------------------------
Metrics Comparison
------------------
WORKLOAD METRIC MIN. MAX DELTA UNITS
-------- ------ ---- --- ----- -----
deepbench GOPS/a_t=n/b_t=n/k=1024/m=128/n=1 0.699 0.696 -0.003 !!!
deepbench GOPS/a_t=n/b_t=n/k=1024/m=3072/n=1 0.471 0.715 0.244
deepbench GOPS/a_t=n/b_t=n/k=1024/m=3072/n=1500 23.514 36.432 12.918
deepbench GOPS/a_t=n/b_t=n/k=1216/m=64/n=1 0.333 0.333 -0.000 !!!
deepbench GOPS/a_t=n/b_t=n/k=128/m=3072/n=1 0.405 1.073 0.668
deepbench GOPS/a_t=n/b_t=n/k=128/m=3072/n=1500 19.914 34.966 15.052
deepbench GOPS/a_t=n/b_t=n/k=128/m=4224/n=1 0.232 0.486 0.255
deepbench GOPS/a_t=n/b_t=n/k=1280/m=128/n=1500 20.721 31.654 10.933
deepbench GOPS/a_t=n/b_t=n/k=1408/m=128/n=1 0.701 0.702 0.001
deepbench GOPS/a_t=n/b_t=n/k=1408/m=176/n=1500 19.902 29.116 9.214
deepbench GOPS/a_t=n/b_t=n/k=176/m=4224/n=1500 26.030 39.550 13.519
deepbench GOPS/a_t=n/b_t=n/k=2048/m=35/n=700 10.884 23.615 12.731
deepbench GOPS/a_t=n/b_t=n/k=2048/m=5124/n=700 26.740 37.334 10.593
deepbench execution_time 318.758 220.629 -98.129 seconds !!!
deepbench time (msec)/a_t=n/b_t=n/k=1024/m=128/n=1 0.375 0.377 0.002 !!!
deepbench time (msec)/a_t=n/b_t=n/k=1024/m=3072/n=1 13.358 8.793 -4.565
deepbench time (msec)/a_t=n/b_t=n/k=1024/m=3072/n=1500 401.338 259.036 -142.302
deepbench time (msec)/a_t=n/b_t=n/k=1216/m=64/n=1 0.467 0.467 0.000 !!!
deepbench time (msec)/a_t=n/b_t=n/k=128/m=3072/n=1 1.943 0.733 -1.210
deepbench time (msec)/a_t=n/b_t=n/k=128/m=3072/n=1500 59.237 33.737 -25.500
deepbench time (msec)/a_t=n/b_t=n/k=128/m=4224/n=1 4.666 2.224 -2.442
deepbench time (msec)/a_t=n/b_t=n/k=1280/m=128/n=1500 23.721 15.528 -8.193
deepbench time (msec)/a_t=n/b_t=n/k=1408/m=128/n=1 0.514 0.513 -0.001
deepbench time (msec)/a_t=n/b_t=n/k=1408/m=176/n=1500 37.354 25.533 -11.821
deepbench time (msec)/a_t=n/b_t=n/k=176/m=4224/n=1500 85.679 56.391 -29.288
deepbench time (msec)/a_t=n/b_t=n/k=2048/m=35/n=700 9.220 4.249 -4.970
deepbench time (msec)/a_t=n/b_t=n/k=2048/m=5124/n=700 549.413 393.517 -155.896
sysbench approx. 95 percentile 3.800 1.450 -2.350 ms
sysbench execution_time 1.790 1.437 -0.353 seconds !!!
sysbench response time avg 1.400 1.120 -0.280 ms
sysbench response time max 40.740 42.760 2.020 ms !!!
sysbench response time min 0.710 0.710 0.000 ms
sysbench thread fairness events avg 1250.000 1250.000 0.000
sysbench thread fairness events stddev 772.650 213.040 -559.610
sysbench thread fairness execution time avg 1.753 1.401 -0.352 !!!
sysbench thread fairness execution time stddev 0.000 0.000 0.000
sysbench total number of events 10000.000 10000.000 0.000
sysbench total time 1.761 1.409 -0.352 s
---------------------------
Developer Reference¶
Framework Overview¶
Execution Model¶
At the high level, the execution model looks as follows:
After some initial setup, the framework initializes the device, loads and initialized instruments and output processors and begins executing jobs defined by the workload specs in the agenda. Each job executes in basic stages:
- initialize
- Perform any once-per-run initialization of a workload instance, i.e. binary resource resolution.
- setup
- Initial setup for the workload is performed. E.g. required assets are deployed to the devices, required services or applications are launched, etc. Run time configuration of the device for the workload is also performed at this time.
- setup_rerun (apk based workloads only)
- For some apk based workloads the application is required to be started
twice. If the
requires_rerun
attribute of the workload is set toTrue
then after the first setup method is called the application will be killed and then restarted. This method can then be used to perform any additional setup required. - run
- This is when the workload actually runs. This is defined as the part of the workload that is to be measured. Exactly what happens at this stage depends entirely on the workload.
- extract results
- Extract any results that have been generated during the execution of the workload from the device and back to that target. Any files pulled from the devices should be added as artifacts to the run context.
- update output
- Perform any required parsing and processing of any collected results and add any generated metrics to the run context.
- teardown
- Final clean up is performed, e.g. applications may closed, files generated during execution deleted, etc.
Signals are dispatched (see below) at each stage of
workload execution, which installed instruments can hook into in order to
collect measurements, alter workload execution, etc. Instruments implementation
usually mirrors that of workloads, defining initialization, setup, teardown and
output processing stages for a particular instrument. Instead of a run
method instruments usually implement start
and stop
methods instead
which triggered just before and just after a workload run. However, the signal
dispatch mechanism gives a high degree of flexibility to instruments allowing
them to hook into almost any stage of a WA run (apart from the very early
initialization).
Metrics and artifacts generated by workloads and instruments are accumulated by the framework and are then passed to active output processors. This happens after each individual workload execution and at the end of the run. A output processor may chose to act at either or both of these points.
Control Flow¶
This section goes into more detail explaining the relationship between the major components of the framework and how control passes between them during a run. It will only go through the major transitions and interactions and will not attempt to describe every single thing that happens.
Note
This is the control flow for the wa run
command which is the main
functionality of WA. Other commands are much simpler and most of what
is described below does not apply to them.
wa.framework.entrypoint
parses the command from the arguments, creates awa.framework.configuration.execution.ConfigManager
and executes the run command (wa.commands.run.RunCommand
) passing it the ConfigManger.- Run command initializes the output directory and creates a
wa.framework.configuration.parsers.AgendaParser
and will parser an agenda and populate the ConfigManger based on the command line arguments. Finally it instantiates awa.framework.execution.Executor
and passes it the completed ConfigManager. - The Executor uses the ConfigManager to create a
wa.framework.configuration.core.RunConfiguration
and fully defines the configuration for the run (which will be serialised into__meta
subdirectory under the output directory). - The Executor proceeds to instantiate a TargetManager, used to handle the
device connection and configuration, and a
wa.framework.execution.ExecutionContext
which is used to track the current state of the run execution and also serves as a means of communication between the core framework and plugins. After this any required instruments and output processors are initialized and installed. - Finally, the Executor instantiates a
wa.framework.execution.Runner
, initializes its job queue with workload specs from the RunConfiguration, and kicks it off. - The Runner performs the run time configuration of the device and goes
through the workload specs (in the order defined by
execution_order
setting), running each spec according to the execution model described in the previous section and sending signals (see below) at appropriate points during execution. - At the end of the run, the control is briefly passed back to the Executor, which outputs a summary for the run.
Signal Dispatch¶
WA uses the louie (formerly, pydispatcher) library for signal dispatch. Callbacks can be registered for signals emitted during the run. WA uses a version of louie that has been modified to introduce priority to registered callbacks (so that callbacks that are know to be slow can be registered with a lower priority and therefore do not interfere with other callbacks).
This mechanism is abstracted for instruments. Methods of an
wa.framework.Instrument
subclass automatically get hooked to
appropriate signals based on their names when the instrument is “installed”
for the run. Priority can then be specified by adding extremely_fast
,
very_fast
, fast
, slow
, very_slow
or extremely_slow
decorators to the method definitions.
The full list of method names and the signals they map to may be seen at the instrument method map.
Signal dispatching mechanism may also be used directly, for example to
dynamically register callbacks at runtime or allow plugins other than
Instruments
to access stages of the run they are normally not aware of.
Signals can be either paired or non paired signals. Non paired signals are one
off signals that are sent to indicate special events or transitions in execution
stages have occurred for example TARGET_CONNECTED
. Paired signals are used to
signify the start and end of a particular event. If the start signal has been
sent the end signal is guaranteed to also be sent, whether the operation was a
successes or not, however in the case of correct operation an additional success
signal will also be sent. For example in the event of a successful reboot of the
the device, the following signals will be sent BEFORE_REBOOT
,
SUCCESSFUL_REBOOT
and AFTER_REBOOT
.
An overview of what signals are sent at which point during execution can be seen below. Most of the paired signals have been removed from the diagram for clarity and shown as being dispatched from a particular stage of execution, however in reality these signals will be sent just before and just after these stages are executed. As mentioned above for each of these signals there will be at least 2 and up to 3 signals sent. If the “BEFORE_X” signal (sent just before the stage is ran) is sent then the “AFTER_X” (sent just after the stage is ran) signal is guaranteed to also be sent, and under normal operation a “SUCCESSFUL_X” signal is also sent just after stage has been completed. The diagram also lists the conditional signals that can be sent at any time during execution if something unexpected happens, for example an error occurs or the user aborts the run.
For more information see Instrumentation Signal-Method Mapping.
Plugins¶
Workload Automation offers several plugin points (or plugin types). The most interesting of these are
workloads: | These are the tasks that get executed and measured on the device. These can be benchmarks, high-level use cases, or pretty much anything else. |
---|---|
targets: | These are interfaces to the physical devices (development boards or end-user devices, such as smartphones) that use cases run on. Typically each model of a physical device would require its own interface class (though some functionality may be reused by subclassing from an existing base). |
instruments: | Instruments allow collecting additional data from workload execution (e.g. system traces). Instruments are not specific to a particular workload. Instruments can hook into any stage of workload execution. |
output processors: | |
These are used to format the results of workload execution once they have been collected. Depending on the callback used, these will run either after each iteration and/or at the end of the run, after all of the results have been collected. |
You can create a plugin by subclassing the appropriate base class, defining
appropriate methods and attributes, and putting the .py file containing the
class into the “plugins” subdirectory under ~/.workload_automation
(or
equivalent) where it will be automatically picked up by WA.
Plugin Basics¶
This section contains reference information common to plugins of all types.
The Context¶
Note
For clarification on the meaning of “workload specification” (“spec”), “job” and “workload” and the distiction between them, please see the glossary.
The majority of methods in plugins accept a context argument. This is an
instance of wa.framework.execution.ExecutionContext
. It contains
information about the current state of execution of WA and keeps track of things
like which workload is currently running.
Notable methods of the context are:
context.get_resource(resource, strict=True): | |
---|---|
This method should be used to retrieve a resource using the resource getters rather than using the ResourceResolver directly as this method additionally record any found resources hash in the output metadata. | |
context.add_artifact(name, host_file_path, kind, description=None, classifier=None): | |
Plugins can add artifacts of various kinds to the run output directory for WA and associate them with a description and/or classifier. | |
context.add_metric(name, value, units=None, lower_is_better=False, classifiers=None): | |
This method should be used to add metrics that have been generated from a workload, this will allow WA to process the results accordingly depending on which output processors are enabled. |
Notable attributes of the context are:
context.workload: | |
---|---|
wa.framework.workload object that is currently being executed. |
|
context.tm: | This is the target manager that can be used to access various information about the target including initialization parameters. |
context.current_job: | |
This is an instance of wa.framework.job.Job and contains all
the information relevant to the workload job currently being executed. |
|
context.current_job.spec: | |
The current workload specification being executed. This is an
instance of wa.framework.configuration.core.JobSpec
and defines the workload and the parameters under which it is
being executed. |
|
context.current_job.current_iteration: | |
The current iteration of the spec that is being executed. Note that this is the iteration for that spec, i.e. the number of times that spec has been run, not the total number of all iterations have been executed so far. | |
context.job_output: | |
This is the output object for the current iteration which
is an instance of wa.framework.output.JobOutput . It contains
the status of the iteration as well as the metrics and artifacts
generated by the job. |
In addition to these, context also defines a few useful paths (see below).
Paths¶
You should avoid using hard-coded absolute paths in your plugins whenever possible, as they make your code too dependent on a particular environment and may mean having to make adjustments when moving to new (host and/or device) platforms. To help avoid hard-coded absolute paths, WA defines a number of standard locations. You should strive to define your paths relative to one of these.
On the host¶
Host paths are available through the context object, which is passed to most plugin methods.
- context.run_output_directory
- This is the top-level output directory for all WA results (by default, this will be “wa_output” in the directory in which WA was invoked.
- context.output_directory
- This is the output directory for the current iteration. This will an
iteration-specific subdirectory under the main results location. If
there is no current iteration (e.g. when processing overall run results)
this will point to the same location as
root_output_directory
.
Additionally, the global wa.settings
object exposes on other location:
- settings.dependency_directory
- this is the root directory for all plugin dependencies (e.g. media files, assets etc) that are not included within the plugin itself.
As per Python best practice, it is recommended that methods and values in
os.path
standard library module are used for host path manipulation.
On the target¶
Workloads and instruments have a target
attribute, which is an interface to
the target used by WA. It defines the following location:
- target.working_directory
- This is the directory for all WA-related files on the target. All files
deployed to the target should be pushed to somewhere under this location
(the only exception being executables installed with
target.install
method).
Since there could be a mismatch between path notation used by the host and the
target, the os.path
modules should not be used for on-target path
manipulation. Instead target has an equipment module exposed through
target.path
attribute. This has all the same attributes and behaves the
same way as os.path
, but is guaranteed to produce valid paths for the target,
irrespective of the host’s path notation. For example:
result_file = self.target.path.join(self.target.working_directory, "result.txt")
self.command = "{} -a -b -c {}".format(target_binary, result_file)
Note
Output processors, unlike workloads and instruments, do not have their own target attribute as they are designed to be able to be run offline.
Parameters¶
All plugins can be parametrized. Parameters are specified using
parameters
class attribute. This should be a list of
wa.framework.plugin.Parameter
instances. The following attributes can be
specified on parameter creation:
name: | This is the only mandatory argument. The name will be used to create a corresponding attribute in the plugin instance, so it must be a valid Python identifier. |
---|---|
kind: | This is the type of the value of the parameter. This must be an
callable. Normally this should be a standard Python type, e.g. Note Irrespective of the |
allowed_values: | A list of the only allowed values for this parameter. Note For composite types, such as |
default: | The default value to be used for this parameter if one has not been
specified by the user. Defaults to |
mandatory: | A Note Specifying a Note Mandatory parameters are bad. If at all possible, you should
strive to provide a sensible |
constraint: | This is an additional constraint to be enforced on the parameter beyond
its type or fixed allowed values set. This should be a predicate (a function
that takes a single argument – the user-supplied value – and returns
a |
override: | A parameter name must be unique not only within an plugin but also
with that plugin’s class hierarchy. If you try to declare a parameter
with the same name as already exists, you will get an error. If you do
want to override a parameter from further up in the inheritance
hierarchy, you can indicate that by setting When overriding, you do not need to specify every other attribute of the parameter, just the ones you what to override. Values for the rest will be taken from the parameter in the base class. |
Validation and cross-parameter constraints¶
A plugin will get validated at some point after construction. When exactly this occurs depends on the plugin type, but it will be validated before it is used.
You can implement validate
method in your plugin (that takes no arguments
beyond the self
) to perform any additional internal validation in your
plugin. By “internal”, I mean that you cannot make assumptions about the
surrounding environment (e.g. that the device has been initialized).
The contract for validate
method is that it should raise an exception
(either wa.framework.exception.ConfigError
or plugin-specific exception type – see
further on this page) if some validation condition has not, and cannot, been met.
If the method returns without raising an exception, then the plugin is in a
valid internal state.
Note that validate
can be used not only to verify, but also to impose a
valid internal state. In particular, this where cross-parameter constraints can
be resolved. If the default
or allowed_values
of one parameter depend on
another parameter, there is no way to express that declaratively when specifying
the parameters. In that case the dependent attribute should be left unspecified
on creation and should instead be set inside validate
.
Logging¶
Every plugin class has it’s own logger that you can access through
self.logger
inside the plugin’s methods. Generally, a Target
will
log everything it is doing, so you shouldn’t need to add much additional logging
for device actions. However you might what to log additional information, e.g.
what settings your plugin is using, what it is doing on the host, etc.
(Operations on the host will not normally be logged, so your plugin should
definitely log what it is doing on the host). One situation in particular where
you should add logging is before doing something that might take a significant
amount of time, such as downloading a file.
Documenting¶
All plugins and their parameter should be documented. For plugins
themselves, this is done through description
class attribute. The convention
for an plugin description is that the first paragraph should be a short
summary description of what the plugin does and why one would want to use it
(among other things, this will get extracted and used by wa list
command).
Subsequent paragraphs (separated by blank lines) can then provide a more
detailed description, including any limitations and setup instructions.
For parameters, the description is passed as an argument on creation. Please
note that if default
, allowed_values
, or constraint
, are set in the
parameter, they do not need to be explicitly mentioned in the description (wa
documentation utilities will automatically pull those). If the default
is set
in validate
or additional cross-parameter constraints exist, this should
be documented in the parameter description.
Both plugins and their parameters should be documented using reStructureText markup (standard markup for Python documentation). See:
http://docutils.sourceforge.net/rst.html
Aside from that, it is up to you how you document your plugin. You should try to provide enough information so that someone unfamiliar with your plugin is able to use it, e.g. you should document all settings and parameters your plugin expects (including what the valid values are).
Error Notification¶
When you detect an error condition, you should raise an appropriate exception to
notify the user. The exception would typically be ConfigError
or
(depending the type of the plugin)
WorkloadError
/DeviceError
/InstrumentError
/OutputProcessorError
.
All these errors are defined in wa.framework.exception
module.
A ConfigError
should be raised where there is a problem in configuration
specified by the user (either through the agenda or config files). These errors
are meant to be resolvable by simple adjustments to the configuration (and the
error message should suggest what adjustments need to be made. For all other
errors, such as missing dependencies, mis-configured environment, problems
performing operations, etc., the plugin type-specific exceptions should be
used.
If the plugin itself is capable of recovering from the error and carrying on, it may make more sense to log an ERROR or WARNING level message using the plugin’s logger and to continue operation.
Metrics¶
This is what WA uses to store a single metric collected from executing a workload.
name: the name of the metric. Uniquely identifies the metric within the results. value: The numerical value of the metric for this execution of a workload. This can be either an int or a float. units: Units for the collected value. Can be None if the value has no units (e.g. it’s a count or a standardised score). lower_is_better: Boolean flag indicating where lower values are better than higher ones. Defaults to False. classifiers: A set of key-value pairs to further classify this metric beyond current iteration (e.g. this can be used to identify sub-tests).
Metrics can be added to WA output via the context:
context.add_metric("score", 9001)
context.add_metric("time", 2.35, "seconds", lower_is_better=True)
You only need to specify the name and the value for the metric. Units and
classifiers are optional, and, if not specified otherwise, it will be assumed
that higher values are better (lower_is_better=False
).
The metric will be added to the result for the current job, if there is one; otherwise, it will be added to the overall run result.
Artifacts¶
This is an artifact generated during execution/post-processing of a workload.
Unlike metrics, this represents an actual artifact, such as a
file, generated. This may be “output”, such as trace, or it could be “meta
data” such as logs. These are distinguished using the kind
attribute, which
also helps WA decide how it should be handled. Currently supported kinds are:
log: A log file. Not part of the “output” as such but contains information about the run/workload execution that be useful for diagnostics/meta analysis.
meta: A file containing metadata. This is not part of the “output”, but contains information that may be necessary to reproduce the results (contrast with
log
artifacts which are not necessary).data: This file contains new data, not available otherwise and should be considered part of the “output” generated by WA. Most traces would fall into this category.
export: Exported version of results or some other artifact. This signifies that this artifact does not contain any new data that is not available elsewhere and that it may be safely discarded without losing information.
raw: Signifies that this is a raw dump/log that is normally processed to extract useful information and is then discarded. In a sense, it is the opposite of
export
, but in general may also be discarded.Note
whether a file is marked as
log
/data
orraw
depends on how important it is to preserve this file, e.g. when archiving, vs how much space it takes up. Unlikeexport
artifacts which are (almost) always ignored by other exporters as that would never result in data loss,raw
files may be processed by exporters if they decided that the risk of losing potentially (though unlikely) useful data is greater than the time/space cost of handling the artifact (e.g. a database uploader may choose to ignoreraw
artifacts, whereas a network filer archiver may choose to archive them).
As with Metrics, artifacts are added via the context:
context.add_artifact("benchmark-output", "bech-out.txt", kind="raw",
description="stdout from running the benchmark")
Note
The file must exist on the host by the point at which the artifact is added, otherwise an error will be raised.
The artifact will be added to the result of the current job, if there is one;
otherwise, it will be added to the overall run result. In some situations, you
may wish to add an artifact to the overall run while being inside a job context,
this can be done with add_run_artifact
:
context.add_run_artifact("score-summary", "scores.txt", kind="export",
description="""
Summary of the scores so far. Updated after
every job.
""")
In this case, you also need to make sure that the file represented by the artifact is written to the output directory for the run and not the current job.
Metadata¶
There may be additional data collected by your plugin that you want to record as part of the result, but that does not fall under the definition of a “metric”. For example, you may want to record the version of the binary you’re executing. You can do this by adding a metadata entry:
context.add_metadata("exe-version", 1.3)
Metadata will be added either to the current job result, or to the run result, depending on the current context. Metadata values can be scalars or nested structures of dicts/sequences; the only constraint is that all constituent objects of the value must be POD (Plain Old Data) types – see WA POD types.
There is special support for handling metadata entries that are dicts of values.
The following call adds a metadata entry "versions"
who’s value is
{"my_exe": 1.3}
:
context.add_metadata("versions", "my_exe", 1.3)
If you attempt to add a metadata entry that already exists, an error will be
raised, unless force=True
is specified, in which case, it will be
overwritten.
Updating an existing entry whose value is a collection can be done with
update_metadata
:
context.update_metadata("ran_apps", "my_exe")
context.update_metadata("versions", "my_other_exe", "2.3.0")
The first call appends "my_exe"
to the list at metadata entry
"ran_apps"
. The second call updates the "versions"
dict in the metadata
with an entry for "my_other_exe"
.
If an entry does not exit, update_metadata
will create it, so it’s
recommended to always use that for non-scalar entries, unless the intention is
specifically to ensure that the entry does not exist at the time of the call.
Classifiers¶
Classifiers are key-value pairs of tags that can be attached to metrics, artifacts, jobs, or the entire run. Run and job classifiers get propagated to metrics and artifacts. Classifier keys should be strings, and their values should be simple scalars (i.e. strings, numbers, or bools).
Classifiers can be thought of as “tags” that are used to annotate metrics and
artifacts, in order to make it easier to sort through them later. WA itself does
not do anything with them, however output processors will augment the output
they generate with them (for example, csv
processor can add additional
columns for classifier keys).
Classifiers are typically added by the user to attach some domain-specific
information (e.g. experiment configuration identifier) to the results, see
using classifiers. However, plugins can also attach
additional classifiers, by specifying them in add_metric()
and
add_artifacts()
calls.
Metadata vs Classifiers¶
Both metadata and classifiers are sets of essentially opaque key-value pairs that get included in WA output. While they may seem somewhat similar and interchangeable, they serve different purposes and are handled differently by the framework.
Classifiers are used to annotate generated metrics and artifacts in order to assist post-processing tools in sorting through them. Metadata is used to record additional information that is not necessary for processing the results, but that may be needed in order to reproduce them or to make sense of them in a grander context.
These are specific differences in how they are handled:
- Classifiers are often provided by the user via the agenda (though can also be added by plugins). Metadata in only created by the framework and plugins.
- Classifier values must be simple scalars; metadata values can be nested collections, such as lists or dicts.
- Classifiers are used by output processors to augment the output the latter generated; metadata typically isn’t.
- Classifiers are essentially associated with the individual metrics and artifacts (though in the agenda they’re specified at workload, section, or global run levels); metadata is associated with a particular job or run, and not with metrics or artifacts.
Execution Decorators¶
The following decorators are available for use in order to control how often a method should be able to be executed.
For example, if we want to ensure that no matter how many iterations of a particular workload are ran, we only execute the initialize method for that instance once, we would use the decorator as follows:
from wa.utils.exec_control import once
@once
def initialize(self, context):
# Perform one time initialization e.g. installing a binary to target
# ..
@once_per_instance¶
The specified method will be invoked only once for every bound instance within the environment.
@once_per_class¶
The specified method will be invoked only once for all instances of a class within the environment.
@once¶
The specified method will be invoked only once within the environment.
Warning
If a method containing a super call is decorated, this will also cause stop propagation up the hierarchy, unless this is the desired effect, additional functionality should be implemented in a separate decorated method which can then be called allowing for normal propagation to be retained.
Utils¶
Workload Automation defines a number of utilities collected under
wa.utils
subpackage. These utilities were created to help with the
implementation of the framework itself, but may be also be useful when
implementing plugins.
Workloads¶
All of the type inherit from the same base Workload
and its API can be
seen in the API section.
Workload methods (except for validate
) take a single argument that is a
wa.framework.execution.ExecutionContext
instance. This object keeps
track of the current execution state (such as the current workload, iteration
number, etc), and contains, among other things, a
wa.framework.output.JobOutput
instance that should be populated from
the update_output
method with the results of the execution. For more
information please see the context documentation.
# ...
def update_output(self, context):
# ...
context.add_metric('energy', 23.6, 'Joules', lower_is_better=True)
# ...
Workload Types¶
There are multiple workload types that you can inherit from depending on the purpose of your workload, the different types along with an output of their intended use cases are outlined below.
Basic (wa.Workload
)¶
This type of the workload is the simplest type of workload and is left the to developer to implement its full functionality.
Apk (wa.ApkWorkload
)¶
This workload will simply deploy and launch an android app in its basic form with no UI interaction.
UiAuto (wa.UiautoWorkload
)¶
This workload is for android targets which will use UiAutomator to interact with UI elements without a specific android app, for example performing manipulation of android itself. This is the preferred type of automation as the results are more portable and reproducible due to being able to wait for UI elements to appear rather than having to rely on human recordings.
ApkUiAuto (wa.ApkUiautoWorkload
)¶
The is the same as the UiAuto workload however it is also associated with an android app e.g. AdobeReader and will automatically deploy and launch the android app before running the automation.
Revent (wa.ReventWorkload
)¶
Revent workloads are designed primarily for games as these are unable to be automated with UiAutomator due to the fact that they are rendered within a single UI element. They require a recording to be performed manually and currently will need re-recording for each different device. For more information on revent workloads been please see Automating GUI Interactions With Revent
APKRevent (wa.ApkReventWorkload
)¶
The is the same as the Revent workload however it is also associated with an android app e.g. AngryBirds and will automatically deploy and launch the android app before running the automation.
Revent Recordings¶
Convention for Naming revent Files for Revent Workloads¶
There is a convention for naming revent files which you should follow if you
want to record your own revent files. Each revent file must start with the
device name(case sensitive) then followed by a dot ‘.’ then the stage name
then ‘.revent’. All your custom revent files should reside at
'~/.workload_automation/dependencies/WORKLOAD NAME/'
. These are the current
supported stages:
setup: This stage is where the application is loaded (if present). It is a good place to record an revent here to perform any tasks to get ready for the main part of the workload to start. run: This stage is where the main work of the workload should be performed. This will allow for more accurate results if the revent file for this stage only records the main actions under test. extract_results: This stage is used after the workload has been completed to retrieve any metrics from the workload e.g. a score. teardown: This stage is where any final actions should be performed to clean up the workload.
Only the run stage is mandatory, the remaining stages will be replayed if a recording is present otherwise no actions will be performed for that particular stage.
For instance, to add a custom revent files for a device named “mydevice” and
a workload name “myworkload”, you need to add the revent files to the directory
/home/$WA_USER_HOME/dependencies/myworkload/revent_files
creating it if
necessary.
mydevice.setup.revent
mydevice.run.revent
mydevice.extract_results.revent
mydevice.teardown.revent
Any revent file in the dependencies will always overwrite the revent file in the workload directory. So for example it is possible to just provide one revent for setup in the dependencies and use the run.revent that is in the workload directory.
File format of revent recordings¶
You do not need to understand recording format in order to use revent. This section is intended for those looking to extend revent in some way, or to utilize revent recordings for other purposes.
Format Overview¶
Recordings are stored in a binary format. A recording consists of three sections:
+-+-+-+-+-+-+-+-+-+-+-+
| Header |
+-+-+-+-+-+-+-+-+-+-+-+
| |
| Device Description |
| |
+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| Event Stream |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+
The header contains metadata describing the recording. The device description contains information about input devices involved in this recording. Finally, the event stream contains the recorded input events.
All fields are either fixed size or prefixed with their length or the number of (fixed-sized) elements.
Note
All values below are little endian
Recording Header¶
An revent recoding header has the following structure
It starts with the “magic” string
REVENT
to indicate that this is an revent recording.The magic is followed by a 16 bit version number. This indicates the format version of the recording that follows. Current version is
2
.The next 16 bits indicate the type of the recording. This dictates the structure of the Device Description section. Valid values are:
0
This is a general input event recording. The device description contains a list of paths from which the events where recorded.
1
This a gamepad recording. The device description contains the description of the gamepad used to create the recording.
The header is zero-padded to 128 bits.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'R' | 'E' | 'V' | 'E' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'N' | 'T' | Version |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Mode | PADDING |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PADDING |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Device Description¶
This section describes the input devices used in the recording. Its structure is
determined by the value of Mode
field in the header.
General Recording¶
Note
This is the only format supported prior to version 2
.
The recording has been made from all available input devices. This section
contains the list of /dev/input
paths for the devices, prefixed with total
number of the devices recorded.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of devices |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Device paths +-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Similarly, each device path is a length-prefixed string. Unlike C strings, the path is not NULL-terminated.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of device path |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Device path |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Gamepad Recording¶
The recording has been made from a specific gamepad. All events in the stream
will be for that device only. The section describes the device properties that
will be used to create a virtual input device using /dev/uinput
. Please
see linux/input.h
header in the Linux kernel source for more information
about the fields in this section.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| bustype | vendor |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| product | version |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| name_length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| name |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ev_bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| key_bits (96 bytes) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| rel_bits (96 bytes) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| abs_bits (96 bytes) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| num_absinfo |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| |
| |
| absinfo entries |
| |
| |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Each absinfo
entry consists of six 32 bit values. The number of entries is
determined by the abs_bits
field.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| value |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| minimum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| maximum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fuzz |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flat |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| resolution |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Event Stream¶
The majority of an revent recording will be made up of the input events that were recorded. The event stream is prefixed with the number of events in the stream, and start and end times for the recording.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of events |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of events (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Seconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Microseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Microseconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Seconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Microseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Microseconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| Events |
| |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Event Structure¶
Each event entry structured as follows:
- An unsigned short integer representing which device from the list of device paths this event is for (zero indexed). E.g. Device ID = 3 would be the 4th device in the list of device paths.
- A unsigned long integer representing the number of seconds since “epoch” when the event was recorded.
- A unsigned long integer representing the microseconds part of the timestamp.
- An unsigned integer representing the event type
- An unsigned integer representing the event code
- An unsigned integer representing the event value
For more information about the event type, code and value please read: https://www.kernel.org/doc/Documentation/input/event-codes.txt
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Device ID | Timestamp Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Seconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Seconds (cont.) | stamp Micoseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Micoseconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Micoseconds (cont.) | Event Type |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Event Code | Event Value |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Event Value (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Parser¶
WA has a parser for revent recordings. This can be used to work with revent recordings in scripts. Here is an example:
from wa.utils.revent import ReventRecording
with ReventRecording('/path/to/recording.revent') as recording:
print "Recording: {}".format(recording.filepath)
print "There are {} input events".format(recording.num_events)
print "Over a total of {} seconds".format(recording.duration)
Serialization¶
Overview of Serialization¶
WA employs a serialization mechanism in order to store some of its internal structures inside the output directory. Serialization is performed in two stages:
- A serializable object is converted into a POD (Plain Old Data) structure consisting of primitive Python types, and a few additional types (see WA POD Types below).
- The POD structure is serialized into a particular format by a generic parser for that format. Currently, yaml and json are supported.
Deserialization works in reverse order – first the serialized text is parsed into a POD, which is then converted to the appropriate object.
Implementing Serializable Objects¶
In order to be considered serializable, an object must either be a POD, or it
must implement the to_pod()
method and from_pod
static/class method,
which will perform the conversion to/form pod.
As an example, below as a (somewhat trimmed) implementation of the Event
class:
class Event(object):
@staticmethod
def from_pod(pod):
instance = Event(pod['message'])
instance.timestamp = pod['timestamp']
return instance
def __init__(self, message):
self.timestamp = datetime.utcnow()
self.message = message
def to_pod(self):
return dict(
timestamp=self.timestamp,
message=self.message,
)
Serialization API¶
-
read_pod
(source, fmt=None)¶
-
write_pod
(pod, dest, fmt=None)¶ These read and write PODs from a file. The format will be inferred, if possible, from the extension of the file, or it may be specified explicitly with
fmt
.source
anddest
can be either strings, in which case they will be interpreted as paths, or they can be file-like objects.
-
is_pod
(obj)¶ Returns
True
ifobj
is a POD, andFalse
otherwise.
-
dump
(o, wfh, fmt='json', *args, **kwargs)¶
-
load
(s, fmt='json', *args, **kwargs)¶ These implment an altenative serialization interface, which matches the interface exposed by the parsers for the supported formats.
WA POD Types¶
POD types are types that can be handled by a serializer directly, without a need for any additional information. These consist of the build-in python types
list
tuple
dict
set
str
unicode
int
float
bool
…the standard library types
OrderedDict
datetime
…and the WA-defined types
regex_type
none_type
level
cpu_mask
Any structure consisting entirely of these types is a POD and can be serialized and then deserialized without losing information. It is important to note that only these specific types are considered POD, their subclasses are not.
Note
dict
s get deserialized as OrderedDict
s.
Serialization Formats¶
WA utilizes two serialization formats: YAML and JSON. YAML is used for files intended to be primarily written and/or read by humans; JSON is used for files intended to be primarily written and/or read by WA and other programs.
The parsers and serializers for these formats used by WA have been modified to handle additional types (e.g. regular expressions) that are typically not supported by the formats. This was done in such a way that the resulting files are still valid and can be parsed by any parser for that format.
Contributing¶
Code¶
We welcome code contributions via GitHub pull requests. To help with maintainability of the code line we ask that the code uses a coding style consistent with the rest of WA code. Briefly, it is
- PEP8 with line length and block
comment rules relaxed (the wrapper for PEP8 checker inside
dev_scripts
will run it with appropriate configuration). - Four-space indentation (no tabs!).
- Title-case for class names, underscore-delimited lower case for functions, methods, and variables.
- Use descriptive variable names. Delimit words with
'_'
for readability. Avoid shortening words, skipping vowels, etc (common abbreviations such as “stats” for “statistics”, “config” for “configuration”, etc are OK). Do not use Hungarian notation (so preferbirth_date
overdtBirth
).
New extensions should also follow implementation guidelines specified in the Writing Plugins section of the documentation.
We ask that the following checks are performed on the modified code prior to submitting a pull request:
Note
You will need pylint and pep8 static checkers installed:
pip install pep8
pip install pylint
It is recommended that you install via pip rather than through your distribution’s package manager because the latter is likely to contain out-of-date version of these tools.
./dev_scripts/pylint
should be run without arguments and should produce no output (any output should be addressed by making appropriate changes in the code or adding a pylint ignore directive, if there is a good reason for keeping the code as is)../dev_scripts/pep8
should be run without arguments and should produce no output (any output should be addressed by making appropriate changes in the code).If the modifications touch core framework (anything under
wa/framework
), unit tests should be run usingnosetests
, and they should all pass.- If significant additions have been made to the framework, unit tests should be added to cover the new functionality.
If modifications have been made to documentation (this includes description attributes for Parameters and Extensions), documentation should be built to make sure no errors or warning during build process, and a visual inspection of new/updated sections in resulting HTML should be performed to ensure everything renders as expected.
Once you have your contribution is ready, please follow instructions in GitHub documentation to create a pull request.
Documentation¶
Headings¶
To allow for consistent headings to be used through out the document the following character sequences should be used when creating headings
=========
Heading 1
=========
Only used for top level headings which should also have an entry in the
navigational side bar.
*********
Heading 2
*********
Main page heading used for page title, should not have a top level entry in the
side bar.
Heading 3
==========
Regular section heading.
Heading 4
---------
Sub-heading.
Heading 5
~~~~~~~~~
Heading 6
^^^^^^^^^
Heading 7
"""""""""
Configuration Listings¶
To keep a consistent style for presenting configuration options, the preferred style is to use a Field List.
(See: http://docutils.sourceforge.net/docs/user/rst/quickref.html#field-lists)
Example:
:parameter: My Description
Will render as:
parameter: My Description
API Style¶
When documenting an API the currently preferred style is to provide a short description of the class, followed by the attributes of the class in a Definition List followed by the methods using the method directive.
(See: http://docutils.sourceforge.net/docs/user/rst/quickref.html#definition-lists)
Example:
API
===
:class:`MyClass`
----------------
:class:`MyClass` is an example class to demonstrate API documentation.
``attribute1``
The first attribute of the example class.
``attribute2``
Another attribute example.
methods
"""""""
.. method:: MyClass.retrieve_output(name)
Retrieve the output for ``name``.
:param name: The output that should be returned.
:return: An :class:`Output` object for ``name``.
:raises NotFoundError: If no output can be found.
Will render as:
MyClass
is an example class to demonstrate API documentation.
attribute1
- The first attribute of the example class.
attribute2
- Another attribute example.