Writing Extensions¶

Workload Automation offers several extension points (or plugin types).The most interesting of these are

result processors:
workloads:	These are the tasks that get executed and measured on the device. These can be benchmarks, high-level use cases, or pretty much anything else.
devices:	These are interfaces to the physical devices (development boards or end-user devices, such as smartphones) that use cases run on. Typically each model of a physical device would require its own interface class (though some functionality may be reused by subclassing from an existing base).
instruments:	Instruments allow collecting additional data from workload execution (e.g. system traces). Instruments are not specific to a particular Workload. Instruments can hook into any stage of workload execution.
	These are used to format the results of workload execution once they have been collected. Depending on the callback used, these will run either after each iteration or at the end of the run, after all of the results have been collected.

You create an extension by subclassing the appropriate base class, defining appropriate methods and attributes, and putting the .py file with the class into an appropriate subdirectory under ~/.workload_automation (there is one for each extension type).

Extension Basics¶

This sub-section covers things common to implementing extensions of all types. It is recommended you familiarize yourself with the information here before proceeding onto guidance for specific extension types.

To create an extension, you basically subclass an appropriate base class and them implement the appropriate methods

The Context¶

The majority of methods in extensions accept a context argument. This is an instance of wlauto.core.execution.ExecutionContext. If contains of information about current state of execution of WA and keeps track of things like which workload is currently running and the current iteration.

Notable attributes of the context are

context.spec: the current workload specification being executed. This is an instance of wlauto.core.configuration.WorkloadRunSpec and defines the workload and the parameters under which it is being executed.
context.workload: Workload object that is currently being executed.
context.current_iteration: The current iteration of the spec that is being executed. Note that this is the iteration for that spec, i.e. the number of times that spec has been run, not the total number of all iterations have been executed so far.
context.result: This is the result object for the current iteration. This is an instance of wlauto.core.result.IterationResult. It contains the status of the iteration as well as the metrics and artifacts generated by the workload and enable instrumentation.
context.device: The device interface object that can be used to interact with the device. Note that workloads and instruments have their own device attribute and they should be using that instead.

In addition to these, context also defines a few useful paths (see below).

Paths¶

You should avoid using hard-coded absolute paths in your extensions whenever possible, as they make your code too dependent on a particular environment and may mean having to make adjustments when moving to new (host and/or device) platforms. To help avoid hard-coded absolute paths, WA automation defines a number of standard locations. You should strive to define your paths relative to one of those.

On the host¶

Host paths are available through the context object, which is passed to most extension methods.

context.run_output_directory: This is the top-level output directory for all WA results (by default, this will be “wa_output” in the directory in which WA was invoked.
context.output_directory: This is the output directory for the current iteration. This will an iteration-specific subdirectory under the main results location. If there is no current iteration (e.g. when processing overall run results) this will point to the same location as root_output_directory.
context.host_working_directory: This an addition location that may be used by extensions to store non-iteration specific intermediate files (e.g. configuration).

Additionally, the global wlauto.settings object exposes on other location:

settings.dependency_directory: this is the root directory for all extension dependencies (e.g. media files, assets etc) that are not included within the extension itself.

As per Python best practice, it is recommended that methods and values in os.path standard library module are used for host path manipulation.

On the device¶

Workloads and instruments have a device attribute, which is an interface to the device used by WA. It defines the following location:

device.working_directory: This is the directory for all WA-related files on the device. All files deployed to the device should be pushed to somewhere under this location (the only exception being executables installed with device.install method).

Since there could be a mismatch between path notation used by the host and the device, the os.path modules should not be used for on-device path manipulation. Instead device has an equipment module exposed through device.path attribute. This has all the same attributes and behaves the same way as os.path, but is guaranteed to produce valid paths for the device, irrespective of the host’s path notation. For example:

result_file = self.device.path.join(self.device.working_directory, "result.txt")
self.command = "{} -a -b -c {}".format(target_binary, result_file)

Note

result processors, unlike workloads and instruments, do not have their own device attribute; however they can access the device through the context.

Deploying executables to a device¶

Some devices may have certain restrictions on where executable binaries may be placed and how they should be invoked. To ensure your extension works with as wide a range of devices as possible, you should use WA APIs for deploying and invoking executables on a device, as outlined below.

As with other resources (see Dynamic Resource Resolution) , host-side paths to the exectuable: binary to be deployed should be obtained via the resource resolver. A special resource type, Executable is used to identify a binary to be deployed. This is simiar to the regular File resource, however it takes an additional parameter that specifies the ABI for which executable was compiled.

In order for the binary to be obtained in this way, it must be stored in one of the locations scanned by the resource resolver in a directry structure <root>/bin/<abi>/<binary> (where root is the base resource location to be searched, e.g. ~/.workload_automation/depencencies/<extension name>, and <abi> is the ABI for which the exectuable has been compiled, as returned by self.device.abi).

Once the path to the host-side binary has been obtained, it may be deployed using one of two methods of a Device instace – install or install_if_needed. The latter will check a version of that binary has been perviously deployed by WA and will not try to re-install.

from wlauto import Executable

host_binary = context.resolver.get(Executable(self, self.device.abi, 'some_binary'))
target_binary = self.device.install_if_needed(host_binary)

Note

Please also note that the check is done based solely on the binary name. For more information please see: wlauto.common.linux.BaseLinuxDevice.install_if_needed()

Both of the above methods will return the path to the installed binary on the device. The executable should be invoked only via that path; do not assume that it will be in PATH on the target (or that the executable with the same name in PATH is the version deployed by WA.

self.command = "{} -a -b -c".format(target_binary)
self.device.execute(self.command)

Parameters¶

All extensions can be parameterized. Parameters are specified using parameters class attribute. This should be a list of wlauto.core.Parameter instances. The following attributes can be specified on parameter creation:

name

This is the only mandatory argument. The name will be used to create a corresponding attribute in the extension instance, so it must be a valid Python identifier.

kind

This is the type of the value of the parameter. This could be an callable. Normally this should be a standard Python type, e.g. int` or ``float, or one the types defined in wlauto.utils.types. If not explicitly specified, this will default to str.

Note

Irrespective of the kind specified, None is always a valid value for a parameter. If you don’t want to allow None, then set mandatory (see below) to True.

allowed_values

A list of the only allowed values for this parameter.

Note

For composite types, such as list_of_strings or list_of_ints in wlauto.utils.types, each element of the value will be checked against allowed_values rather than the composite value itself.

default

The default value to be used for this parameter if one has not been specified by the user. Defaults to None.

mandatory

A bool indicating whether this parameter is mandatory. Setting this to True will make None an illegal value for the parameter. Defaults to False.

Note

Specifying a default will mean that this parameter will, effectively, be ignored (unless the user sets the param to None).

Note

Mandatory parameters are bad. If at all possible, you should strive to provide a sensible default or to make do without the parameter. Only when the param is absolutely necessary, and there really is no sensible default that could be given (e.g. something like login credentials), should you consider making it mandatory.

constraint

This is an additional constraint to be enforced on the parameter beyond its type or fixed allowed values set. This should be a predicate (a function that takes a single argument – the user-supplied value – and returns a bool indicating whether the constraint has been satisfied).

override

A parameter name must be unique not only within an extension but also with that extension’s class hierarchy. If you try to declare a parameter with the same name as already exists, you will get an error. If you do want to override a parameter from further up in the inheritance hierarchy, you can indicate that by setting override attribute to True.

When overriding, you do not need to specify every other attribute of the parameter, just the ones you what to override. Values for the rest will be taken from the parameter in the base class.

Validation and cross-parameter constraints¶

An extension will get validated at some point after constructions. When exactly this occurs depends on the extension type, but it will be validated before it is used.

You can implement validate method in your extension (that takes no arguments beyond the self) to perform any additions internal validation in your extension. By “internal”, I mean that you cannot make assumptions about the surrounding environment (e.g. that the device has been initialized).

The contract for validate method is that it should raise an exception (either wlauto.exceptions.ConfigError or extension-specific exception type – see further on this page) if some validation condition has not, and cannot, been met. If the method returns without raising an exception, then the extension is in a valid internal state.

Note that validate can be used not only to verify, but also to impose a valid internal state. In particular, this where cross-parameter constraints can be resolved. If the default or allowed_values of one parameter depend on another parameter, there is no way to express that declaratively when specifying the parameters. In that case the dependent attribute should be left unspecified on creation and should instead be set inside validate.

Logging¶

Every extension class has it’s own logger that you can access through self.logger inside the extension’s methods. Generally, a Device will log everything it is doing, so you shouldn’t need to add much additional logging in your expansion’s. But you might what to log additional information, e.g. what settings your extension is using, what it is doing on the host, etc. Operations on the host will not normally be logged, so your extension should definitely log what it is doing on the host. One situation in particular where you should add logging is before doing something that might take a significant amount of time, such as downloading a file.

Documenting¶

All extensions and their parameter should be documented. For extensions themselves, this is done through description class attribute. The convention for an extension description is that the first paragraph should be a short summary description of what the extension does and why one would want to use it (among other things, this will get extracted and used by wa list command). Subsequent paragraphs (separated by blank lines) can then provide a more detailed description, including any limitations and setup instructions.

For parameters, the description is passed as an argument on creation. Please note that if default, allowed_values, or constraint, are set in the parameter, they do not need to be explicitly mentioned in the description (wa documentation utilities will automatically pull those). If the default is set in validate or additional cross-parameter constraints exist, this should be documented in the parameter description.

Both extensions and their parameters should be documented using reStructureText markup (standard markup for Python documentation). See:

http://docutils.sourceforge.net/rst.html

Aside from that, it is up to you how you document your extension. You should try to provide enough information so that someone unfamiliar with your extension is able to use it, e.g. you should document all settings and parameters your extension expects (including what the valid value are).

Error Notification¶

When you detect an error condition, you should raise an appropriate exception to notify the user. The exception would typically be ConfigError or (depending the type of the extension) WorkloadError/DeviceError/InstrumentError/ResultProcessorError. All these errors are defined in wlauto.exception module.

ConfigError should be raised where there is a problem in configuration specified by the user (either through the agenda or config files). These errors are meant to be resolvable by simple adjustments to the configuration (and the error message should suggest what adjustments need to be made. For all other errors, such as missing dependencies, mis-configured environment, problems performing operations, etc., the extension type-specific exceptions should be used.

If the extension itself is capable of recovering from the error and carrying on, it may make more sense to log an ERROR or WARNING level message using the extension’s logger and to continue operation.

Utils¶

Workload Automation defines a number of utilities collected under wlauto.utils subpackage. These utilities were created to help with the implementation of the framework itself, but may be also be useful when implementing extensions.

Adding a Workload¶

Note

You can use wa create workload [name] script to generate a new workload structure for you. This script can also create the boilerplate for UI automation, if your workload needs it. See wa create -h for more details.

New workloads can be added by subclassing wlauto.core.workload.Workload

The Workload class defines the following interface:

class Workload(Extension):

    name = None

    def init_resources(self, context):
        pass

    def validate(self):
        pass

    def initialize(self, context):
        pass

    def setup(self, context):
        pass

    def setup(self, context):
        pass

    def run(self, context):
        pass

    def update_result(self, context):
        pass

    def teardown(self, context):
        pass

    def finalize(self, context):
        pass

Note

Please see Conventions section for notes on how to interpret this.

The interface should be implemented as follows

name:
This identifies the workload (e.g. it used to specify it in the agenda.

init_resources:
This method may be optionally override to implement dynamic resource discovery for the workload. This method executes early on, before the device has been initialized, so it should only be used to initialize resources that do not depend on the device to resolve. This method is executed once per run for each workload instance.

validate:
This method can be used to validate any assumptions your workload makes about the environment (e.g. that required files are present, environment variables are set, etc) and should raise a wlauto.exceptions.WorkloadError if that is not the case. The base class implementation only makes sure sure that the name attribute has been set.

initialize:
This method will be executed exactly once per run (no matter how many instances of the workload there are). It will run after the device has been initialized, so it may be used to perform device-dependent initialization that does not need to be repeated on each iteration (e.g. as installing executables required by the workload on the device).

setup:
Everything that needs to be in place for workload execution should be done in this method. This includes copying files to the device, starting up an application, configuring communications channels, etc.

run:
This method should perform the actual task that is being measured. When this method exits, the task is assumed to be complete.

Note

Instrumentation is kicked off just before calling this method and is disabled right after, so everything in this method is being measured. Therefore this method should contain the least code possible to perform the operations you are interested in measuring. Specifically, things like installing or starting applications, processing results, or copying files to/from the device should be done elsewhere if possible.

update_result:
This method gets invoked after the task execution has finished and should be used to extract metrics and add them to the result (see below).

teardown:
This could be used to perform any cleanup you may wish to do, e.g. Uninstalling applications, deleting file on the device, etc.

finalize:
This is the complement to initialize. This will be executed exactly once at the end of the run. This should be used to perform any final clean up (e.g. uninstalling binaries installed in the initialize).

Workload methods (except for validate) take a single argument that is a wlauto.core.execution.ExecutionContext instance. This object keeps track of the current execution state (such as the current workload, iteration number, etc), and contains, among other things, a wlauto.core.workload.WorkloadResult instance that should be populated from the update_result method with the results of the execution.

# ...

def update_result(self, context):
   # ...
   context.result.add_metric('energy', 23.6, 'Joules', lower_is_better=True)

# ...

Example¶

This example shows a simple workload that times how long it takes to compress a file of a particular size on the device.

Note

This is intended as an example of how to implement the Workload interface. The methodology used to perform the actual measurement is not necessarily sound, and this Workload should not be used to collect real measurements.

import os
from wlauto import Workload, Parameter

class ZiptestWorkload(Workload):

    name = 'ziptest'
    description = '''
                  Times how long it takes to gzip a file of a particular size on a device.

                  This workload was created for illustration purposes only. It should not be
                  used to collect actual measurements.

                  '''

    parameters = [
            Parameter('file_size', kind=int, default=2000000,
                      description='Size of the file (in bytes) to be gzipped.')
    ]

    def setup(self, context):
            # Generate a file of the specified size containing random garbage.
            host_infile = os.path.join(context.output_directory, 'infile')
            command = 'openssl rand -base64 {} > {}'.format(self.file_size, host_infile)
            os.system(command)
            # Set up on-device paths
            devpath = self.device.path  # os.path equivalent for the device
            self.device_infile = devpath.join(self.device.working_directory, 'infile')
            self.device_outfile = devpath.join(self.device.working_directory, 'outfile')
            # Push the file to the device
            self.device.push_file(host_infile, self.device_infile)

    def run(self, context):
            self.device.execute('cd {} && (time gzip {}) &>> {}'.format(self.device.working_directory,
                                                                    self.device_infile,
                                                                    self.device_outfile))

    def update_result(self, context):
            # Pull the results file to the host
            host_outfile = os.path.join(context.output_directory, 'outfile')
            self.device.pull_file(self.device_outfile, host_outfile)
            # Extract metrics form the file's contents and update the result
            # with them.
            content = iter(open(host_outfile).read().strip().split())
            for value, metric in zip(content, content):
            mins, secs = map(float, value[:-1].split('m'))
            context.result.add_metric(metric, secs + 60 * mins)

    def teardown(self, context):
            # Clean up on-device file.
            self.device.delete_file(self.device_infile)
            self.device.delete_file(self.device_outfile)

Adding revent-dependent Workload:¶

wlauto.common.game.GameWorkload is the base class for all the workloads that depend on revent files. It implements all the methods needed to push the files to the device and run them. New GameWorkload can be added by subclassing wlauto.common.game.GameWorkload:

The GameWorkload class defines the following interface:

class GameWorkload(Workload):

    name = None
    package = None
    activity = None

The interface should be implemented as follows

name: This identifies the workload (e.g. it used to specify it in the agenda.

package: This is the name of the ‘.apk’ package without its file extension.

activity: The name of the main activity that runs the package.

Example:¶

This example shows a simple GameWorkload that plays a game.

from wlauto.common.game import GameWorkload

class MyGame(GameWorkload):

    name = 'mygame'
    package = 'com.mylogo.mygame'
    activity = 'myActivity.myGame'

Convention for Naming revent Files for `wlauto.common.game.GameWorkload`¶

There is a convention for naming revent files which you should follow if you want to record your own revent files. Each revent file must start with the device name(case sensitive) then followed by a dot ‘.’ then the stage name then ‘.revent’. All your custom revent files should reside at ‘~/.workload_automation/dependencies/WORKLOAD NAME/’. These are the current supported stages:

setup: This stage is where the game is loaded. It is a good place to record revent here to modify the game settings and get it ready to start.

run: This stage is where the game actually starts. This will allow for more accurate results if the revent file for this stage only records the game being played.

For instance, to add a custom revent files for a device named mydevice and a workload name mygame, you create a new directory called mygame in ‘~/.workload_automation/dependencies/’. Then you add the revent files for the stages you want in ~/.workload_automation/dependencies/mygame/:

mydevice.setup.revent
mydevice.run.revent

Any revent file in the dependencies will always overwrite the revent file in the workload directory. So it is possible for example to just provide one revent for setup in the dependencies and use the run.revent that is in the workload directory.

Adding an Instrument¶

Instruments can be used to collect additional measurements during workload execution (e.g. collect power readings). An instrument can hook into almost any stage of workload execution. A typical instrument would implement a subset of the following interface:

class Instrument(Extension):

    name = None
    description = None

    parameters = [
    ]

    def initialize(self, context):
        pass

    def setup(self, context):
        pass

    def start(self, context):
        pass

    def stop(self, context):
        pass

    def update_result(self, context):
        pass

    def teardown(self, context):
        pass

    def finalize(self, context):
        pass

This is similar to a Workload, except all methods are optional. In addition to the workload-like methods, instruments can define a number of other methods that will get invoked at various points during run execution. The most useful of which is perhaps initialize that gets invoked after the device has been initialised for the first time, and can be used to perform one-time setup (e.g. copying files to the device – there is no point in doing that for each iteration). The full list of available methods can be found in Signals Documentation.

Prioritization¶

Callbacks (e.g. setup() methods) for all instrumentation get executed at the same point during workload execution, one after another. The order in which the callbacks get invoked should be considered arbitrary and should not be relied on (e.g. you cannot expect that just because instrument A is listed before instrument B in the config, instrument A’s callbacks will run first).

In some cases (e.g. in start() and stop() methods), it is important to ensure that a particular instrument’s callbacks run a closely as possible to the workload’s invocations in order to maintain accuracy of readings; or, conversely, that a callback is executed after the others, because it takes a long time and may throw off the accuracy of other instrumentation. You can do this by prepending fast_ or slow_ to your callbacks’ names. For example:

class PreciseInstrument(Instument):

    # ...

    def fast_start(self, context):
        pass

    def fast_stop(self, context):
        pass

    # ...

PreciseInstrument will be started after all other instrumentation (i.e. just before the workload runs), and it will stopped before all other instrumentation (i.e. just after the workload runs). It is also possible to use very_fast_ and very_slow_ prefixes when you want to be really sure that your callback will be the last/first to run.

If more than one active instrument have specified fast (or slow) callbacks, then their execution order with respect to each other is not guaranteed. In general, having a lot of instrumentation enabled is going to necessarily affect the readings. The best way to ensure accuracy of measurements is to minimize the number of active instruments (perhaps doing several identical runs with different instruments enabled).

Example¶

Below is a simple instrument that measures the execution time of a workload:

class ExecutionTimeInstrument(Instrument):
    """
    Measure how long it took to execute the run() methods of a Workload.

    """

    name = 'execution_time'

    def initialize(self, context):
        self.start_time = None
        self.end_time = None

    def fast_start(self, context):
        self.start_time = time.time()

    def fast_stop(self, context):
        self.end_time = time.time()

    def update_result(self, context):
        execution_time = self.end_time - self.start_time
        context.result.add_metric('execution_time', execution_time, 'seconds')

Adding a Result Processor¶

A result processor is responsible for processing the results. This may involve formatting and writing them to a file, uploading them to a database, generating plots, etc. WA comes with a few result processors that output results in a few common formats (such as csv or JSON).

You can add your own result processors by creating a Python file in ~/.workload_automation/result_processors with a class that derives from wlauto.core.result.ResultProcessor, which has the following interface:

class ResultProcessor(Extension):

    name = None
    description = None

    parameters = [
    ]

    def initialize(self, context):
            pass

    def process_iteration_result(self, result, context):
            pass

    def export_iteration_result(self, result, context):
            pass

    def process_run_result(self, result, context):
            pass

    def export_run_result(self, result, context):
            pass

    def finalize(self, context):
            pass

The method names should be fairly self-explanatory. The difference between “process” and “export” methods is that export methods will be invoke after process methods for all result processors have been generated. Process methods may generated additional artifacts (metrics, files, etc), while export methods should not – the should only handle existing results (upload them to a database, archive on a filer, etc).

The result object passed to iteration methods is an instance of wlauto.core.result.IterationResult, the result object passed to run methods is an instance of wlauto.core.result.RunResult. Please refer to their API documentation for details.

Example¶

Here is an example result processor that formats the results as a column-aligned table:

import os
from wlauto import ResultProcessor
from wlauto.utils.misc import write_table


class Table(ResultProcessor):

    name = 'table'
    description = 'Gerates a text file containing a column-aligned table with run results.'

    def process_run_result(self, result, context):
        rows = []
        for iteration_result in result.iteration_results:
            for metric in iteration_result.metrics:
                rows.append([metric.name, str(metric.value), metric.units or '',
                            metric.lower_is_better  and '-' or '+'])

        outfile =  os.path.join(context.output_directory, 'table.txt')
        with open(outfile, 'w') as wfh:
            write_table(rows, wfh)

Adding a Resource Getter¶

A resource getter is a new extension type added in version 2.1.3. A resource getter implement a method of acquiring resources of a particular type (such as APK files or additional workload assets). Resource getters are invoked in priority order until one returns the desired resource.

If you want WA to look for resources somewhere it doesn’t by default (e.g. you have a repository of APK files), you can implement a getter for the resource and register it with a higher priority than the standard WA getters, so that it gets invoked first.

Instances of a resource getter should implement the following interface:

class ResourceGetter(Extension):

    name = None
    resource_type = None
    priority = GetterPriority.environment

    def get(self, resource, **kwargs):
        raise NotImplementedError()

The getter should define a name (as with all extensions), a resource type, which should be a string, e.g. 'jar', and a priority (see Getter Prioritization below). In addition, get method should be implemented. The first argument is an instance of wlauto.core.resource.Resource representing the resource that should be obtained. Additional keyword arguments may be used by the invoker to provide additional information about the resource. This method should return an instance of the resource that has been discovered (what “instance” means depends on the resource, e.g. it could be a file path), or None if this getter was unable to discover that resource.

Getter Prioritization¶

A priority is an integer with higher numeric values indicating a higher priority. The following standard priority aliases are defined for getters:

cached: The cached version of the resource. Look here first. This priority also implies that the resource at this location is a “cache” and is not the only version of the resource, so it may be cleared without losing access to the resource.

preferred: Take this resource in favour of the environment resource.

environment: Found somewhere under ~/.workload_automation/ or equivalent, or from environment variables, external configuration files, etc. These will override resource supplied with the package.

package: Resource provided with the package.

remote: Resource will be downloaded from a remote location (such as an HTTP server or a samba share). Try this only if no other getter was successful.

These priorities are defined as class members of wlauto.core.resource.GetterPriority, e.g. GetterPriority.cached.

Most getters in WA will be registered with either environment or package priorities. So if you want your getter to override the default, it should typically be registered as preferred.

You don’t have to stick to standard priority levels (though you should, unless there is a good reason). Any integer is a valid priority. The standard priorities range from -20 to 20 in increments of 10.

Example¶

The following is an implementation of a getter for a workload APK file that looks for the file under ~/.workload_automation/dependencies/<workload_name>:

import os
import glob

from wlauto import ResourceGetter, GetterPriority, settings
from wlauto.exceptions import ResourceError


class EnvironmentApkGetter(ResourceGetter):

    name =  'environment_apk'
    resource_type = 'apk'
    priority = GetterPriority.environment

    def get(self, resource):
        resource_dir = _d(os.path.join(settings.dependency_directory, resource.owner.name))
        version = kwargs.get('version')
        found_files = glob.glob(os.path.join(resource_dir, '*.apk'))
        if version:
            found_files = [ff for ff in found_files if version.lower() in ff.lower()]
        if len(found_files) == 1:
            return found_files[0]
        elif not found_files:
            return None
        else:
            raise ResourceError('More than one .apk found in {} for {}.'.format(resource_dir,
                                                                                resource.owner.name))

Adding a Device¶

At the moment, only Android devices are supported. Most of the functionality for interacting with a device is implemented in wlauto.common.AndroidDevice and is exposed through generic_android device interface, which should suffice for most purposes. The most common area where custom functionality may need to be implemented is during device initialization. Usually, once the device gets to the Android home screen, it’s just like any other Android device (modulo things like differences between Android versions).

If your device doesn’t not work with generic_device interface and you need to write a custom interface to handle it, you would do that by subclassing AndroidDevice and then just overriding the methods you need. Typically you will want to override one or more of the following:

reset: Trigger a device reboot. The default implementation just sends adb reboot to the device. If this command does not work, an alternative implementation may need to be provided.
hard_reset: This is a harsher reset that involves cutting the power to a device (e.g. holding down power button or removing battery from a phone). The default implementation is a no-op that just sets some internal flags. If you’re dealing with unreliable prototype hardware that can crash and become unresponsive, you may want to implement this in order for WA to be able to recover automatically.
connect: When this method returns, adb connection to the device has been established. This gets invoked after a reset. The default implementation just waits for the device to appear in the adb list of connected devices. If this is not enough (e.g. your device is connected via Ethernet and requires an explicit adb connect call), you may wish to override this to perform the necessary actions before invoking the AndroidDevices version.
init: This gets called once at the beginning of the run once the connection to the device has been established. There is no default implementation. It’s there to allow whatever custom initialisation may need to be performed for the device (setting properties, configuring services, etc).

Please refer to the API documentation for wlauto.common.AndroidDevice for the full list of its methods and their functionality.

Other Extension Types¶

In addition to extension types covered above, there are few other, more specialized ones. They will not be covered in as much detail. Most of them expose relatively simple interfaces with only a couple of methods and it is expected that if the need arises to extend them, the API-level documentation that accompanies them, in addition to what has been outlined here, should provide enough guidance.

commands:	This allows extending WA with additional sub-commands (to supplement exiting ones outlined in the invocation section).
modules:	Modules are “extensions for extensions”. They can be loaded by other extensions to expand their functionality (for example, a flashing module maybe loaded by a device in order to support flashing).

Packaging Your Extensions¶

If your have written a bunch of extensions, and you want to make it easy to deploy them to new systems and/or to update them on existing systems, you can wrap them in a Python package. You can use wa create package command to generate appropriate boiler plate. This will create a setup.py and a directory for your package that you can place your extensions into.

For example, if you have a workload inside my_workload.py and a result processor in my_result_processor.py, and you want to package them as my_wa_exts package, first run the create command

wa create package my_wa_exts

This will create a my_wa_exts directory which contains a my_wa_exts/setup.py and a subdirectory my_wa_exts/my_wa_exts which is the package directory for your extensions (you can rename the top-level my_wa_exts directory to anything you like – it’s just a “container” for the setup.py and the package directory). Once you have that, you can then copy your extensions into the package directory, creating my_wa_exts/my_wa_exts/my_workload.py and my_wa_exts/my_wa_exts/my_result_processor.py. If you have a lot of extensions, you might want to organize them into subpackages, but only the top-level package directory is created by default, and it is OK to have everything in there.

Note

When discovering extensions thorugh this mechanism, WA traveries the Python module/submodule tree, not the directory strucuter, therefore, if you are going to create subdirectories under the top level dictory created for you, it is important that your make sure they are valid Python packages; i.e. each subdirectory must contain a __init__.py (even if blank) in order for the code in that directory and its subdirectories to be discoverable.

At this stage, you may want to edit params structure near the bottom of the setup.py to add correct author, license and contact information (see “Writing the Setup Script” section in standard Python documentation for details). You may also want to add a README and/or a COPYING file at the same level as the setup.py. Once you have the contents of your package sorted, you can generate the package by running

cd my_wa_exts
python setup.py sdist

This will generate my_wa_exts/dist/my_wa_exts-0.0.1.tar.gz package which can then be deployed on the target system with standard Python package management tools, e.g.

sudo pip  install my_wa_exts-0.0.1.tar.gz

As part of the installation process, the setup.py in the package, will write the package’s name into ~/.workoad_automoation/packages. This will tell WA that the package contains extension and it will load them next time it runs.

Note

There are no unistall hooks in setuputils, so if you ever uninstall your WA extensions package, you will have to manually remove it from ~/.workload_automation/packages otherwise WA will complain abou a missing package next time you try to run it.

Writing Extensions¶

Extension Basics¶

The Context¶

Paths¶

On the host¶

On the device¶

Deploying executables to a device¶

Parameters¶

Validation and cross-parameter constraints¶

Logging¶

Documenting¶

Error Notification¶

Utils¶

Adding a Workload¶

Example¶

Adding revent-dependent Workload:¶

Example:¶

Convention for Naming revent Files for `wlauto.common.game.GameWorkload`¶

Adding an Instrument¶

Prioritization¶

Example¶

Adding a Result Processor¶

Example¶

Adding a Resource Getter¶

Getter Prioritization¶

Example¶

Adding a Device¶

Other Extension Types¶

Packaging Your Extensions¶

Table Of Contents

Previous topic

Next topic

This Page

name:	This identifies the workload (e.g. it used to specify it in the agenda.
init_resources:	This method may be optionally override to implement dynamic resource discovery for the workload. This method executes early on, before the device has been initialized, so it should only be used to initialize resources that do not depend on the device to resolve. This method is executed once per run for each workload instance.
validate:	This method can be used to validate any assumptions your workload makes about the environment (e.g. that required files are present, environment variables are set, etc) and should raise a `wlauto.exceptions.WorkloadError` if that is not the case. The base class implementation only makes sure sure that the name attribute has been set.
initialize:	This method will be executed exactly once per run (no matter how many instances of the workload there are). It will run after the device has been initialized, so it may be used to perform device-dependent initialization that does not need to be repeated on each iteration (e.g. as installing executables required by the workload on the device).
setup:	Everything that needs to be in place for workload execution should be done in this method. This includes copying files to the device, starting up an application, configuring communications channels, etc.
run:	This method should perform the actual task that is being measured. When this method exits, the task is assumed to be complete. Note Instrumentation is kicked off just before calling this method and is disabled right after, so everything in this method is being measured. Therefore this method should contain the least code possible to perform the operations you are interested in measuring. Specifically, things like installing or starting applications, processing results, or copying files to/from the device should be done elsewhere if possible.
update_result:	This method gets invoked after the task execution has finished and should be used to extract metrics and add them to the result (see below).
teardown:	This could be used to perform any cleanup you may wish to do, e.g. Uninstalling applications, deleting file on the device, etc.
finalize:	This is the complement to `initialize`. This will be executed exactly once at the end of the run. This should be used to perform any final clean up (e.g. uninstalling binaries installed in the `initialize`).

name:	This identifies the workload (e.g. it used to specify it in the agenda.
package:	This is the name of the ‘.apk’ package without its file extension.
activity:	The name of the main activity that runs the package.

setup:	This stage is where the game is loaded. It is a good place to record revent here to modify the game settings and get it ready to start.
run:	This stage is where the game actually starts. This will allow for more accurate results if the revent file for this stage only records the game being played.

cached:	The cached version of the resource. Look here first. This priority also implies that the resource at this location is a “cache” and is not the only version of the resource, so it may be cleared without losing access to the resource.
preferred:	Take this resource in favour of the environment resource.
environment:	Found somewhere under ~/.workload_automation/ or equivalent, or from environment variables, external configuration files, etc. These will override resource supplied with the package.
package:	Resource provided with the package.
remote:	Resource will be downloaded from a remote location (such as an HTTP server or a samba share). Try this only if no other getter was successful.

Writing Extensions¶

Extension Basics¶

The Context¶

Paths¶

On the host¶

On the device¶

Deploying executables to a device¶

Parameters¶

Validation and cross-parameter constraints¶

Logging¶

Documenting¶

Error Notification¶

Utils¶

Adding a Workload¶

Example¶

Adding revent-dependent Workload:¶

Example:¶

Convention for Naming revent Files for wlauto.common.game.GameWorkload¶

Adding an Instrument¶

Prioritization¶

Example¶

Adding a Result Processor¶

Example¶

Adding a Resource Getter¶

Getter Prioritization¶

Example¶

Adding a Device¶

Other Extension Types¶

Packaging Your Extensions¶

Convention for Naming revent Files for `wlauto.common.game.GameWorkload`¶