wlauto.core package

Submodules

wlauto.core.agenda module

class wlauto.core.agenda.Agenda(source=None)

Bases: object

add_workload_entry(w)

get_workload_entry(w)

class wlauto.core.agenda.AgendaEntry

Bases: object

to_dict()

class wlauto.core.agenda.AgendaGlobalEntry(**kwargs)

Bases: wlauto.core.agenda.AgendaEntry

Workload configuration global to all workloads.

class wlauto.core.agenda.AgendaSectionEntry(agenda, **kwargs)

Bases: wlauto.core.agenda.AgendaEntry

Specifies execution of a workload, including things like the number of iterations, device runtime_parameters configuration, etc.

to_dict()

class wlauto.core.agenda.AgendaWorkloadEntry(**kwargs)

Bases: wlauto.core.agenda.AgendaEntry

Specifies execution of a workload, including things like the number of iterations, device runtime_parameters configuration, etc.

wlauto.core.agenda.dict_constructor(loader, node)

wlauto.core.agenda.dict_representer(dumper, data)

wlauto.core.agenda.get_aliased_param(d, aliases, default=None, pop=True)

wlauto.core.bootstrap module

class wlauto.core.bootstrap.ConfigLoader

Bases: object

This class is responsible for loading and validating config files.

get_config_paths()

log_file

meta_directory

update(source)

update_from_dict(source)

update_from_file(source)

wlauto.core.bootstrap.init_environment(env_root, dep_dir, extension_paths, overwrite_existing=False)

Initialise a fresh user environment creating the workload automation

wlauto.core.command module

class wlauto.core.command.Command(subparsers)

Bases: wlauto.core.extension.Extension

Defines a Workload Automation command. This will be executed from the command line as wa <command> [args ...]. This defines the name to be used when invoking wa, the code that will actually be executed on invocation and the argument parser to be used to parse the reset of the command line arguments.

aliases = AC([])

artifacts = AC([])

core_modules = []

description = None

epilog = None

execute(args)

Execute this command.

Args:An argparse.Namespace containing command line arguments (as returned by argparse.ArgumentParser.parse_args(). This would usually be the result of invoking self.parser.

finalize(*args, **kwargs)

formatter_class = None

help = None

initialize(*args, **kwargs)

parameters = AC(["Param({'kind': <type 'list'>, 'mandatory': None, 'name': 'modules', 'constraint': None, 'default': None, 'allowed_values': None, 'global_alias': None, 'override': False})"])

usage = None

validate(*args, **kwargs)

wlauto.core.configuration module

class wlauto.core.configuration.ConfigurationJSONEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, encoding='utf-8', default=None)

Bases: json.encoder.JSONEncoder

default(obj)

class wlauto.core.configuration.RebootPolicy(policy)

Bases: object

Represents the reboot policy for the execution – at what points the device should be rebooted. This, in turn, is controlled by the policy value that is passed in on construction and would typically be read from the user’s settings. Valid policy values are:

Never:The device will never be rebooted. As_needed:Only reboot the device if it becomes unresponsive, or needs to be flashed, etc. Initial:The device will be rebooted when the execution first starts, just before executing the first workload spec. Each_spec:The device will be rebooted before running a new workload spec. Each_iteration:The device will be rebooted before each new iteration.

can_reboot

perform_initial_boot

reboot_on_each_iteration

reboot_on_each_spec

valid_policies = ['never', 'as_needed', 'initial', 'each_spec', 'each_iteration']

class wlauto.core.configuration.RunConfiguration(ext_loader)

Bases: object

Loads and maintains the unified configuration for this run. This includes configuration for WA execution as a whole, and parameters for specific specs.

WA configuration mechanism aims to be flexible and easy to use, while at the same time providing storing validation and early failure on error. To meet these requirements, the implementation gets rather complicated. This is going to be a quick overview of the underlying mechanics.

Note

You don’t need to know this to use WA, or to write extensions for it. From the point of view of extension writers, configuration from various sources “magically” appears as attributes of their classes. This explanation peels back the curtain and is intended for those who, for one reason or another, need to understand how the magic works.

terminology

run

A single execution of a WA agenda.

run config(uration) (object)

An instance of this class. There is one per run.

config(uration) item

A single configuration entry or “setting”, e.g. the device interface to use. These can be for the run as a whole, or for a specific extension.

(workload) spec

A specification of a single workload execution. This combines workload configuration with things like the number of iterations to run, which instruments to enable, etc. More concretely, this is an instance of WorkloadRunSpec.

overview

There are three types of WA configuration:

1. “Meta” configuration that determines how the rest of the configuration is processed (e.g. where extensions get loaded from). Since this does not pertain to run configuration, it will not be covered further.
2. Global run configuration, e.g. which workloads, result processors and instruments will be enabled for a run.
3. Per-workload specification configuration, that determines how a particular workload instance will get executed (e.g. what workload parameters will be used, how many iterations.

run configuration

Run configuration may appear in a config file (usually ~/.workload_automation/config.py), or in the config section of an agenda. Configuration is specified as a nested structure of dictionaries (associative arrays, or maps) and lists in the syntax following the format implied by the file extension (currently, YAML and Python are supported). If the same configuration item appears in more than one source, they are merged with conflicting entries taking the value from the last source that specified them.

In addition to a fixed set of global configuration items, configuration for any WA Extension (instrument, result processor, etc) may also be specified, namespaced under the extension’s name (i.e. the extensions name is a key in the global config with value being a dict of parameters and their values). Some Extension parameters also specify a “global alias” that may appear at the top-level of the config rather than under the Extension’s name. It is not an error to specify configuration for an Extension that has not been enabled for a particular run; such configuration will be ignored.

per-workload configuration

Per-workload configuration can be specified in three places in the agenda: the workload entry in the workloads list, the global entry (configuration there will be applied to every workload entry), and in a section entry in sections list ( configuration in every section will be applied to every workload entry separately, creating a “cross-product” of section and workload configurations; additionally, sections may specify their own workload lists).

If they same configuration item appears in more than one of the above places, they will be merged in the following order: global, section, workload, with conflicting scalar values in the later overriding those from previous locations.

Global parameter aliases

As mentioned above, an Extension’s parameter may define a global alias, which will be specified and picked up from the top-level config, rather than config for that specific extension. It is an error to specify the value for a parameter both through a global alias and through extension config dict in the same configuration file. It is, however, possible to use a global alias in one file, and specify extension configuration for the same parameter in another file, in which case, the usual merging rules would apply.

Loading and validation of configuration

Validation of user-specified configuration happens at several stages of run initialisation, to ensure that appropriate context for that particular type of validation is available and that meaningful errors can be reported, as early as is feasible.

• Syntactic validation is performed when configuration is first loaded. This is done by the loading mechanism (e.g. YAML parser), rather than WA itself. WA propagates any errors encountered as ConfigErrors.
• Once a config file is loaded into a Python structure, it scanned to extract settings. Static configuration is validated and added to the config. Extension configuration is collected into a collection of “raw” config, and merged as appropriate, but is not processed further at this stage.
• Once all configuration sources have been processed, the configuration as a whole is validated (to make sure there are no missing settings, etc).
• Extensions are loaded through the run config object, which instantiates them with appropriate parameters based on the “raw” config collected earlier. When an Extension is instantiated in such a way, its config is “officially” added to run configuration tracked by the run config object. Raw config is discarded at the end of the run, so that any config that wasn’t loaded in this way is not recoded (as it was not actually used).
• Extension parameters a validated individually (for type, value ranges, etc) as they are loaded in the Extension’s __init__.
• An extension’s validate() method is invoked before it is used (exactly when this happens depends on the extension’s type) to perform any final validation that does not rely on the target being present (i.e. this would happen before WA connects to the target). This can be used perform inter-parameter validation for an extension (e.g. when valid range for one parameter depends on another), and more general WA state assumptions (e.g. a result processor can check that an instrument it depends on has been installed).
• Finally, it is the responsibility of individual extensions to validate any assumptions they make about the target device (usually as part of their setup()).

Handling of Extension aliases.

WA extensions can have zero or more aliases (not to be confused with global aliases for extension parameters). An extension allows associating an alternative name for the extension with a set of parameter values. In other words aliases associate common configurations for an extension with a name, providing a shorthand for it. For example, “t-rex_offscreen” is an alias for “glbenchmark” workload that specifies that “use_case” should be “t-rex” and “variant” should be “offscreen”.

special loading rules

Note that as a consequence of being able to specify configuration for any Extension namespaced under the Extension’s name in the top-level config, two distinct mechanisms exist form configuring devices and workloads. This is valid, however due to their nature, they are handled in a special way. This may be counter intuitive, so configuration of devices and workloads creating entries for their names in the config is discouraged in favour of using the “normal” mechanisms of configuring them (device_config for devices and workload specs in the agenda for workloads).

In both cases (devices and workloads), “normal” config will always override named extension config irrespective of which file it was specified in. So a adb_name name specified in device_config inside ~/.workload_automation/config.py will override adb_name specified for juno in the agenda (even when device is set to “juno”).

Again, this ignores normal loading rules, so the use of named extension configuration for devices and workloads is discouraged. There maybe some situations where this behaviour is useful however (e.g. maintaining configuration for different devices in one config file).

all_instrumentation

default_execution_order = 'by_iteration'

default_reboot_policy = 'as_needed'

finalize()

This must be invoked once all configuration sources have been loaded. This will do the final processing, setting instrumentation and result processor configuration for the run And making sure that all the mandatory config has been specified.

general_config = [<wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>]

get_extension(ext_name, *args)

get_reboot_policy()

ignore_names = ['logging', 'remote_assets_mount_point']

load_config(source)

Load configuration from the specified source. The source must be either a path to a valid config file or a dict-like object. Currently, config files can be either python modules (.py extension) or YAML documents (.yaml extension).

reboot_policy

serialize(wfh)

set_agenda(agenda, selectors=None)

Set the agenda for this run; Unlike with config files, there can only be one agenda.

set_reboot_policy(value)

to_dict()

workload_config = [<wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>, <wlauto.core.configuration.RunConfigurationItem object>]

class wlauto.core.configuration.RunConfigurationItem(name, category, method)

Bases: object

This represents a predetermined “configuration point” (an individual setting) and describes how it must be handled when encountered.

combine(*args)

Combine the provided values according to the method for this configuration item. Order matters – values are assumed to be in the order they were specified by the user. The resulting value is also checked to patch the specified type.

valid_categories = {'dict': {}, 'scalar': None, 'list': []}

valid_methods = ['keep', 'replace', 'merge']

class wlauto.core.configuration.SharedConfiguration

Bases: object

class wlauto.core.configuration.WorkloadRunSpec(id=None, number_of_iterations=None, workload_name=None, boot_parameters=None, label=None, section_id=None, workload_parameters=None, runtime_parameters=None, instrumentation=None, flash=None, classifiers=None)

Bases: object

Specifies execution of a workload, including things like the number of iterations, device runtime_parameters configuration, etc.

copy()

framework_mandatory_parameters = ['id', 'number_of_iterations']

load(device, ext_loader)

Loads the workload for the specified device using the specified loader. This must be done before attempting to execute the spec.

mandatory_parameters = ['workload_name']

match_selectors(selectors)

Returns True if this spec matches the specified selectors, and False otherwise. selectors must be a dict-like object with attribute names mapping onto selector values. At the moment, only equality selection is supported; i.e. the value of the attribute of the spec must match exactly the corresponding value specified in the selectors dict.

secition

set(param, value)

to_dict()

validate()

workload

class wlauto.core.configuration.status_list

Bases: list

append(item)

wlauto.core.device module

Base classes for device interfaces.

Device:The base class for all devices. This defines the interface that must be implemented by all devices and therefore any workload and instrumentation can always rely on. AndroidDevice:Implements most of the Device interface, and extends it with a number of Android-specific methods. BigLittleDevice:  Subclasses AndroidDevice to implement big.LITTLE-specific runtime parameters. SimpleMulticoreDevice:  Subclasses AndroidDevice to implement homogeneous cores device runtime parameters.

class wlauto.core.device.RuntimeParameter(name, getter, setter, getter_args=None, setter_args=None, value_name='value', override=False)

Bases: object

A runtime parameter which has its getter and setter methods associated it with it.

class wlauto.core.device.CoreParameter(name, getter, setter, getter_args=None, setter_args=None, value_name='value', override=False)

Bases: wlauto.core.device.RuntimeParameter

A runtime parameter that will get expanded into a RuntimeParameter for each core type.

get_runtime_parameters(core_names)

class wlauto.core.device.Device(**kwargs)

Bases: wlauto.core.extension.Extension

Base class for all devices supported by Workload Automation. Defines the interface the rest of WA uses to interact with devices.

name:

Unique name used to identify the device.

platform:

The name of the device’s platform (e.g. Android) this may be used by workloads and instrumentation to assess whether they can run on the device.

working_directory:  

a string of the directory which is going to be used by the workloads on the device.

binaries_directory:  

a string of the binary directory for the device.

has_gpu:

Should be True if the device as a separate GPU, and False if graphics processing is done on a CPU.

Note

Pretty much all devices currently on the market have GPUs, however this may not be the case for some development boards.

path_module:

The name of one of the modules implementing the os.path interface, e.g. posixpath or ntpath. You can provide your own implementation rather than relying on one of the standard library modules, in which case you need to specify the full path to you module. e.g. ‘/home/joebloggs/mypathimp.py’

parameters:

A list of RuntimeParameter objects. The order of the objects is very important as the setters and getters will be called in the order the RuntimeParameter objects inserted.

active_cores:

This should be a list of all the currently active cpus in the device in '/sys/devices/system/cpu/online'. The returned list should be read from the device at the time of read request.

active_cores = None

aliases = AC([])

artifacts = AC([])

boot(*args, **kwargs)

Perform the seteps necessary to boot the device to the point where it is ready to accept other commands.

Changed in version 2.1.3: no longer expected to wait until boot completes.

capture_screen(filepath)

Captures the current device screen into the specified file in a PNG format.

connect(*args, **kwargs)

Establish a connection to the device that will be used for subsequent commands.

Added in version 2.1.3.

core_modules = []

default_working_directory = None

delete_file(filepath)

Delete the specified file on the device.

disconnect()

Close the established connection to the device.

dynamic_modules = AC([])

execute(command, timeout=None, **kwargs)

Execute the specified command command on the device and return the output.

Parameters:

• command – Command to be executed on the device.
• timeout – If the command does not return after the specified time, execute() will abort with an error. If there is no timeout for the command, this should be set to 0 or None.

Other device-specific keyword arguments may also be specified.

Returns:The stdout output from the command.

file_exists(filepath)

Check if the specified file or directory exist on the device.

finalize(*args, **kwargs)

get_pids_of(process_name)

Returns a list of PIDs of the specified process name.

get_properties(output_path)

Captures and saves the device configuration properties version and any other relevant information. Return them in a dict

get_runtime_parameter_names()

get_runtime_parameters()

returns the runtime parameters that have been set.

get_sysfile_value(sysfile, kind=None, binary=False)

Get the contents of the specified sysfile.

Parameters:

• sysfile – The file who’s contents will be returned.
• kind – The type of value to be expected in the sysfile. This can be any Python callable that takes a single str argument. If not specified or is None, the contents will be returned as a string.
• binary – Whether the value should be encoded into base64 for reading to deal with binary format.

has_gpu = None

initialize(*args, **kwargs)

install(filepath, **kwargs)

Install the specified file on the device. What “install” means is device-specific and may possibly also depend on the type of file.

is_network_connected()

Checks if the device is connected to the internet

kill(pid, as_root=False)

Kill the process with the specified PID.

killall(process_name, as_root=False)

Kill all running processes with the specified name.

listdir(path, **kwargs)

List the contents of the specified directory.

name = None

parameters = AC(["Param({'kind': <type 'list'>, 'mandatory': None, 'name': 'modules', 'constraint': None, 'default': None, 'allowed_values': None, 'global_alias': None, 'override': False})", "Param({'kind': <class 'wlauto.utils.types.list_of_caseless_strings'>, 'mandatory': True, 'name': 'core_names', 'constraint': None, 'default': None, 'allowed_values': None, 'global_alias': None, 'override': False})", "Param({'kind': <function list_of_ints>, 'mandatory': True, 'name': 'core_clusters', 'constraint': None, 'default': None, 'allowed_values': None, 'global_alias': None, 'override': False})"])

path_module = None

ping()

This must return successfully if the device is able to receive commands, or must raise wlauto.exceptions.DeviceUnresponsiveError if the device cannot respond.

platform = None

pull_file(source, dest)

Pull a file from device system onto the host file system.

push_file(source, dest)

Push a file from the host file system onto the device.

reset()

Initiate rebooting of the device.

Added in version 2.1.3.

runtime_parameters = AC([])

set_runtime_parameters(params)

The parameters are taken from the keyword arguments and are specific to a particular device. See the device documentation.

set_sysfile_value(filepath, value, verify=True, binary=False)

Write the specified value to the specified file on the device and verify that the value has actually been written.

Parameters:

• file – The file to be modified.
• value – The value to be written to the file. Must be an int or a string convertable to an int.
• verify – Specifies whether the value should be verified, once written.
• binary – Specifies whether the value should be written as binary data.

Should raise DeviceError if could write value.

sleep(seconds)

Sleep for the specified time on the target device.

Parameters:seconds – Time in seconds to sleep on the device

The sleep is executed on the device using self.execute(). We set the timeout for this command to be 10 seconds longer than the sleep itself to make sure the command has time to complete before we timeout.

start()

This gets invoked before an iteration is started and is endented to help the device manange any internal supporting functions.

stop()

This gets invoked after iteration execution has completed and is endented to help the device manange any internal supporting functions.

uninstall(filepath)

Uninstall the specified file on the device. What “uninstall” means is device-specific and may possibly also depend on the type of file.

validate(*args, **kwargs)

class wlauto.core.device.DeviceMeta

Bases: wlauto.core.extension.ExtensionMeta

to_propagate = [('parameters', <class 'wlauto.core.extension.Param'>, <class 'wlauto.core.extension.AttributeCollection'>), ('artifacts', <class 'wlauto.core.extension.Artifact'>, <class 'wlauto.core.extension.AttributeCollection'>), ('core_modules', <type 'str'>, <class 'wlauto.core.extension.ListCollection'>), ('runtime_parameters', <class 'wlauto.core.device.RuntimeParameter'>, <class 'wlauto.core.extension.AttributeCollection'>), ('dynamic_modules', <class 'wlauto.core.device.DynamicModuleSpec'>, <class 'wlauto.core.extension.AttributeCollection'>)]

wlauto.core.entry_point module

wlauto.core.entry_point.convert_TERM_into_INT_handler(signal, frame)

wlauto.core.entry_point.load_commands(subparsers)

wlauto.core.entry_point.main()

wlauto.core.execution module

This module contains the execution logic for Workload Automation. It defines the following actors:

WorkloadSpec: Identifies the workload to be run and defines parameters under

which it should be executed.

Executor: Responsible for the overall execution process. It instantiates

and/or intialises the other actors, does any necessary vaidation and kicks off the whole process.

Execution Context: Provides information about the current state of run

execution to instrumentation.

RunInfo: Information about the current run.

Runner: This executes workload specs that are passed to it. It goes through

stages of execution, emitting an appropriate signal at each step to allow instrumentation to do its stuff.

class wlauto.core.execution.ByIterationRunner(device, context, result_manager)

Bases: wlauto.core.execution.Runner

Runs the first iteration for all benchmarks first, before proceeding to the next iteration, i.e. A1, B1, C1, A2, B2, C2… instead of A1, A1, B1, B2, C1, C2…

If multiple sections where specified in the agenda, this will run all sections for the first global spec first, followed by all sections for the second spec, etc.

e.g. given sections X and Y, and global specs A and B, with 2 iterations, this will run

X.A1, Y.A1, X.B1, Y.B1, X.A2, Y.A2, X.B2, Y.B2

init_queue(specs)

class wlauto.core.execution.BySectionRunner(device, context, result_manager)

Bases: wlauto.core.execution.Runner

Runs the first iteration for all benchmarks first, before proceeding to the next iteration, i.e. A1, B1, C1, A2, B2, C2… instead of A1, A1, B1, B2, C1, C2…

If multiple sections where specified in the agenda, this will run all specs for the first section followed by all specs for the seciod section, etc.

e.g. given sections X and Y, and global specs A and B, with 2 iterations, this will run

X.A1, X.B1, Y.A1, Y.B1, X.A2, X.B2, Y.A2, Y.B2

init_queue(specs)

class wlauto.core.execution.BySpecRunner(device, context, result_manager)

Bases: wlauto.core.execution.Runner

This is that “classic” implementation that executes all iterations of a workload spec before proceeding onto the next spec.

init_queue(specs)

class wlauto.core.execution.ExecutionContext(device, config)

Bases: object

Provides a context for instrumentation. Keeps track of things like current workload and iteration.

This class also provides two status members that can be used by workloads and instrumentation to keep track of arbitrary state. result is reset on each new iteration of a workload; run_status is maintained throughout a Workload Automation run.

add_artifact(name, path, kind, *args, **kwargs)

add_classifiers(**kwargs)

add_iteration_artifact(name, path, kind, *args, **kwargs)

add_metric(*args, **kwargs)

add_run_artifact(name, path, kind, *args, **kwargs)

current_iteration

default_run_artifacts = [<wlauto.core.extension.Artifact object>]

end_job()

get_artifact(name)

initialize()

job_status

next_job(job)

Invoked by the runner when starting a new iteration of workload execution.

result

spec

workload

class wlauto.core.execution.Executor

Bases: object

The Executor’s job is to set up the execution context and pass to a Runner along with a loaded run specification. Once the Runner has done its thing, the Executor performs some final reporint before returning.

The initial context set up involves combining configuration from various sources, loading of requided workloads, loading and installation of instruments and result processors, etc. Static validation of the combined configuration is also performed.

execute(agenda, selectors=None)

Execute the run specified by an agenda. Optionally, selectors may be used to only selecute a subset of the specified agenda.

Params:

:agenda: an ``Agenda`` instance to be executed.
:selectors: A dict mapping selector name to the coresponding values.

Selectors

Currently, the following seectors are supported:

ids

The value must be a sequence of workload specfication IDs to be executed. Note that if sections are specified inthe agenda, the workload specifacation ID will be a combination of the section and workload IDs.

execute_postamble()

This happens after the run has completed. The overall results of the run are summarised to the user.

class wlauto.core.execution.RandomRunner(device, context, result_manager)

Bases: wlauto.core.execution.Runner

This will run specs in a random order.

init_queue(specs)

class wlauto.core.execution.RunInfo(config)

Bases: object

Information about the current run, such as its unique ID, run time, etc.

to_dict()

class wlauto.core.execution.Runner(device, context, result_manager)

Bases: object

This class is responsible for actually performing a workload automation run. The main responsibility of this class is to emit appropriate signals at the various stages of the run to allow things like traces an other instrumentation to hook into the process.

This is an abstract base class that defines each step of the run, but not the order in which those steps are executed, which is left to the concrete derived classes.

config

current_job

init_queue(specs)

next_job

previous_job

run()

spec_changed

spec_will_change

class wlauto.core.execution.RunnerJob(spec, retry=0)

Bases: object

Represents a single execution of a RunnerJobDescription. There will be one created for each iteration specified by RunnerJobDescription.number_of_iterations.

wlauto.core.extension module

class wlauto.core.extension.Alias(name, **kwargs)

Bases: object

This represents a configuration alias for an extension, mapping an alternative name to a set of parameter values, effectively providing an alternative set of default values.

validate(ext)

class wlauto.core.extension.AliasCollection

Bases: wlauto.core.extension.AttributeCollection

class wlauto.core.extension.Artifact(name, path, kind, level='run', mandatory=False, description=None)

Bases: object

This is an artifact generated during execution/post-processing of a workload. Unlike metrics, this represents an actual artifact, such as a file, generated. This may be “result”, such as trace, or it could be “meta data” such as logs. These are distinguished using the kind attribute, which also helps WA decide how it should be handled. Currently supported kinds are:

log:

A log file. Not part of “results” as such but contains information about the run/workload execution that be useful for diagnostics/meta analysis.

meta:

A file containing metadata. This is not part of “results”, but contains information that may be necessary to reproduce the results (contrast with log artifacts which are not necessary).

data:

This file contains new data, not available otherwise and should be considered part of the “results” generated by WA. Most traces would fall into this category.

export:

Exported version of results or some other artifact. This signifies that this artifact does not contain any new data that is not available elsewhere and that it may be safely discarded without losing information.

raw:

Signifies that this is a raw dump/log that is normally processed to extract useful information and is then discarded. In a sense, it is the opposite of export, but in general may also be discarded.

Note

whether a file is marked as log/data or raw depends on how important it is to preserve this file, e.g. when archiving, vs how much space it takes up. Unlike export artifacts which are (almost) always ignored by other exporters as that would never result in data loss, raw files may be processed by exporters if they decided that the risk of losing potentially (though unlikely) useful data is greater than the time/space cost of handling the artifact (e.g. a database uploader may choose to ignore raw artifacts, where as a network filer archiver may choose to archive them).

ITERATION = 'iteration'

RUN = 'run'

exists(context)

Returns True if artifact exists within the specified context, and False otherwise.

to_dict()

valid_kinds = ['log', 'meta', 'data', 'export', 'raw']

class wlauto.core.extension.AttributeCollection(attrcls, owner)

Bases: object

Accumulator for extension attribute objects (such as Parameters or Artifacts). This will replace any class member list accumulating such attributes through the magic of metaprogramming[*].

[*]	which is totally safe and not going backfire in any way…

add(p)

append(p)

values

class wlauto.core.extension.Extension(**kwargs)

Bases: object

Base class for all WA extensions. An extension is basically a plug-in. It extends the functionality of WA in some way. Extensions are discovered and loaded dynamically by the extension loader upon invocation of WA scripts. Adding an extension is a matter of placing a class that implements an appropriate interface somewhere it would be discovered by the loader. That “somewhere” is typically one of the extension subdirectories under ~/.workload_automation/.

aliases = AC([])

artifacts = AC([])

can(capability)

Check if this extension has the specified capability. The alternative method can is identical to this. Which to use is up to the caller depending on what makes semantic sense in the context of the capability, e.g. can('hard_reset') vs has('active_cooling').

check_artifacts(context, level)

Make sure that all mandatory artifacts have been generated.

core_modules = []

dependencies_directory

finalize(*args, **kwargs)

get_config()

Returns current configuration (i.e. parameter values) of this extension.

classmethod get_default_config()

has(capability)

Check if this extension has the specified capability. The alternative method can is identical to this. Which to use is up to the caller depending on what makes semantic sense in the context of the capability, e.g. can('hard_reset') vs has('active_cooling').

initialize(*args, **kwargs)

kind = None

load_modules(loader)

Load the modules specified by the “modules” Parameter using the provided loader. A loader can be any object that has an atribute called “get_module” that implements the following signature:

get_module(name, owner, **kwargs)

and returns an instance of wlauto.core.extension.Module. If the module with the specified name is not found, the loader must raise an appropriate exception.

name = None

parameters = AC(["Param({'kind': <type 'list'>, 'mandatory': None, 'name': 'modules', 'constraint': None, 'default': None, 'allowed_values': None, 'global_alias': None, 'override': False})"])

validate(*args, **kwargs)

class wlauto.core.extension.ExtensionMeta

Bases: type

This basically adds some magic to extensions to make implementing new extensions, such as workloads less complicated.

It ensures that certain class attributes (specified by the to_propagate attribute of the metaclass) get propagated down the inheritance hierarchy. The assumption is that the values of the attributes specified in the class are iterable; if that is not met, Bad Things (tm) will happen.

This also provides virtual method implementation, similar to those in C-derived OO languages, and alias specifications.

global_virtuals = ['initialize', 'finalize']

to_propagate = [('parameters', <class 'wlauto.core.extension.Param'>, <class 'wlauto.core.extension.AttributeCollection'>), ('artifacts', <class 'wlauto.core.extension.Artifact'>, <class 'wlauto.core.extension.AttributeCollection'>), ('core_modules', <type 'str'>, <class 'wlauto.core.extension.ListCollection'>)]

virtual_methods = ['validate', 'initialize', 'finalize']

class wlauto.core.extension.ListCollection(attrcls, owner)

Bases: list

class wlauto.core.extension.Module(owner, **kwargs)

Bases: wlauto.core.extension.Extension

This is a “plugin” for an extension this is intended to capture functionality that may be optional for an extension, and so may or may not be present in a particular setup; or, conversely, functionality that may be reusable between multiple devices, even if they are not with the same inheritance hierarchy.

In other words, a Module is roughly equivalent to a kernel module and its primary purpose is to implement WA “drivers” for various peripherals that may or may not be present in a particular setup.

Note

A mudule is itself an Extension and can therefore have its own modules.

aliases = AC([])

artifacts = AC([])

capabilities = []

core_modules = []

finalize(*args, **kwargs)

initialize(*args, **kwargs)

parameters = AC(["Param({'kind': <type 'list'>, 'mandatory': None, 'name': 'modules', 'constraint': None, 'default': None, 'allowed_values': None, 'global_alias': None, 'override': False})"])

root_owner

validate(*args, **kwargs)

class wlauto.core.extension.Param(name, kind=None, mandatory=None, default=None, override=False, allowed_values=None, description=None, constraint=None, global_alias=None, convert_types=True)

Bases: object

This is a generic parameter for an extension. Extensions instantiate this to declare which parameters are supported.

get_type_name()

kind_map = {<type 'bool'>: <function boolean>, <type 'int'>: <function integer>}

set_value(obj, value=None)

wlauto.core.extension.Parameter

alias of Param

wlauto.core.extension_loader module

class wlauto.core.extension_loader.ExtensionLoader(packages=None, paths=None, ignore_paths=None, keep_going=False, load_defaults=True)

Bases: object

Discovers, enumerates and loads available devices, configs, etc. The loader will attempt to discover things on construction by looking in predetermined set of locations defined by default_paths. Optionally, additional locations may specified through paths parameter that must be a list of additional Python module paths (i.e. dot-delimited).

clear()

Clear all discovered items.

get_default_config(ext_name)

Returns the default configuration for the specified extension name. The name may be an alias, in which case, the returned config will be augmented with appropriate alias overrides.

get_extension(name, *args, **kwargs)

Return extension of the specified kind with the specified name. Any additional parameters will be passed to the extension’s __init__.

get_extension_class(name, kind=None)

Return the class for the specified extension if found or raises ValueError.

get_load_defaults()

has_extension(name, kind=None)

Returns True if an extensions with the specified name has been discovered by the loader. If kind was specified, only returns True if the extension has been found, and it is of the specified kind.

list_extensions(kind=None)

List discovered extension classes. Optionally, only list extensions of a particular type.

load_defaults

reload()

Clear all discovered items and re-run the discovery.

resolve_alias(alias_name)

Try to resolve the specified name as an extension alias. Returns a two-tuple, the first value of which is actual extension name, and the second is a dict of parameter values for this alias. If the name passed is already an extension name, then the result is (alias_name, {}).

set_load_defaults(value)

update(packages=None, paths=None, ignore_paths=None)

Load extensions from the specified paths/packages without clearing or reloading existing extension.

class wlauto.core.extension_loader.ExtensionLoaderItem(ext_tuple)

Bases: object

class wlauto.core.extension_loader.GlobalParameterAlias(name)

Bases: object

Represents a “global alias” for an extension parameter. A global alias is specified at the top-level of config rather namespaced under an extension name.

Multiple extensions may have parameters with the same global_alias if they are part of the same inheritance hierarchy and one parameter is an override of the other. This class keeps track of all such cases in its extensions dict.

get_param(ext)

iteritems()

update(other_ext)

wlauto.core.exttype module

wlauto.core.exttype.get_extension_type(ext)

Given an instance of wlauto.core.Extension, return a string representing the type of the extension (e.g. 'workload' for a Workload subclass instance).

wlauto.core.instrumentation module

Adding New Instrument

Any new instrument should be a subclass of Instrument and it must have a name. When a new instrument is added to Workload Automation, the methods of the new instrument will be found automatically and hooked up to the supported signals. Once a signal is broadcasted, the corresponding registered method is invoked.

Each method in Instrument must take two arguments, which are self and context. Supported signals can be found in [… link to signals …] To make implementations easier and common, the basic steps to add new instrument is similar to the steps to add new workload.

Hence, the following methods are sufficient to implement to add new instrument:

• setup: This method is invoked after the workload is setup. All the

  necessary setups should go inside this method. Setup, includes operations like, pushing the files to the target device, install them, clear logs, etc.

• start: It is invoked just before the workload start execution. Here is

  where instrument measures start being registered/taken.

• stop: It is invoked just after the workload execution stops. The measures

  should stop being taken/registered.

• update_result: It is invoked after the workload updated its result.

  update_result is where the taken measures are added to the result so it can be processed by Workload Automation.

• teardown is invoked after the workload is teared down. It is a good place

  to clean any logs generated by the instrument.

For example, to add an instrument which will trace device errors, we subclass Instrument and overwrite the variable name.:

#BINARY_FILE = os.path.join(os.path.dirname(__file__), 'trace')
class TraceErrorsInstrument(Instrument):

    name = 'trace-errors'

    def __init__(self, device):
        super(TraceErrorsInstrument, self).__init__(device)
        self.trace_on_device = os.path.join(self.device.working_directory, 'trace')

We then declare and implement the aforementioned methods. For the setup method, we want to push the file to the target device and then change the file mode to 755

def setup(self, context):
    self.device.push_file(BINARY_FILE, self.device.working_directory)
    self.device.execute('chmod 755 {}'.format(self.trace_on_device))

Then we implemented the start method, which will simply run the file to start tracing.

def start(self, context):
    self.device.execute('{} start'.format(self.trace_on_device))

Lastly, we need to stop tracing once the workload stops and this happens in the stop method:

def stop(self, context):
    self.device.execute('{} stop'.format(self.trace_on_device))

The generated result can be updated inside update_result, or if it is trace, we just pull the file to the host device. context has a result variable which has add_metric method. It can be used to add the instrumentation results metrics to the final result for the workload. The method can be passed 4 params, which are metric key, value, unit and lower_is_better, which is a boolean.

def update_result(self, context):
    # pull the trace file to the device
    result = os.path.join(self.device.working_directory, 'trace.txt')
    self.device.pull_file(result, context.working_directory)

    # parse the file if needs to be parsed, or add result to
    # context.result

At the end, we might want to delete any files generated by the instrumentation and the code to clear these file goes in teardown method.

def teardown(self, context):
    self.device.delete_file(os.path.join(self.device.working_directory, 'trace.txt'))

class wlauto.core.instrumentation.Instrument(device, **kwargs)

Bases: wlauto.core.extension.Extension

Base class for instrumentation implementations.

aliases = AC([])

artifacts = AC([])

core_modules = []

finalize(*args, **kwargs)

initialize(*args, **kwargs)

parameters = AC(["Param({'kind': <type 'list'>, 'mandatory': None, 'name': 'modules', 'constraint': None, 'default': None, 'allowed_values': None, 'global_alias': None, 'override': False})"])

validate(*args, **kwargs)

class wlauto.core.instrumentation.ManagedCallback(instrument, callback)

Bases: object

This wraps instruments’ callbacks to ensure that errors do interfer with run execution.

wlauto.core.instrumentation.check_failures()

wlauto.core.instrumentation.disable(to_disable)

wlauto.core.instrumentation.disable_all()

wlauto.core.instrumentation.enable(to_enable)

wlauto.core.instrumentation.enable_all()

wlauto.core.instrumentation.get_disabled()

wlauto.core.instrumentation.get_enabled()

wlauto.core.instrumentation.get_instrument(inst)

wlauto.core.instrumentation.install(instrument)

This will look for methods (or any callable members) with specific names in the instrument and hook them up to the corresponding signals.

Parameters:instrument – Instrument instance to install.

wlauto.core.instrumentation.is_enabled(instrument)

wlauto.core.instrumentation.is_installed(instrument)

wlauto.core.instrumentation.reset_failures()

wlauto.core.instrumentation.uninstall(instrument)

wlauto.core.instrumentation.validate()

wlauto.core.resolver module

Defines infrastructure for resource resolution. This is used to find various dependencies/assets/etc that WA objects rely on in a flexible way.

class wlauto.core.resolver.ResourceResolver(config)

Bases: object

Discovers and registers getters, and then handles requests for resources using registered getters.

get(resource, strict=True, *args, **kwargs)

Uses registered getters to attempt to discover a resource of the specified kind and matching the specified criteria. Returns path to the resource that has been discovered. If a resource has not been discovered, this will raise a ResourceError or, if strict has been set to False, will return None.

load()

Discover getters under the specified source. The source could be either a python package/module or a path.

register(getter, kind, priority=0)

Register the specified resource getter as being able to discover a resource of the specified kind with the specified priority.

This method would typically be invoked by a getter inside its __init__. The idea being that getters register themselves for resources they know they can discover.

priorities

getters that are registered with the highest priority will be invoked first. If multiple getters are registered under the same priority, they will be invoked in the order they were registered (i.e. in the order they were discovered). This is essentially non-deterministic.

Generally getters that are more likely to find a resource, or would find a “better” version of the resource should register with higher (positive) priorities. Fall-back getters that should only be invoked if a resource is not found by usual means should register with lower (negative) priorities.

unregister(getter, kind)

Unregister a getter that has been registered earlier.

wlauto.core.resource module

class wlauto.core.resource.GetterPriority

Bases: object

Enumerates standard ResourceGetter priorities. In general, getters should register under one of these, rather than specifying other priority values.

Cached:The cached version of the resource. Look here first. This priority also implies that the resource at this location is a “cache” and is not the only version of the resource, so it may be cleared without losing access to the resource. Preferred:Take this resource in favour of the environment resource. Environment:Found somewhere under ~/.workload_automation/ or equivalent, or from environment variables, external configuration files, etc. These will override resource supplied with the package. External_package:  Resource provided by another package. Package:Resource provided with the package. Remote:Resource will be downloaded from a remote location (such as an HTTP server or a samba share). Try this only if no other getter was successful.

cached = 20

environment = 0

external_package = -5

package = -10

preferred = 10

remote = -4

class wlauto.core.resource.Resource(owner)

Bases: object

Represents a resource that needs to be resolved. This can be pretty much anything: a file, environment variable, a Python object, etc. The only thing a resource has to have is an owner (which would normally be the Workload/Instrument/Device/etc object that needs the resource). In addition, a resource have any number of attributes to identify, but all of them are resource type specific.

delete(instance)

Delete an instance of this resource type. This must be implemented by the concrete subclasses based on what the resource looks like, e.g. deleting a file or a directory tree, or removing an entry from a database.

Note:Implementation should not contain any logic for deciding whether or not a resource should be deleted, only the actual deletion. The assumption is that if this method is invoked, then the decision has already been made.

name = None

class wlauto.core.resource.ResourceGetter(resolver, **kwargs)

Bases: wlauto.core.extension.Extension

Base class for implementing resolvers. Defines resolver interface. Resolvers are responsible for discovering resources (such as particular kinds of files) they know about based on the parameters that are passed to them. Each resolver also has a dict of attributes that describe its operation, and may be used to determine which get invoked. There is no pre-defined set of attributes and resolvers may define their own.

Class attributes:

Name:Name that uniquely identifies this getter. Must be set by any concrete subclass. Resource_type:Identifies resource type(s) that this getter can handle. This must be either a string (for a single type) or a list of strings for multiple resource types. This must be set by any concrete subclass. Priority:Priority with which this getter will be invoked. This should be one of the standard priorities specified in GetterPriority enumeration. If not set, this will default to GetterPriority.environment.

aliases = AC([])

artifacts = AC([])

core_modules = []

delete(resource, *args, **kwargs)

Delete the resource if it is discovered. All arguments are passed to a call to``self.get()``. If that call returns a resource, it is deleted.

Returns:True if the specified resource has been discovered and deleted, and False otherwise.

finalize(*args, **kwargs)

get(resource, **kwargs)

This will get invoked by the resolver when attempting to resolve a resource, passing in the resource to be resolved as the first parameter. Any additional parameters would be specific to a particular resource type.

This method will only be invoked for resource types that the getter has registered for.

Parameters:resource – an instance of wlauto.core.resource.Resource. Returns:Implementations of this method must return either the discovered resource or None if the resource could not be discovered.

initialize(*args, **kwargs)

name = None

parameters = AC(["Param({'kind': <type 'list'>, 'mandatory': None, 'name': 'modules', 'constraint': None, 'default': None, 'allowed_values': None, 'global_alias': None, 'override': False})"])

priority = 0

register()

Registers with a resource resolver. Concrete implementations must override this to invoke self.resolver.register() method to register self for specific resource types.

resource_type = None

unregister()

Unregister from a resource resolver.

validate(*args, **kwargs)

wlauto.core.result module

This module defines the classes used to handle result processing inside Workload Automation. There will be a wlauto.core.workload.WorkloadResult object generated for every workload iteration executed. This object will have a list of wlauto.core.workload.WorkloadMetric objects. This list will be populated by the workload itself and may also be updated by instrumentation (e.g. to add power measurements). Once the result object has been fully populated, it will be passed into the process_iteration_result method of ResultProcessor. Once the entire run has completed, a list containing result objects from all iterations will be passed into process_results method of :class`ResultProcessor`.

Which result processors will be active is defined by the result_processors list in the ~/.workload_automation/config.py. Only the result_processors who’s names appear in this list will be used.

A ResultsManager keeps track of active results processors.

class wlauto.core.result.IterationResult(spec)

Bases: object

Contains the result of running a single iteration of a workload. It is the responsibility of a workload to instantiate a IterationResult, populate it, and return it form its get_result() method.

Status explanations:

NOT_STARTED:This iteration has not yet started. RUNNING:This iteration is currently running and no errors have been detected. OK:This iteration has completed and no errors have been detected PARTIAL:One or more instruments have failed (the iteration may still be running). FAILED:The workload itself has failed. ABORTED:The user interupted the workload SKIPPED:The iteration was skipped due to a previous failure

ABORTED = 'ABORTED'

FAILED = 'FAILED'

NONCRITICAL = 'NONCRITICAL'

NOT_STARTED = 'NOT_STARTED'

OK = 'OK'

PARTIAL = 'PARTIAL'

RUNNING = 'RUNNING'

SKIPPED = 'SKIPPED'

add_event(message)

add_metric(name, value, units=None, lower_is_better=False, classifiers=None)

has_metric(name)

to_dict()

values = ['NOT_STARTED', 'RUNNING', 'OK', 'NONCRITICAL', 'PARTIAL', 'FAILED', 'ABORTED', 'SKIPPED']

class wlauto.core.result.Metric(name, value, units=None, lower_is_better=False, classifiers=None)

Bases: object

This is a single metric collected from executing a workload.

Parameters:

• name – the name of the metric. Uniquely identifies the metric within the results.
• value – The numerical value of the metric for this execution of a workload. This can be either an int or a float.
• units – Units for the collected value. Can be None if the value has no units (e.g. it’s a count or a standardised score).
• lower_is_better – Boolean flag indicating where lower values are better than higher ones. Defaults to False.
• classifiers – A set of key-value pairs to further classify this metric beyond current iteration (e.g. this can be used to identify sub-tests).

to_dict()

class wlauto.core.result.ResultManager

Bases: object

Keeps track of result processors and passes on the results onto the individual processors.

add_result(result, context)

finalize(context)

initialize(context)

install(processor)

process_run_result(result, context)

uninstall(processor)

validate()

class wlauto.core.result.ResultProcessor(**kwargs)

Bases: wlauto.core.extension.Extension

Base class for result processors. Defines an interface that should be implemented by the subclasses. A result processor can be used to do any kind of post-processing of the results, from writing them out to a file, to uploading them to a database, performing calculations, generating plots, etc.

aliases = AC([])

artifacts = AC([])

core_modules = []

export_iteration_result(result, context)

export_run_result(result, context)

finalize(*args, **kwargs)

initialize(*args, **kwargs)

parameters = AC(["Param({'kind': <type 'list'>, 'mandatory': None, 'name': 'modules', 'constraint': None, 'default': None, 'allowed_values': None, 'global_alias': None, 'override': False})"])

process_iteration_result(result, context)

process_run_result(result, context)

validate(*args, **kwargs)

class wlauto.core.result.RunEvent(message)

Bases: object

An event that occured during a run.

to_dict()

class wlauto.core.result.RunResult(run_info, output_directory=None)

Bases: object

Contains overall results for a run.

FAILED = 'FAILED'

OK = 'OK'

OKISH = 'OKISH'

PARTIAL = 'PARTIAL'

UNKNOWN = 'UNKNOWN'

status

values = ['OK', 'OKISH', 'PARTIAL', 'FAILED', 'UNKNOWN']

wlauto.core.signal module

This module wraps louie signalling mechanism. It relies on modified version of loiue that has prioritization added to handler invocation.

class wlauto.core.signal.Signal(name, invert_priority=False)

Bases: object

This class implements the signals to be used for notifiying callbacks registered to respond to different states and stages of the execution of workload automation.

wlauto.core.signal.connect(handler, signal, sender=<class 'louie.sender.Any'>, priority=0)

Connects a callback to a signal, so that the callback will be automatically invoked when that signal is sent.

Parameters:

handler:

This can be any callable that that takes the right arguments for the signal. For most siginals this means a single argument that will be an ExecutionContext instance. But please see documentaion for individual signals in the signals reference.

signal:

The signal to which the hanlder will be subscribed. Please see signals reference for the list of standard WA signals.

Note

There is nothing that prevents instrumentation from sending their own signals that are not part of the standard set. However the signal must always be an wlauto.core.signal.Signal instance.

sender:

The handler will be invoked only for the signals emitted by this sender. By default, this is set to louie.dispatcher.Any, so the handler will be invoked for signals from any sentder.

priority:

An integer (positive or negative) the specifies the priority of the handler. Handlers with higher priority will be called before handlers with lower priority. The call order of handlers with the same priority is not specified. Defaults to 0.

Note

Priorities for some signals are inverted (so highest priority handlers get executed last). Please see signals reference for details.

wlauto.core.signal.disconnect(handler, signal, sender=<class 'louie.sender.Any'>)

Disconnect a previously connected handler form the specified signal, optionally, only for the specified sender.

Parameters:

handler:The callback to be disconnected. signal:The signal the handler is to be disconnected form. It will be an wlauto.core.signal.Signal instance. sender:If specified, the handler will only be disconnected from the signal sent by this sender.

wlauto.core.signal.send(signal, sender, *args, **kwargs)

Sends a signal, causing connected handlers to be invoked.

Paramters:

signal:Signal to be sent. This must be an instance of wlauto.core.signal.Signal or its subclasses. sender:The sender of the signal (typically, this would be self). Some handlers may only be subscribed to signals from a particular sender.

The rest of the parameters will be passed on as aruments to the handler.

wlauto.core.version module

wlauto.core.version.VersionTuple

alias of Version

wlauto.core.version.get_wa_version()

wlauto.core.workload module

A workload is the unit of execution. It represents a set of activities are are performed and measured together, as well as the necessary setup and teardown procedures. A single execution of a workload produces one wlauto.core.result.WorkloadResult that is populated with zero or more wlauto.core.result.WorkloadMetrics and/or wlauto.core.result.Artifacts by the workload and active instrumentation.

class wlauto.core.workload.Workload(device, **kwargs)

Bases: wlauto.core.extension.Extension

This is the base class for the workloads executed by the framework. Each of the methods throwing NotImplementedError must be implemented by the derived classes.

aliases = AC([])

artifacts = AC([])

check_network_connected()

core_modules = []

finalize(*args, **kwargs)

init_resources(context)

This method may be used to perform early resource discovery and initialization. This is invoked during the initial loading stage and before the device is ready, so cannot be used for any device-dependent initialization. This method is invoked before the workload instance is validated.

initialize(*args, **kwargs)

parameters = AC(["Param({'kind': <type 'list'>, 'mandatory': None, 'name': 'modules', 'constraint': None, 'default': None, 'allowed_values': None, 'global_alias': None, 'override': False})"])

requires_network = False

run(context)

Execute the workload. This is the method that performs the actual “work” of the

setup(context)

Perform the setup necessary to run the workload, such as copying the necessary files to the device, configuring the environments, etc.

This is also the place to perform any on-device checks prior to attempting to execute the workload.

summary_metrics = []

supported_devices = []

supported_platforms = []

teardown(context)

Perform any final clean up for the Workload.

update_result(context)

Update the result within the specified execution context with the metrics form this workload iteration.

validate(*args, **kwargs)

Module contents