Output¶

A WA output directory can be accessed via a RunOutput object. There are two ways of getting one – either instantiate it with a path to a WA output directory, or use discover_wa_outputs() to traverse a directory tree iterating over all WA output directories found.

discover_wa_outputs(path)¶

Recursively traverse path looking for WA output directories. Return an iterator over RunOutput objects for each discovered output.

Parameters:	path – The directory to scan for WA output

class RunOutput(path)¶

The main interface into a WA output directory.

Parameters:	path – must be the path to the top-level output directory (the one containing `__meta` subdirectory and `run.log`).

WA output stored in a Postgres database by the Postgres output processor can be accessed via a RunDatabaseOutput which can be initialized as follows:

class RunDatabaseOutput(password, host='localhost', user='postgres', port='5432', dbname='wa', run_uuid=None, list_runs=False)¶

The main interface into Postgres database containing WA results.

Parameters:

password – The password used to authenticate with
host – The database host address. Defaults to 'localhost'
user – The user name used to authenticate with. Defaults to 'postgres'
port – The database connection port number. Defaults to '5432'
dbname – The database name. Defaults to 'wa'
run_uuid – The run_uuid to identify the selected run
list_runs – Will connect to the database and will print out the available runs with their corresponding run_uuids. Defaults to False

Example¶

See also

Processing WA Output

To demonstrate how we can use the output API if we have an existing WA output called wa_output in the current working directory we can initialize a RunOutput as follows:

In [1]: from wa import RunOutput
   ...:
   ...: output_directory = 'wa_output'
   ...: run_output = RunOutput(output_directory)

Alternatively if the results have been stored in a Postgres database we can initialize a RunDatabaseOutput as follows:

In [1]: from wa import RunDatabaseOutput
   ...:
   ...: db_settings = {
   ...:                host: 'localhost',
   ...:                port: '5432',
   ...:                dbname: 'wa'
   ...:                user: 'postgres',
   ...:                password: 'wa'
   ...:                }
   ...:
   ...: RunDatabaseOutput(list_runs=True, **db_settings)
Available runs are:
========= ============ ============= =================== =================== ====================================
 Run Name      Project Project Stage          Start Time            End Time                             run_uuid
========= ============ ============= =================== =================== ====================================
Test Run    my_project          None 2018-11-29 14:53:08 2018-11-29 14:53:24 aa3077eb-241a-41d3-9610-245fd4e552a9
run_1       my_project          None 2018-11-29 14:53:34 2018-11-29 14:53:37 4c2885c9-2f4a-49a1-bbc5-b010f8d6b12a
========= ============ ============= =================== =================== ====================================

In [2]: run_uuid = '4c2885c9-2f4a-49a1-bbc5-b010f8d6b12a'
   ...: run_output = RunDatabaseOutput(run_uuid=run_uuid, **db_settings)

From here we can retrieve various information about the run. For example if we want to see what the overall status of the run was, along with the runtime parameters and the metrics recorded from the first job was we can do the following:

In [2]: run_output.status
Out[2]: OK(7)

# List all of the jobs for the run
In [3]: run_output.jobs
Out[3]:
[<wa.framework.output.JobOutput at 0x7f70358a1f10>,
 <wa.framework.output.JobOutput at 0x7f70358a1150>,
 <wa.framework.output.JobOutput at 0x7f7035862810>,
 <wa.framework.output.JobOutput at 0x7f7035875090>]

# Examine the first job that was ran
In [4]: job_1 = run_output.jobs[0]

In [5]: job_1.label
Out[5]: u'dhrystone'

# Print out all the runtime parameters and their values for this job
In [6]: for k, v in job_1.spec.runtime_parameters.items():
   ...:     print (k, v)
(u'airplane_mode': False)
(u'brightness': 100)
(u'governor': 'userspace')
(u'big_frequency': 1700000)
(u'little_frequency': 1400000)

# Print out all the metrics available for this job
In [7]: job_1.metrics
Out[7]:
[<thread 0 score: 14423105 (+)>,
 <thread 0 DMIPS: 8209 (+)>,
 <thread 1 score: 14423105 (+)>,
 <thread 1 DMIPS: 8209 (+)>,
 <thread 2 score: 14423105 (+)>,
 <thread 2 DMIPS: 8209 (+)>,
 <thread 3 score: 18292638 (+)>,
 <thread 3 DMIPS: 10411 (+)>,
 <thread 4 score: 17045532 (+)>,
 <thread 4 DMIPS: 9701 (+)>,
 <thread 5 score: 14150917 (+)>,
 <thread 5 DMIPS: 8054 (+)>,
 <time: 0.184497 seconds (-)>,
 <total DMIPS: 52793 (+)>,
 <total score: 92758402 (+)>]

# Load the run results csv file into pandas
In [7]: pd.read_csv(run_output.get_artifact_path('run_result_csv'))
Out[7]:
            id   workload  iteration          metric          value    units
0   450000-wk1  dhrystone          1  thread 0 score   1.442310e+07      NaN
1   450000-wk1  dhrystone          1  thread 0 DMIPS   8.209700e+04      NaN
2   450000-wk1  dhrystone          1  thread 1 score   1.442310e+07      NaN
3   450000-wk1  dhrystone          1  thread 1 DMIPS   8.720900e+04      NaN
...

We can also retrieve information about the target that the run was performed on for example:

# Print out the target's abi:
In [9]: run_output.target_info.abi
Out[9]: u'arm64'

# The os the target was running
In [9]: run_output.target_info.os
Out[9]: u'android'

# And other information about the os version
In [10]: run_output.target_info.os_version
Out[10]:
OrderedDict([(u'all_codenames', u'REL'),
             (u'incremental', u'3687331'),
             (u'preview_sdk', u'0'),
             (u'base_os', u''),
             (u'release', u'7.1.1'),
             (u'codename', u'REL'),
             (u'security_patch', u'2017-03-05'),
             (u'sdk', u'25')])

`RunOutput`¶

RunOutput provides access to the output of a WA run, including metrics, artifacts, metadata, and configuration. It has the following attributes:

jobs: A list of JobOutput objects for each job that was executed during the run.
status: Run status. This indicates whether the run has completed without problems (Status.OK) or if there were issues.
metrics: A list of Metrics for the run.

Note

these are overall run metrics only. Metrics for individual jobs are contained within the corresponding JobOutputs.
artifacts: A list of Artifacts for the run. These are usually backed by a file and can contain traces, raw data, logs, etc.

Note

these are overall run artifacts only. Artifacts for individual jobs are contained within the corresponding JobOutputs.
info: A RunInfo object that contains information about the run itself for example it’s duration, name, uuid etc.
target_info: A TargetInfo object which can be used to access various information about the target that was used during the run for example it’s abi, hostname, os etc.
run_config: A RunConfiguration object that can be used to access all the configuration of the run itself, for example the reboot_policy, execution_order, device_config etc.
classifiers: classifiers defined for the entire run.
metadata: metadata associated with the run.
events: A list of any events logged during the run, that are not associated with a particular job.
event_summary: A condensed summary of any events that occurred during the run.
augmentations: A list of the augmentations that were enabled during the run (these augmentations may or may not have been active for a particular job).
basepath: A (relative) path to the WA output directory backing this object.

methods¶

RunOutput.get_artifact(name)¶

Return the Artifact specified by name. This will only look at the run artifacts; this will not search the artifacts of the individual jobs.

Parameters:	name – The name of the artifact who’s path to retrieve.
Returns:	The `Artifact` with that name
Raises:	HostError – If the artifact with the specified name does not exist.

RunOutput.get_artifact_path(name)¶

Return the path to the file backing the artifact specified by name. This will only look at the run artifacts; this will not search the artifacts of the individual jobs.

Parameters:	name – The name of the artifact who’s path to retrieve.
Returns:	The path to the artifact
Raises:	HostError – If the artifact with the specified name does not exist.

RunOutput.get_metric(name)¶

Return the Metric associated with the run (not the individual jobs) with the specified name.

Returns:	The `Metric` object for the metric with the specified name.

RunOutput.get_job_spec(spec_id)¶: Return the JobSpec with the specified spec_id. A spec describes the job to be executed. Each Job has an associated JobSpec, though a single spec can be associated with multiple jobs (If the spec specifies multiple iterations).

RunOutput.list_workloads()¶

List unique workload labels that featured in this run. The labels will be in the order in which they first ran.

Returns:	A list of str labels of workloads that were part of this run.

`RunDatabaseOutput`¶

RunDatabaseOutput provides access to the output of a WA run, including metrics,artifacts, metadata, and configuration stored in a postgres database. The majority of attributes and methods are the same RunOutput however the noticeable differences are:

jobs: A list of JobDatabaseOutput objects for each job that was executed during the run.
basepath: A representation of the current database and host information backing this object.

methods¶

RunDatabaseOutput.get_artifact(name)¶

Return the Artifact specified by name. This will only look at the run artifacts; this will not search the artifacts of the individual jobs. The path attribute of the Artifact will be set to the Database OID of the object.

Parameters:	name – The name of the artifact who’s path to retrieve.
Returns:	The `Artifact` with that name
Raises:	HostError – If the artifact with the specified name does not exist.

RunDatabaseOutput.get_artifact_path(name)¶

Returns a StringIO object containing the contents of the artifact specified by name. This will only look at the run artifacts; this will not search the artifacts of the individual jobs.

Parameters:	name – The name of the artifact who’s path to retrieve.
Returns:	A StringIO object with the contents of the artifact
Raises:	HostError – If the artifact with the specified name does not exist.

`JobOutput`¶

JobOutput provides access to the output of a single job executed during a WA run, including metrics, artifacts, metadata, and configuration. It has the following attributes:

status: Job status. This indicates whether the job has completed without problems (Status.OK) or if there were issues.

Note

Under typical configuration, WA will make a number of attempts to re-run a job in case of issue. This status (and the rest of the output) will represent the the latest attempt. I.e. a Status.OK indicates that the latest attempt was successful, but it does mean that there weren’t prior failures. You can check the retry attribute (see below) to whether this was the first attempt or not.
retry: Retry number for this job. If a problem is detected during job execution, the job will be re-run up to max_retries times. This indicates the final retry number for the output. A value of 0 indicates that the job succeeded on the first attempt, and no retries were necessary.

Note

Outputs for previous attempts are moved into __failed subdirectory of WA output. These are currently not exposed via the API.
id: The ID of the spec associated with with job. This ID is unique to the spec, but not necessary to the job – jobs representing multiple iterations of the same spec will share the ID.
iteration: The iteration number of this job. Together with the id (above), this uniquely identifies a job with a run.
label: The workload label associated with this job. Usually, this will be the name or alias of the workload, however maybe overwritten by the user in the agenda.
metrics: A list of Metrics for the job.
artifacts: A list of Artifacts for the job These are usually backed by a file and can contain traces, raw data, logs, etc.
classifiers: classifiers defined for the job.
metadata: metadata associated with the job.
events: A list of any events logged during the execution of the job.
event_summary: A condensed summary of any events that occurred during the execution of the job.
augmentations: A list of the augmentations that were enabled for this job. This may be different from overall augmentations specified for the run, as they may be enabled/disabled on per-job basis.
basepath: A (relative) path to the WA output directory backing this object.

methods¶

RunOutput.get_artifact(name)

Return the Artifact specified by name associated with this job.

Parameters:	name – The name of the artifact to retrieve.
Returns:	The `Artifact` with that name
Raises:	HostError – If the artifact with the specified name does not exist.

RunOutput.get_artifact_path(name)

Return the path to the file backing the artifact specified by name, associated with this job.

Parameters:	name – The name of the artifact who’s path to retrieve.
Returns:	The path to the artifact
Raises:	HostError – If the artifact with the specified name does not exist.

RunOutput.get_metric(name)

Return the Metric associated with this job with the specified name.

Returns:	The `Metric` object for the metric with the specified name.

`JobDatabaseOutput`¶

JobOutput provides access to the output of a single job executed during a WA run, including metrics, artifacts, metadata, and configuration stored in a postgres database. The majority of attributes and methods are the same JobOutput however the noticeable differences are:

basepath: A representation of the current database and host information backing this object.

methods¶

JobDatabaseOutput.get_artifact(name)¶

Return the Artifact specified by name associated with this job. The path attribute of the Artifact will be set to the Database OID of the object.

Parameters:	name – The name of the artifact to retrieve.
Returns:	The `Artifact` with that name
Raises:	HostError – If the artifact with the specified name does not exist.

JobDatabaseOutput.get_artifact_path(name)¶

Returns a StringIO object containing the contents of the artifact specified by name associated with this job.

Parameters:	name – The name of the artifact who’s path to retrieve.
Returns:	A StringIO object with the contents of the artifact
Raises:	HostError – If the artifact with the specified name does not exist.

`Metric`¶

A metric represent a single numerical measurement/score collected as a result of running the workload. It would be generated either by the workload or by one of the augmentations active during the execution of the workload.

A Metric has the following attributes:

name: The name of the metric.

Note

A name of the metric is not necessarily unique, even for the same job. Some workloads internally run multiple sub-tests, each generating a metric with the same name. In such cases, classifiers are used to distinguish between them.
value: The value of the metrics collected.
units: The units of the metrics. This maybe None if the metric has no units.
lower_is_better: The default assumption is that higher metric values are better. This may be overridden by setting this to True, e.g. if metrics such as “run time” or “latency”. WA does not use this internally (at the moment) but this may be used by external parties to sensibly process WA results in a generic way.
classifiers: These can be user-defined classifiers propagated from the job/run, or they may have been added by the workload to help distinguish between otherwise identical metrics.
label: This is a string constructed from the name and classifiers, to provide a more unique identifier, e.g. for grouping values across iterations. The format is in the form name/cassifier1=value1/classifier2=value2/....

`Artifact`¶

An artifact is a file that is created on the host as part of executing a workload. This could be trace, logging, raw output, or pretty much anything else. Pretty much every file under WA output directory that is not already represented by some other framework object will have an Artifact associated with it.

An Artifact has the following attributes:

name

The name of this artifact. This will be unique for the job/run (unlike metric names). This is intended as a consistent “handle” for this artifact. The actual file name for the artifact may vary from job to job (e.g. some benchmarks that create files with results include timestamps in the file names), however the name will always be the same.

path

Partial path to the file associated with this artifact. Often, this is just the file name. To get the complete path that maybe used to access the file, use get_artifact_path() of the corresponding output object.

kind

Describes the nature of this artifact to facilitate generic processing. Possible kinds are:

log:	A log file. Not part of the “output” as such but contains information about the run/workload execution that be useful for diagnostics/meta analysis.
meta:	A file containing metadata. This is not part of the “output”, but contains information that may be necessary to reproduce the results (contrast with `log` artifacts which are not necessary).
data:	This file contains new data, not available otherwise and should be considered part of the “output” generated by WA. Most traces would fall into this category.
export:	Exported version of results or some other artifact. This signifies that this artifact does not contain any new data that is not available elsewhere and that it may be safely discarded without losing information.
raw:	Signifies that this is a raw dump/log that is normally processed to extract useful information and is then discarded. In a sense, it is the opposite of `export`, but in general may also be discarded. Note Whether a file is marked as `log`/`data` or `raw` depends on how important it is to preserve this file, e.g. when archiving, vs how much space it takes up. Unlike `export` artifacts which are (almost) always ignored by other exporters as that would never result in data loss, `raw` files may be processed by exporters if they decided that the risk of losing potentially (though unlikely) useful data is greater than the time/space cost of handling the artifact (e.g. a database uploader may choose to ignore `raw` artifacts, where as a network filer archiver may choose to archive them).

Note

The kind parameter is intended to represent the logical function of a particular artifact, not it’s intended means of processing – this is left entirely up to the output processors.

description

This may be used by the artifact’s creator to provide additional free-form information about the artifact. In practice, this is often None

classifiers

Job- and run-level classifiers will be propagated to the artifact.

Additional run info¶

RunOutput object has target_info and run_info attributes that contain structures that provide additional information about the run and device.

`TargetInfo`¶

The TargetInfo class presents various pieces of information about the target device. An instance of this class will be instantiated and populated automatically from the devlib target created during a WA run and serialized to a json file as part of the metadata exported by WA at the end of a run.

The available attributes of the class are as follows:

target: The name of the target class that was uised ot interact with the device during the run E.g. "AndroidTarget", "LinuxTarget" etc.
cpus: A list of CpuInfo objects describing the capabilities of each CPU.
os: A generic name of the OS the target was running (e.g. "android").
os_version: A dict that contains a mapping of OS version elements to their values. This mapping is OS-specific.
abi: The ABI of the target device.
hostname: The hostname of the the device the run was executed on.
is_rooted: A boolean value specifying whether root was detected on the device.
kernel_version: The version of the kernel on the target device. This returns a KernelVersion instance that has separate version and release fields.
kernel_config: A KernelConfig instance that contains parsed kernel config from the target device. This may be None if the kernel config could not be extracted.
sched_features: A list of the available tweaks to the scheduler, if available from the device.
hostid: The unique identifier of the particular device the WA run was executed on.

`RunInfo`¶

The RunInfo provides general run information. It has the following attributes:

uuid: A unique identifier for that particular run.
run_name: The name of the run (if provided)
project: The name of the project the run belongs to (if provided)
project_stage: The project stage the run is associated with (if provided)
duration: The length of time the run took to complete.
start_time: The time the run was stared.
end_time: The time at which the run finished.

Output¶

Example¶

RunOutput¶

methods¶

RunDatabaseOutput¶

methods¶

JobOutput¶

methods¶

JobDatabaseOutput¶

methods¶

Metric¶

Artifact¶

Additional run info¶

TargetInfo¶

RunInfo¶

`RunOutput`¶

`RunDatabaseOutput`¶

`JobOutput`¶

`JobDatabaseOutput`¶

`Metric`¶

`Artifact`¶

`TargetInfo`¶

`RunInfo`¶