Output
A WA output directory can be accessed via a RunOutput
object. There are
two ways of getting one – either instantiate it with a path to a WA output
directory, or use discover_wa_outputs()
to traverse a directory tree
iterating over all WA output directories found.
- discover_wa_outputs(path)
Recursively traverse
path
looking for WA output directories. Return an iterator overRunOutput
objects for each discovered output.- Parameters
path – The directory to scan for WA output
- class RunOutput(path)
The main interface into a WA output directory.
- Parameters
path – must be the path to the top-level output directory (the one containing
__meta
subdirectory andrun.log
).
WA output stored in a Postgres database by the Postgres
output processor
can be accessed via a RunDatabaseOutput
which can be initialized as follows:
- class RunDatabaseOutput(password, host='localhost', user='postgres', port='5432', dbname='wa', run_uuid=None, list_runs=False)
The main interface into Postgres database containing WA results.
- Parameters
password – The password used to authenticate with
host – The database host address. Defaults to
'localhost'
user – The user name used to authenticate with. Defaults to
'postgres'
port – The database connection port number. Defaults to
'5432'
dbname – The database name. Defaults to
'wa'
run_uuid – The
run_uuid
to identify the selected runlist_runs – Will connect to the database and will print out the available runs with their corresponding run_uuids. Defaults to
False
Example
See also
To demonstrate how we can use the output API if we have an existing WA output
called wa_output
in the current working directory we can initialize a
RunOutput
as follows:
In [1]: from wa import RunOutput
...:
...: output_directory = 'wa_output'
...: run_output = RunOutput(output_directory)
Alternatively if the results have been stored in a Postgres database we can
initialize a RunDatabaseOutput
as follows:
In [1]: from wa import RunDatabaseOutput
...:
...: db_settings = {
...: host: 'localhost',
...: port: '5432',
...: dbname: 'wa'
...: user: 'postgres',
...: password: 'wa'
...: }
...:
...: RunDatabaseOutput(list_runs=True, **db_settings)
Available runs are:
========= ============ ============= =================== =================== ====================================
Run Name Project Project Stage Start Time End Time run_uuid
========= ============ ============= =================== =================== ====================================
Test Run my_project None 2018-11-29 14:53:08 2018-11-29 14:53:24 aa3077eb-241a-41d3-9610-245fd4e552a9
run_1 my_project None 2018-11-29 14:53:34 2018-11-29 14:53:37 4c2885c9-2f4a-49a1-bbc5-b010f8d6b12a
========= ============ ============= =================== =================== ====================================
In [2]: run_uuid = '4c2885c9-2f4a-49a1-bbc5-b010f8d6b12a'
...: run_output = RunDatabaseOutput(run_uuid=run_uuid, **db_settings)
From here we can retrieve various information about the run. For example if we want to see what the overall status of the run was, along with the runtime parameters and the metrics recorded from the first job was we can do the following:
In [2]: run_output.status
Out[2]: OK(7)
# List all of the jobs for the run
In [3]: run_output.jobs
Out[3]:
[<wa.framework.output.JobOutput at 0x7f70358a1f10>,
<wa.framework.output.JobOutput at 0x7f70358a1150>,
<wa.framework.output.JobOutput at 0x7f7035862810>,
<wa.framework.output.JobOutput at 0x7f7035875090>]
# Examine the first job that was ran
In [4]: job_1 = run_output.jobs[0]
In [5]: job_1.label
Out[5]: u'dhrystone'
# Print out all the runtime parameters and their values for this job
In [6]: for k, v in job_1.spec.runtime_parameters.items():
...: print (k, v)
(u'airplane_mode': False)
(u'brightness': 100)
(u'governor': 'userspace')
(u'big_frequency': 1700000)
(u'little_frequency': 1400000)
# Print out all the metrics available for this job
In [7]: job_1.metrics
Out[7]:
[<thread 0 score: 14423105 (+)>,
<thread 0 DMIPS: 8209 (+)>,
<thread 1 score: 14423105 (+)>,
<thread 1 DMIPS: 8209 (+)>,
<thread 2 score: 14423105 (+)>,
<thread 2 DMIPS: 8209 (+)>,
<thread 3 score: 18292638 (+)>,
<thread 3 DMIPS: 10411 (+)>,
<thread 4 score: 17045532 (+)>,
<thread 4 DMIPS: 9701 (+)>,
<thread 5 score: 14150917 (+)>,
<thread 5 DMIPS: 8054 (+)>,
<time: 0.184497 seconds (-)>,
<total DMIPS: 52793 (+)>,
<total score: 92758402 (+)>]
# Load the run results csv file into pandas
In [7]: pd.read_csv(run_output.get_artifact_path('run_result_csv'))
Out[7]:
id workload iteration metric value units
0 450000-wk1 dhrystone 1 thread 0 score 1.442310e+07 NaN
1 450000-wk1 dhrystone 1 thread 0 DMIPS 8.209700e+04 NaN
2 450000-wk1 dhrystone 1 thread 1 score 1.442310e+07 NaN
3 450000-wk1 dhrystone 1 thread 1 DMIPS 8.720900e+04 NaN
...
We can also retrieve information about the target that the run was performed on for example:
# Print out the target's abi:
In [9]: run_output.target_info.abi
Out[9]: u'arm64'
# The os the target was running
In [9]: run_output.target_info.os
Out[9]: u'android'
# And other information about the os version
In [10]: run_output.target_info.os_version
Out[10]:
OrderedDict([(u'all_codenames', u'REL'),
(u'incremental', u'3687331'),
(u'preview_sdk', u'0'),
(u'base_os', u''),
(u'release', u'7.1.1'),
(u'codename', u'REL'),
(u'security_patch', u'2017-03-05'),
(u'sdk', u'25')])
RunOutput
RunOutput
provides access to the output of a WA run, including metrics,
artifacts, metadata, and configuration. It has the following attributes:
jobs
A list of
JobOutput
objects for each job that was executed during the run.status
Run status. This indicates whether the run has completed without problems (
Status.OK
) or if there were issues.metrics
A list of
Metric
s for the run.Note
these are overall run metrics only. Metrics for individual jobs are contained within the corresponding
JobOutput
s.artifacts
A list of
Artifact
s for the run. These are usually backed by a file and can contain traces, raw data, logs, etc.Note
these are overall run artifacts only. Artifacts for individual jobs are contained within the corresponding
JobOutput
s.info
A RunInfo object that contains information about the run itself for example it’s duration, name, uuid etc.
target_info
A TargetInfo object which can be used to access various information about the target that was used during the run for example it’s
abi
,hostname
,os
etc.run_config
A RunConfiguration object that can be used to access all the configuration of the run itself, for example the
reboot_policy
,execution_order
,device_config
etc.classifiers
classifiers defined for the entire run.
metadata
metadata associated with the run.
events
A list of any events logged during the run, that are not associated with a particular job.
event_summary
A condensed summary of any events that occurred during the run.
augmentations
A list of the augmentations that were enabled during the run (these augmentations may or may not have been active for a particular job).
basepath
A (relative) path to the WA output directory backing this object.
methods
- RunOutput.get_artifact(name)
Return the
Artifact
specified byname
. This will only look at the run artifacts; this will not search the artifacts of the individual jobs.- Parameters
name – The name of the artifact who’s path to retrieve.
- Returns
The
Artifact
with that name- Raises
HostError – If the artifact with the specified name does not exist.
- RunOutput.get_artifact_path(name)
Return the path to the file backing the artifact specified by
name
. This will only look at the run artifacts; this will not search the artifacts of the individual jobs.- Parameters
name – The name of the artifact who’s path to retrieve.
- Returns
The path to the artifact
- Raises
HostError – If the artifact with the specified name does not exist.
- RunOutput.get_metric(name)
Return the
Metric
associated with the run (not the individual jobs) with the specified name.- Returns
The
Metric
object for the metric with the specified name.
- RunOutput.get_job_spec(spec_id)
Return the
JobSpec
with the specified spec_id. A spec describes the job to be executed. EachJob
has an associatedJobSpec
, though a single spec can be associated with multiple jobs (If the spec specifies multiple iterations).
- RunOutput.list_workloads()
List unique workload labels that featured in this run. The labels will be in the order in which they first ran.
- Returns
A list of str labels of workloads that were part of this run.
- RunOutput.add_classifier(name, value, overwrite=False)
Add a classifier to the run as a whole. If a classifier with the specified
name
already exists, a``ValueError`` will be raised, unless overwrite=True is specified.
RunDatabaseOutput
RunDatabaseOutput
provides access to the output of a WA run,
including metrics,artifacts, metadata, and configuration stored in a postgres database.
The majority of attributes and methods are the same RunOutput
however the
noticeable differences are:
jobs
A list of
JobDatabaseOutput
objects for each job that was executed during the run.basepath
A representation of the current database and host information backing this object.
methods
- RunDatabaseOutput.get_artifact(name)
Return the
Artifact
specified byname
. This will only look at the run artifacts; this will not search the artifacts of the individual jobs. The path attribute of theArtifact
will be set to the Database OID of the object.- Parameters
name – The name of the artifact who’s path to retrieve.
- Returns
The
Artifact
with that name- Raises
HostError – If the artifact with the specified name does not exist.
- RunDatabaseOutput.get_artifact_path(name)
If the artifcat is a file this method returns a StringIO object containing the contents of the artifact specified by
name
. If the aritifcat is a directory, the method returns a path to a locally extracted version of the directory which is left to the user to remove after use. This will only look at the run artifacts; this will not search the artifacts of the individual jobs.- Parameters
name – The name of the artifact who’s path to retrieve.
- Returns
A StringIO object with the contents of the artifact
- Raises
HostError – If the artifact with the specified name does not exist.
JobOutput
JobOutput
provides access to the output of a single job
executed during a WA run, including metrics,
artifacts, metadata, and configuration. It has the following attributes:
status
Job status. This indicates whether the job has completed without problems (
Status.OK
) or if there were issues.Note
Under typical configuration, WA will make a number of attempts to re-run a job in case of issue. This status (and the rest of the output) will represent the the latest attempt. I.e. a
Status.OK
indicates that the latest attempt was successful, but it does mean that there weren’t prior failures. You can check theretry
attribute (see below) to whether this was the first attempt or not.retry
Retry number for this job. If a problem is detected during job execution, the job will be re-run up to
max_retries
times. This indicates the final retry number for the output. A value of0
indicates that the job succeeded on the first attempt, and no retries were necessary.Note
Outputs for previous attempts are moved into
__failed
subdirectory of WA output. These are currently not exposed via the API.id
The ID of the spec associated with with job. This ID is unique to the spec, but not necessary to the job – jobs representing multiple iterations of the same spec will share the ID.
iteration
The iteration number of this job. Together with the
id
(above), this uniquely identifies a job with a run.label
The workload label associated with this job. Usually, this will be the name or alias of the workload, however maybe overwritten by the user in the agenda.
metrics
A list of
Metric
s for the job.artifacts
A list of
Artifact
s for the job These are usually backed by a file and can contain traces, raw data, logs, etc.classifiers
classifiers defined for the job.
metadata
metadata associated with the job.
events
A list of any events logged during the execution of the job.
event_summary
A condensed summary of any events that occurred during the execution of the job.
augmentations
A list of the augmentations that were enabled for this job. This may be different from overall augmentations specified for the run, as they may be enabled/disabled on per-job basis.
basepath
A (relative) path to the WA output directory backing this object.
methods
- JobOutput.get_artifact(name)
Return the
Artifact
specified byname
associated with this job.- Parameters
name – The name of the artifact to retrieve.
- Returns
The
Artifact
with that name- Raises
HostError – If the artifact with the specified name does not exist.
- JobOutput.get_artifact_path(name)
Return the path to the file backing the artifact specified by
name
, associated with this job.- Parameters
name – The name of the artifact who’s path to retrieve.
- Returns
The path to the artifact
- Raises
HostError – If the artifact with the specified name does not exist.
- JobOutput.get_metric(name)
Return the
Metric
associated with this job with the specified name.- Returns
The
Metric
object for the metric with the specified name.
- JobOutput.add_classifier(name, value, overwrite=False)
Add a classifier to the job. The classifier will be propagated to all existing artifacts and metrics, as well as those added afterwards. If a classifier with the specified
name
already exists, aValueError
will be raised, unless overwrite=True is specified.
JobDatabaseOutput
JobOutput
provides access to the output of a single job
executed during a WA run, including metrics, artifacts, metadata, and
configuration stored in a postgres database.
The majority of attributes and methods are the same JobOutput
however the
noticeable differences are:
basepath
A representation of the current database and host information backing this object.
methods
- JobDatabaseOutput.get_artifact(name)
Return the
Artifact
specified byname
associated with this job. The path attribute of theArtifact
will be set to the Database OID of the object.- Parameters
name – The name of the artifact to retrieve.
- Returns
The
Artifact
with that name- Raises
HostError – If the artifact with the specified name does not exist.
- JobDatabaseOutput.get_artifact_path(name)
If the artifcat is a file this method returns a StringIO object containing the contents of the artifact specified by
name
associated with this job. If the aritifcat is a directory, the method returns a path to a locally extracted version of the directory which is left to the user to remove after use.- Parameters
name – The name of the artifact who’s path to retrieve.
- Returns
A StringIO object with the contents of the artifact
- Raises
HostError – If the artifact with the specified name does not exist.
Metric
A metric represent a single numerical measurement/score collected as a result of running the workload. It would be generated either by the workload or by one of the augmentations active during the execution of the workload.
A Metric
has the following attributes:
name
The name of the metric.
Note
A name of the metric is not necessarily unique, even for the same job. Some workloads internally run multiple sub-tests, each generating a metric with the same name. In such cases, classifiers are used to distinguish between them.
value
The value of the metrics collected.
units
The units of the metrics. This maybe
None
if the metric has no units.lower_is_better
The default assumption is that higher metric values are better. This may be overridden by setting this to
True
, e.g. if metrics such as “run time” or “latency”. WA does not use this internally (at the moment) but this may be used by external parties to sensibly process WA results in a generic way.classifiers
These can be user-defined classifiers propagated from the job/run, or they may have been added by the workload to help distinguish between otherwise identical metrics.
label
This is a string constructed from the name and classifiers, to provide a more unique identifier, e.g. for grouping values across iterations. The format is in the form
name/cassifier1=value1/classifier2=value2/...
.
Artifact
An artifact is a file that is created on the host as part of executing a
workload. This could be trace, logging, raw output, or pretty much anything
else. Pretty much every file under WA output directory that is not already
represented by some other framework object will have an Artifact
associated with it.
An Artifact
has the following attributes:
name
The name of this artifact. This will be unique for the job/run (unlike metric names). This is intended as a consistent “handle” for this artifact. The actual file name for the artifact may vary from job to job (e.g. some benchmarks that create files with results include timestamps in the file names), however the name will always be the same.
path
Partial path to the file associated with this artifact. Often, this is just the file name. To get the complete path that maybe used to access the file, use
get_artifact_path()
of the corresponding output object.kind
Describes the nature of this artifact to facilitate generic processing. Possible kinds are:
- log
A log file. Not part of the “output” as such but contains information about the run/workload execution that be useful for diagnostics/meta analysis.
- meta
A file containing metadata. This is not part of the “output”, but contains information that may be necessary to reproduce the results (contrast with
log
artifacts which are not necessary).- data
This file contains new data, not available otherwise and should be considered part of the “output” generated by WA. Most traces would fall into this category.
- export
Exported version of results or some other artifact. This signifies that this artifact does not contain any new data that is not available elsewhere and that it may be safely discarded without losing information.
- raw
Signifies that this is a raw dump/log that is normally processed to extract useful information and is then discarded. In a sense, it is the opposite of
export
, but in general may also be discarded.Note
Whether a file is marked as
log
/data
orraw
depends on how important it is to preserve this file, e.g. when archiving, vs how much space it takes up. Unlikeexport
artifacts which are (almost) always ignored by other exporters as that would never result in data loss,raw
files may be processed by exporters if they decided that the risk of losing potentially (though unlikely) useful data is greater than the time/space cost of handling the artifact (e.g. a database uploader may choose to ignoreraw
artifacts, where as a network filer archiver may choose to archive them).
Note
The kind parameter is intended to represent the logical function of a particular artifact, not it’s intended means of processing – this is left entirely up to the output processors.
description
This may be used by the artifact’s creator to provide additional free-form information about the artifact. In practice, this is often
None
classifiers
Job- and run-level classifiers will be propagated to the artifact.
Additional run info
RunOutput
object has target_info
and run_info
attributes that
contain structures that provide additional information about the run and device.
TargetInfo
The TargetInfo
class presents various pieces of information about the
target device. An instance of this class will be instantiated and populated
automatically from the devlib target created during a WA run
and serialized to a json file as part of the metadata exported
by WA at the end of a run.
The available attributes of the class are as follows:
target
The name of the target class that was uised ot interact with the device during the run E.g.
"AndroidTarget"
,"LinuxTarget"
etc.modules
A list of names of modules that have been loaded by the target. Modules provide additional functionality, such as access to
cpufreq
and which modules are installed may impact how much of theTargetInfo
has been populated.cpus
A list of
CpuInfo
objects describing the capabilities of each CPU.os
A generic name of the OS the target was running (e.g.
"android"
).os_version
A dict that contains a mapping of OS version elements to their values. This mapping is OS-specific.
abi
The ABI of the target device.
hostname
The hostname of the the device the run was executed on.
is_rooted
A boolean value specifying whether root was detected on the device.
kernel_version
The version of the kernel on the target device. This returns a
KernelVersion
instance that has separate version and release fields.kernel_config
A
KernelConfig
instance that contains parsed kernel config from the target device. This may beNone
if the kernel config could not be extracted.sched_features
A list of the available tweaks to the scheduler, if available from the device.
hostid
The unique identifier of the particular device the WA run was executed on.
RunInfo
The RunInfo
provides general run information. It has the following
attributes:
uuid
A unique identifier for that particular run.
run_name
The name of the run (if provided)
project
The name of the project the run belongs to (if provided)
project_stage
The project stage the run is associated with (if provided)
duration
The length of time the run took to complete.
start_time
The time the run was stared.
end_time
The time at which the run finished.