.. _result_processors: Result_processors ================= cpustates --------- Process power ftrace to produce CPU state and parallelism stats. Parses trace-cmd output to extract power events and uses those to generate statistics about parallelism and frequency/idle core residency. .. note:: trace-cmd instrument must be enabled and configured to collect at least ``power:cpu_idle`` and ``power:cpu_frequency`` events. Reporting should also be enabled (it is by default) as ``cpustate`` parses the text version of the trace. Finally, the device should have ``cpuidle`` module installed. This generates two reports for the run: *parallel.csv* Shows what percentage of time was spent with N cores active (for N from 0 to the total number of cores), for a cluster or for a system as a whole. It contain the following columns: :workload: The workload label :iteration: iteration that was run :cluster: The cluster for which statics are reported. The value of ``"all"`` indicates that this row reports statistics for the whole system. :number_of_cores: number of cores active. ``0`` indicates the cluster was idle. :total_time: Total time spent in this state during workload execution :%time: Percentage of total workload execution time spent in this state :%running_time: Percentage of the time the cluster was active (i.e. ignoring time the cluster was idling) spent in this state. *cpustate.csv* Shows percentage of the time a core spent in a particular power state. The first column names the state is followed by a column for each core. Power states include available DVFS frequencies (for heterogeneous systems, this is the union of frequencies supported by different core types) and idle states. Some shallow states (e.g. ARM WFI) will consume different amount of power depending on the current OPP. For such states, there will be an entry for each opp. ``"unknown"`` indicates the percentage of time for which a state could not be established from the trace. This is usually due to core state being unknown at the beginning of the trace, but may also be caused by dropped events in the middle of the trace. parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. first_cluster_state : integer The first idle state which is common to a cluster. default: ``2`` first_system_state : integer The first idle state which is common to all cores. default: ``3`` write_iteration_reports : boolean By default, this instrument will generate reports for the entire run in the overall output directory. Enabling this option will, in addition, create reports in each iteration's output directory. The formats of these reports will be similar to the overall report, except they won't mention the workload name or iteration number (as that is implied by their location). use_ratios : boolean By default proportional values will be reported as percentages, if this flag is enabled, they will be reported as ratios instead. create_timeline : boolean Create a CSV with the timeline of core power states over the course of the run as well as the usual stats reports. default: ``True`` create_utilization_timeline : boolean Create a CSV with the timeline of cpu(s) utilisation over the course of the run as well as the usual stats reports. The values generated are floating point numbers, normalised based on the maximum frequency of the cluster. start_marker_handling : str The trace-cmd instrument inserts a marker into the trace to indicate the beginning of workload execution. In some cases, this marker may be missing in the final output (e.g. due to trace buffer overrun). This parameter specifies how a missing start marker will be handled: :`ignore`: The start marker will be ignored. All events in the trace will be used. :`error`: An error will be raised if the start marker is not found in the trace. :`try`: If the start marker is not found, all events in the trace will be used. allowed values: ``'ignore'``, ``'try'``, ``'error'`` default: ``'try'`` no_idle : boolean Indicate that there will be no idle transitions in the trace. By default, a core will be reported as being in an "unknown" state until the first idle transtion for that core. Normally, this is not an issue, as cores are "nudged" as part of the setup to ensure that there is an idle transtion before the meassured region. However, if all idle states for the core have been disabled, or if the kernel does not have cpuidle, the nudge will not result in an idle transition, which would cause the cores to be reported to be in "unknown" state for the entire execution. If this parameter is set to ``True``, the processor will assuming that cores are running prior to the begining of the issue, and they will leave unknown state on the first frequency transition. csv --- Creates a ``results.csv`` in the output directory containing results for all iterations in CSV format, each line containing a single metric. parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. use_all_classifiers : boolean If set to ``True``, this will add a column for every classifier that features in at least one collected metric. .. note:: This cannot be ``True`` if ``extra_columns`` is set. extra_columns : list_of_strs List of classifiers to use as columns. .. note:: This cannot be set if ``use_all_classifiers`` is ``True``. dvfs ---- Reports DVFS state residency data form ftrace power events. This generates a ``dvfs.csv`` in the top-level results directory that, for each workload iteration, reports the percentage of time each CPU core spent in each of the DVFS frequency states (P-states), as well as percentage of the time spent in idle, during the execution of the workload. .. note:: ``trace-cmd`` instrument *MUST* be enabled in the instrumentation, and at least ``'power*'`` events must be enabled. parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. ipynb_exporter -------------- Generates an IPython notebook from a template with the results and runs it. Optionally it can show the resulting notebook in a web browser. It can also generate a PDF from the notebook. The template syntax is that of `jinja2 `_ and the template should generate a valid ipython notebook. The templates receives ``result`` and ``context`` which correspond to the RunResult and ExecutionContext respectively. You can use those in your ipython notebook template to extract any information you want to parse or show. This results_processor depends on ``ipython`` and ``python-jinja2`` being installed on the system. For example, a simple template that plots a bar graph of the results is:: { "metadata": { "name": "" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "code", "collapsed": false, "input": [ "%pylab inline" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 1 }, { "cell_type": "code", "collapsed": false, "input": [ "results = {", {% for ir in result.iteration_results -%} {% for metric in ir.metrics -%} {% if metric.name in ir.workload.summary_metrics or not ir.workload.summary_metrics -%} "\"{{ ir.spec.label }}_{{ ir.id }}_{{ ir.iteration }}_{{ metric.name }}\": {{ metric.value }}, ", {%- endif %} {%- endfor %} {%- endfor %} "}\n", "width = 0.7\n", "ind = np.arange(len(results))" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 2 }, { "cell_type": "code", "collapsed": false, "input": [ "fig, ax = plt.subplots()\n", "ax.bar(ind, results.values(), width)\n", "ax.set_xticks(ind + width/2)\n", "_ = ax.set_xticklabels(results.keys())" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 3 } ], "metadata": {} } ] } parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. notebook_template : file_path Filename of the ipython notebook template. If no `notebook_template` is specified, the example template above is used. default: ``'template.ipynb'`` notebook_name_prefix : str Prefix of the name of the notebook. The date, time and ``.ipynb`` are appended to form the notebook filename. E.g. if notebook_name_prefix is ``result_`` then a run on 13th April 2015 at 9:54 would generate a notebook called ``result_150413-095400.ipynb``. When generating a PDF, the resulting file will have the same name, but ending in ``.pdf``. default: ``'result_'`` show_notebook : boolean Open a web browser with the resulting notebook. notebook_directory : file_path Path to the notebooks directory served by the ipython notebook server. You must set it if ``show_notebook`` is selected. The ipython notebook will be copied here if specified. notebook_url : str URL of the notebook on the IPython server. If not specified, it will be assumed to be in the root notebooks location on localhost, served on port 8888. Only needed if ``show_notebook`` is selected. .. note:: the URL should not contain the final part (the notebook name) which will be populated automatically. default: ``'http://localhost:8888/notebooks'`` convert_to_html : boolean Convert the resulting notebook to HTML. show_html : boolean Open the exported html notebook at the end of the run. This can only be selected if convert_to_html has also been selected. convert_to_pdf : boolean Convert the resulting notebook to PDF. show_pdf : boolean Open the pdf at the end of the run. This can only be selected if convert_to_pdf has also been selected. json ---- Creates a ``results.json`` in the output directory containing results for all iterations in JSON format. parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. mongodb ------- Uploads run results to a MongoDB instance. MongoDB is a popular document-based data store (NoSQL database). parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. uri : str Connection URI. If specified, this will be used for connecting to the backend, and host/port parameters will be ignored. host : str (mandatory) IP address/name of the machinge hosting the MongoDB server. default: ``'localhost'`` port : integer (mandatory) Port on which the MongoDB server is listening. default: ``27017`` db : str (mandatory) Database on the server used to store WA results. default: ``'wa'`` extra_params : dict Additional connection parameters may be specfied using this (see pymongo documentation. authentication : dict If specified, this will be passed to db.authenticate() upon connection; please pymongo documentaion authentication examples for detail. notify ------ Display a desktop notification when the run finishes Notifications only work in linux systems. It uses the generic freedesktop notification specification. For this results processor to work, you need to have python-notify installed in your system. parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. sqlite ------ Stores results in an sqlite database. This may be used accumulate results of multiple runs in a single file. parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. database : str Full path to the sqlite database to be used. If this is not specified then a new database file will be created in the output directory. This setting can be used to accumulate results from multiple runs in a single database. If the specified file does not exist, it will be created, however the directory of the file must exist. .. note:: The value must resolve to an absolute path, relative paths are not allowed; however the value may contain environment variables and/or the home reference ~. overwrite : boolean If ``True``, this will overwrite the database file if it already exists. If ``False`` (the default) data will be added to the existing file (provided schema versions match -- otherwise an error will be raised). standard -------- Creates a ``result.txt`` file for every iteration that contains metrics for that iteration. The metrics are written in :: metric = value [units] format. parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. status ------ Outputs a txt file containing general status information about which runs failed and which were successful parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. summary_csv ----------- Similar to csv result processor, but only contains workloads' summary metrics. parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. syeg_csv -------- Generates a CSV results file in the format expected by SYEG toolchain. Multiple iterations get parsed into columns, adds additional columns for mean and standard deviation, append number of threads to metric names (where applicable) and add some metadata based on external mapping files. parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. outfile : str The name of the output CSV file. default: ``'syeg_out.csv'`` uxperf ------ Parse logcat for UX_PERF markers to produce performance metrics for workload actions using specified instrumentation. An action represents a series of UI interactions to capture. NOTE: The UX_PERF markers are turned off by default and must be enabled in a agenda file by setting ``markers_enabled`` for the workload to ``True``. parameters ~~~~~~~~~~ modules : list Lists the modules to be loaded by this extension. A module is a plug-in that further extends functionality of an extension. add_timings : boolean If set to ``True``, add per-action timings to result metrics.' default: ``True`` add_frames : boolean If set to ``True``, add per-action frame statistics to result metrics. i.e. fps, frame_count, jank and not_at_vsync. NOTE: This option requires the fps instrument to be enabled. drop_threshold : numeric Data points below this FPS will be dropped as they do not constitute "real" gameplay. The assumption being that while actually running, the FPS in the game will not drop below X frames per second, except on loading screens, menus, etc, which should not contribute to FPS calculation. default: ``5`` generate_csv : boolean If set to ``True``, this will produce temporal per-action fps data in the results directory, in a file named _fps.csv. Note: per-action fps data will appear as discrete step-like values in order to produce a more meainingfull representation, a rolling mean can be applied. default: ``True``