Problem Overview
As software evolves, problematic changes can significantly degrade software performance, i.e., introducing performance regression. Performance regression testing is an effective way to reveal such issues in an early stage. Yet because of its high overhead, this activity is usually performed infrequently. Consequently, when a regression problem is spotted at a certain point, multiple commits might have been merged since last testing. Developers have to spend extra time and efforts narrowing down which commit caused the problem. Existing efforts try to improve performance regression testing efficiency through test case reduction or prioritization.
Solution Overview
We propose a new lightweight and white-box approach, performance risk analysis (PRA), to improve performance regression testing efficiency via testing target prioritization. The analysis statically evaluates a given source code commit’s risk in introducing performance regression. Performance regression testing can leverage the analysis result to test commits with high risks first while delaying or skipping testing on low-risk commits.
We present PerfScope, a tool that statically analyzes a given patch's performance impact to existing source repository. Running PerfScope on 600 randomly picked new commits from six large-scale software, with our tool, developers just need to test only 14-24% of the 600 commits and will still be able to alert 87-95% of the new problematic ones.
Publication
-
Performance Regression Testing Target Prioritization via Performance Risk Analysis
Peng Huang, Xiao Ma, Dongcai Shen, and Yuanyuan Zhou.
Proceedings of the 36th International Conference on Software Engineering (ICSE’14), Hyderabad, India.
Download
Coming soon...
Tool Details
PerfScope consists of several parts to operate:
Parser: parses the patch file for information like what file(s) are modified, which location does it lie, etc. Located: -- lib/parser/PatchCompiler, Debug+Asserts/bin/patch-c -- lib/parser/PatchIRDecoder, library
Mapper: maps the change location in source file to the corresponding programming constructs. Located: -- lib/mapper, library
Filter: prunes out insignificant changes such as stylish changes or renaming. Located: -- tools/PerfDiff, Debug+Asserts/bin/perfdiff
Profile Database: contains a list of frequent or expensive functions. White-list functions can also be added to the profile. Located: -- data
Analyzer: performs Performance Risk Analysis (PRA) on given commit. Located: -- tools/PerfScope, Debug+Asserts/bin/perfscope
TestDB: tracks performance test cases execution information to further recommend which test case may be useful to uncover the performance regression issues. (work in progress)
Two auxiliary tools:
- ListFiles: list the file names and paths in a given module. Located: -- tools/ListFiles, Debug+Asserts/lib/LLVMListFiles.so
- StaticProfiler: generate a list of expensive and frequent functions based on static analysis. Located: -- tools/StaticProfiler, Debug+Asserts/bin/staticprofiler
Most of the tools have a separate README in their location and its usages can be found using -h option.
Contact
For bug report or any other suggestion, please contact ryanhuang@cs.ucsd.edu.