Blogs (1) >>
ASE 2019
Sun 10 - Fri 15 November 2019 San Diego, California, United States
Wed 13 Nov 2019 16:40 - 17:00 at Hillcrest - Performance Chair(s): Tim Menzies

Designing field-representative load tests is an essential step for the quality assurance of large-scale systems. Practitioners may capture user behaviour at different levels of granularity. A coarse-grained load test may miss detailed user behaviour, leading to a non-representative load test; while an extremely fine-grained load test would simply replay user actions step by step, leading to load tests that are costly to develop, execute and maintain. Workload recovery is core of these load tests. Prior research often captures the workload as the frequency of user actions. However, there exists much valuable information in the context and sequences of user actions. Such richer information would ensure that the load tests that leverage such workloads are more field-representative. In this experience paper, we study the use of different granularities of user behaviour, i.e., basic user actions, basic user actions with contextual information and user action sequences with contextual information, when recovering workloads for use in the load testing of large-scale systems. We propose three approaches that are based on the three granularities of user behaviour and evaluate our approaches on four subject systems, namely Apache James, OpenMRS, Google Borg, and an ultra-large-scale industrial system (SA) from CompanyA. Our results show that our approach that is based on user action sequences with contextual information outperforms the other two approaches and can generate more representative load tests with similar throughput and CPU usage to the original field workload (i.e., mostly statistically insignificant or with small/trivial effect sizes). Such representative load tests are generated only based on a small number of clusters of users, leading to a low cost of conducting/maintaining such tests. Finally, we demonstrate that our approaches can detect injected users in the original field workloads with high precision and recall. Our paper demonstrates the importance of user action sequences with contextual information in the workload recovery of large-scale systems.

Wed 13 Nov

Displayed time zone: Tijuana, Baja California change

16:00 - 17:50
PerformanceResearch Papers / Demonstrations at Hillcrest
Chair(s): Tim Menzies North Carolina State University
16:00
20m
Talk
Accurate Modeling of Performance Histories for Evolving Software Systems
Research Papers
Stefan Mühlbauer Bauhaus-University Weimar, Sven Apel Saarland University, Norbert Siegmund Bauhaus-University Weimar
Pre-print
16:20
20m
Talk
An Industrial Experience Report on Performance-Aware Refactoring on a Database-centric Web Application
Research Papers
Boyuan Chen York University, Zhen Ming (Jack) Jiang York University, Paul Matos Copywell Inc., Michael Lacaria Copywell Inc.
Authorizer link Pre-print
16:40
20m
Talk
An Experience Report of Generating Load Tests Using Log-recovered Workloads at Varying Granularities of User Behaviour
Research Papers
Jinfu Chen Jiangsu University, Weiyi Shang Concordia University, Canada, Ahmed E. Hassan Queen's University, Yong Wang Alibaba Group, Jiangbin Lin Alibaba Group
Pre-print
17:00
10m
Talk
How Do API Selections Affect the Runtime Performance of Data Analytics Tasks?
Research Papers
Yida Tao Shenzhen University, Shan Tang Shenzhen University, Yepang Liu Southern University of Science and Technology, Zhiwu Xu Shenzhen University, Shengchao Qin University of Teesside
17:10
10m
Talk
Demystifying Application Performance Management Libraries for Android
Research Papers
Yutian Tang The Hong Kong Polytechnic University, Xian Zhan The Hong Kong Polytechnic University, Hao Zhou The Hong Kong Polytechnic University, Xiapu Luo The Hong Kong Polytechnic University, Zhou Xu Wuhan University, Yajin Zhou Zhejiang University, Qiben Yan Michigan State University
17:20
10m
Demonstration
PeASS: A Tool for Identifying Performance Changes at Code Level
Demonstrations
David Georg Reichelt Universität Leipzig, Stefan Kühne Universität Leipzig, Wilhelm Hasselbring Kiel University
Pre-print Media Attached File Attached
17:30
20m
Talk
ReduKtor: How We Stopped Worrying About Bugs in Kotlin Compiler
Research Papers
Daniil Stepanov Saint Petersburg Polytechnic University, Marat Akhin Saint Petersburg Polytechnic University / JetBrains Research, Mikhail Belyaev Saint Petersburg Polytechnic University
Pre-print