Module 1 | Christian Doppler Laboratory Monitoring and Evolution of Very-Large-Scale Software Systems

Requirements-based Monitoring and Diagnosis in VLSS Evolution

Motivation

Due to the scale, complexity, and heterogeneity of VLSS engineers face significant challenges when determining the compliance of systems with their requirements during development, staging, and operation. This is particularly challenging after evolving a large-scale component of an existing VLSS, a scenario that is very common in the automation software industry. The scale and heterogeneity of VLSS imply significant challenges for requirements monitoring: for instance, different technologies and architectural paradigms need to be considered. VLSS also operate in a complex environment and interact with third-party and legacy systems. Furthermore, requirements in VLSS are owned by diverse stakeholders and exist at different levels, for different systems, and are defined in different artifacts.

Requirements are often overlapping, conflicting, or cross-cutting and cannot easily be allocated to the involved systems. Still, when different systems work together, their requirements need to be aligned to ensure the correct behavior of the VLSS. VLSS also have to satisfy requirements regarding the variability of components as systems are frequently tailored to specific customer needs as well as legal and environmental conditions.

Approach

We have been developing the REMINDS, a REquirements Monitoring INfrastructure for Diagnosing Systems of Systems.

REMINDS comprises a flexible runtime monitoring infrastructure providing support for different roles and a requirements monitoring model covering the requirements to be monitored, the constraints checking adherence of a system’s behavior to its requirements, the events produced by systems at runtime, and the probes instrumenting systems to intercept events and data at runtime.

Results

State of the Art and Practice Analysis

Case Study on Testing, Commissioning, and Operation of VLSS. We conducted an exploratory case study involving engineers and technical project managers of an industrial automation VLSS for metallurgical plants. We report empirical evidence on how VLSS are tested, commissioned, and operated in practice. The paper discusses practical challenges and reports industrial requirements regarding process and tool support.

Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “A Case Study on Testing, Commissioning, and Operation of Very-Large-Scale Software Systems“, In: Proceedings 36th Int’l Conference on Software Engineering (ICSE 2014), Hyderabad, India, pp. 125-134, 2014.

Requirements Monitoring Frameworks: Systematic Literature Review. Through a systematic literature review (SLR) we identified 330 publications, which we used to produce a comprehensive overview of the landscape of runtime monitoring approaches. An analysis of these publications regarding their support for Robinson’s requirements monitoring layers resulted in the identification of 37 frameworks. We discuss how well these frameworks support requirements monitoring in SoS. We conclude that most existing approaches are restricted to certain kinds of checks, particular types of events and data, and mostly also limited to one particular architectural style and technology. This lack of flexibility makes their application in an SoS context difficult. Also, systematic and automated variability management is still missing. Regarding evaluation, existing frameworks focus on measuring the performance overhead, while only few frameworks have been assessed in cases studies on real-world systems.

Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “Requirements Monitoring Frameworks: A Systematic Review“, In: Information and Software Technology, Volume 80, December 2016, pp. 89-109.

REMINDS Framework and the underlying Requirements Monitoring Model

A flexible monitoring infrastructure for VLSS. Our REMINDS infrastructure is split into a core — providing functionality independent of actual technologies or application areas — and extensions containing domain-specific and technology-specific implementations. Specifically, the Probing & Instrumentation Layer directly interacts with the actual systems to be monitored. Probes are used to retrieve data from diverse data sources. The Aggregation & Distribution Layer contains capabilities for collecting data from different systems, persisting data, and distributing events and data to registered clients for visualization or analyses. The Processing & Analysis Layer provides capabilities for data analysis and evaluation. Finally, the Views Layer allows developing different user interfaces, views, and visualizations for different types of users. Variability Management allows managing the variability of probes, constraints, and views wrt the variability of the systems to be monitored.

Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “Monitoring Requirements in Systems of Systems“, IEEE Software, vol. 33(5), pp. 22-24, September/October 2016.

Vierhauser, Michael, Rabiser, Rick, Grünbacher, Paul, Seyerlehner, Klaus, Wallner, Stefan, and Zeisel, Helmut: “ReMinds: A Flexible Runtime Monitoring Framework for Systems of Systems,” Journal of Systems and Software, vol. 112, pp. 123-136, 2016.

Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Danner, Christian; Wallner, Stefan; Zeisel, Helmut, “A Flexible Framework for Runtime Monitoring of System-of-Systems Architectures“, In: Proceedings 11th Working IEEE/IFIP Conference on Software Architecture (WICSA 2014), Sydney, Australia, 2014. (best paper award)

Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “A Requirements Monitoring Infrastructure for Very-Large-Scale Software Systems“, In: Proceedings 20th Int’l Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ 2014), Essen, Germany, 2014.

Vierhauser, Michael, “A Requirements Monitoring Infrastructure for Systems of Systems“, In: Doctoral Symposium at 29th IEEE/ACM Int’l Conference on Automated Software Engineering (ASE 2014), 2014.

A Requirements Monitoring Model for Systems of Systems. We developed a three-dimensional requirements monitoring model (RMM) for SoS: (i) it allows modeling the monitoring scopes of requirements with respect to the SoS architecture; (ii) it employs event models to abstract from different technologies and systems to be monitored; and (iii) it can be instantiated at runtime depending on the actual SoS configuration. To evaluate the feasibility of our RMM we created a RMM for a real-world SoS from the automation software domain. We evaluated the model by instantiating it using our monitoring infrastructure and a simulator running parts of this SoS. The results indicate that the model is sufficiently expressive to support monitoring SoS requirements of a directed SoS. It further facilitates diagnosis by discovering violations of requirements across different levels and systems in realistic monitoring scenarios.

Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Aumayr, Benedikt, “A Requirements Monitoring Model for Systems of Systems“, In: Proceedings of the 23rd International Requirements Engineering Conference (RE’15), Ottawa, Canada, IEEE 2015, pp. 96-105.

REMINDS Constraint DSL and Usefulness Study

A DSL-Based Approach for Event-Based Monitoring of Systems of Systems. Today, diverse approaches exist to define and check runtime behavior and performance characteristics. However, existing approaches often focus on specific types of systems and address certain kinds of checks, thus impeding their use in industrial SoS. Furthermore, as many SoS need to run continuously for long periods, the dynamic definition and deployment of constraints needs to be supported. We describe experiences of developing, applying and evolving a DSL-based approach for monitoring an SoS in the domain of industrial automation software. We evaluate the expressiveness of our DSL as well as the scalability of the constraint checker. We also describe lessons learned. When evolving our approach, we also conducted a systematic mapping study to systematically identify, classify and analyze existing research on DSL evolution.

Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Egyed, Alexander, “Developing a DSL-Based Approach for Event-Based Monitoring of Systems of Systems: Experiences and Lessons Learned“, In: Proceedings of the 30th IEEE/ACM Int’l Conference on Automated Software Engineering IEEE, 2015, pp. 715-725.

Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Egyed, Alexander, “A DSL-Based Approach for Event-Based Monitoring of Systems of Systems“. Software Engineering Konferenz (SE 2016), Wien, Austria, Lecture Notes in Informatics (LNI), Gesellschaft für Informatik, 2016, pp. 93-94.

Thanhofer-Pilisch, Jürgen; Lang, Alexander; Vierhauser, Michael; Rabiser, Rick, “A Systematic Mapping Study on DSL Evolution“, In: 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), IEEE, 2017, pp. 149-156.

Rabiser, Rick; Thanhofer-Pilisch, Jürgen; Vierhauser, Michael; Grünbacher, Paul; Egyed, Alexander, “Developing and Evolving a DSL-Based Approach for Runtime Monitoring of Systems of Systems“, In: Automated Software Engineering, 2018.

Assessing the Usefulness of a Requirements Monitoring Tool: A Study Involving Industrial Software Engineers. Requirements monitoring approaches support defining and checking the run-time behavior and performance characteristics of complex software systems. However, although numerous monitoring tools have been described in the literature, hardly any empirical studies exist on their usefulness for software engineering practitioners. Empirical data on usefulness, however, is important for practitioners to select and adapt capabilities of monitoring tools for their application context. In our paper we first describe common capabilities of requirements monitoring tools and then empirically assess the usefulness of these capabilities as implemented in our REMINDS monitoring tool. We report findings from an initial assessment of the tool performed using the Cognitive Dimensions of Notations Framework. We then present results of a usefulness study involving software engineers of a large company from the domain of automation software. Finally, we discuss implications for developers of requirements monitoring tools.

Rabiser, Rick; Vierhauser, Michael; Grünbacher, Paul, “Assessing the Usefulness of a Requirements Monitoring Tool: A Study Involving Industrial Software Engineers“, In: Proceedings 38th Int’l Conference on Software Engineering (ICSE 2016), ICSE Companion, Austin, USA, 2016, ACM, pp. 122-131.

Kritzinger, Lisa Maria; Krismayer, Thomas; Rabiser, Rick; Grünbacher, Paul, “A User Study on the Usefulness of Visualization Support for Requirements Monitoring“, In: 7th IEEE Working Conference on Software Visualization, IEEE, Cleveland, Ohio, USA, 2019.

Support for Evolution and Variability

Event Capture and Compare for Runtime Monitoring of Systems of Systems. Today’s complex software systems are highly variable systems of systems (SoS) that evolve continuously to meet new customer requirements or to address technological changes. Despite thorough testing of the different parts of an SoS, the full behavior often emerges during operation only. The systems and their interactions thus need to be continuously monitored and checked during operation to determine compliance with requirements. In particular, after changes to one system, it is necessary to check whether the overall SoS still behaves correctly and as intended. Based on an existing monitoring framework we are thus developing support for capturing and comparing event traces in SoS. Our tool-supported approach facilitates, and partly automates, identifying differences in event traces, which often indicate undesirable behavior introduced during evolution.

Thanhofer-Pilisch, Jürgen; Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “Event Capture and Compare for Runtime Monitoring of Systems of Systems“, In: Proceedings 1st Int’l Workshop on Variability and Complexity in Software Design (VACE 2016), in conjunction with ICSE 2016, Austin, TX, USA, 2016, ACM, pp. 1-4.

Thanhofer-Pilisch, Jürgen; Rabiser, Rick; Krismayer, Thomas; Vierhauser, Michael; Grünbacher, Paul; Wallner, Stefan; Seyerlehner, Klaus; Zeisel, Helmut, “An Event-based Capture-and-Compare Approach to Support the Evolution of Systems of Systems“, In: Proceedings 11th ACM International Conference on Distributed and Event-Based Systems (DEBS 2017), Barcelona, Spain, ACM, 2017, pp. 261-270.

Krismayer, Thomas; Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “Comparing Constraints Mined From Execution Logs to Understand Software Evolution“, In: Proceedings of the 35th IEEE International Conference on Software Maintenance and Evolution (ICSME), Cleveland, Ohio, USA, 2019.

Variability Management for a Runtime Monitoring Infrastructure. We use a decision-oriented variability management approach to support the reconfiguration of a monitoring infrastructure at runtime. Our approach provides configuration support for setting up the MI to reflect system variability. It also supports runtime reconfiguration of the MI to reflect the different monitoring tasks of users and to support evolution.

Rabiser, Rick; Vierhauser, Michael; Grünbacher, Paul, “Variability Management for a Runtime Monitoring Infrastructure“, In: Proceedings 9th International Workshop on Variability Modelling of Software-intensive Systems (VaMoS 2015), Hildesheim, Germany, ACM, 2015, pp. 35-42.

Quinton, Clément; Rabiser, Rick; Vierhauser, Michael; Grünbacher, Paul; Baresi, Luciano, “Evolution in Dynamic Software Product Lines: Challenges and Perspectives“, In: Proceedings 19th International Software Product Line Conference (SPLC 2015), Nashville, USA, ACM, 2015, pp. 126-130.

Automated requirements monitoring model creation and maintenance

Mining Constraints for Event-based Monitoring in Systems of Systems. Independent of the monitoring approach, deep domain knowledge is required to specify the desired behavior. This knowledge is often not available in SoS environments with multiple development teams independently working on different, heterogeneous systems. We thus developed an approach that automatically mines constraints for runtime monitoring from event logs recorded in SoS. Our approach combines specification mining and machine learning techniques to mine different types of constraints on event occurrence, event timing, and event data. The approach further presents the mined constraints to users in an existing constraint language and it ranks the constraints using different criteria. We demonstrated the feasibility of our approach by applying it to event logs from a real-world industrial SoS.

Krismayer, Thomas; Rabiser, Rick; Grünbacher, Paul, “Mining Constraints for Event-based Monitoring in Systems of Systems“, In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, IEEE, 2017, pp. 826-831.

Krismayer, Thomas, “Automatic Mining of Constraints for Monitoring Systems of Systems“, In: Doctoral Symposium at 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE 2018), Montpellier, France, pp. 924-927, 2018.

Krismayer, Thomas; Rabiser, Rick; Grünbacher, Paul, “Mining Constraints for Monitoring Systems of Systems“. Proceedings of the 34th ACM/SIGAPP Symposium On Applied Computing (SAC 2019), Limassol, Cyprus, 2019.

Krismayer, Thomas; Kronberger, Peter; Rabiser, Rick; Grünbacher, Paul, “Supporting the Selection of Constraints for Requirements Monitoring from Automatically Mined Constraint Candidates“, In: Proceedings of the 25th International Working Conference on Requirements Engineering: Foundation for Software Quality, Essen, Germany, 2019.

Krismayer, Thomas; Rabiser, Rick; Grünbacher, Paul, “A Constraint Mining Approach to Support Monitoring Cyber-Physical Systems“, In: Proceedings of the 31st International Conference on Advanced Information Systems Engineering (CAiSE) (Giorgini, Paolo; Weber, Barbara, eds.), Springer International Publishing, vol. 11483, Rome, Italy, pp. 659-674, 2019.

Extended support for diagnosing and fixing violations

From Requirements Monitoring to Diagnosis Support in System of Systems. The focus of existing monitoring approaches is mainly on detecting violations of expected behavior, while support for subsequent diagnosis of violations is rather limited and often even neglected. Diagnosis is particularly challenging in systems of systems (SoS), which are characterized by complex heterogeneous architectures and a slew of different development and testing tools. In particular, it is difficult for developers to uncover the root cause of a requirements violation and to track it back to source code or requirements documents without detailed domain knowledge. We work on diagnosis support in SoS, i.e., a tool-supported monitoring framework supporting diagnosis in SoS.

Vierhauser, Michael; Rabiser, Rick; Cleland-Huang, Jane, “From Requirements Monitoring to Diagnosis Support in System of Systems“, In: 23rd International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ 2017), Essen, Germany, Springer, 2017, pp. 181-187.

Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Cleland-Huang, Jane, “Establishing Trace-Links for Runtime Diagnosis Support in System of Systems“. Grand Challenges of Traceability: The Next Ten Years, arXiv:1710.03129 [cs.SE], 2017, pp. 50-51.

Vierhauser, Michael; Cleland-Huang, Jane; Rabiser, Rick; Krismayer, Thomas; Grünbacher, Paul, “Supporting Diagnosis of Requirements Violations in Systems of Systems“. In: 26th IEEE International Requirements Engineering Conference (RE’18), Banff, Canada, IEEE, 2018.

Visualization support

Visualization Support for Requirements Monitoring in Systems of Systems. Industrial software systems are often systems of systems (SoS) whose full behavior only emerges at runtime. The systems and their interactions thus need to be continuously monitored and checked during operation to determine compliance with requirements. Many requirements monitoring approaches have been proposed. However, only few of these come with tools that present and visualize monitoring results and details on requirements violations to end users such as industrial engineers. We have been developing visualization capabilities motivated by industrial scenarios. We complement REMINDS, an existing requirements monitoring framework, which supports collecting, aggregating, and analyzing events and event data in architecturally heterogeneous SoS, with visualizations supporting a drill-down scenario for monitoring and diagnosis: starting from a graphical status overview of the monitored systems and their relations engineers can view trends and statistics about performed analyses and diagnose the root cause of problems by inspecting the events and event data that led to a specific violation. Initial industry feedback we received confirms the usefulness of our tool support.

Kritzinger, Lisa; Krismayer, Thomas; Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “Visualization Support for Requirements Monitoring in Systems of Systems“, In: Proceedings of the 32th IEEE/ACM International Conference on Automated Software Engineering, IEEE, 2017, pp. 889-894.

Applications of REMINDS

Monitoring CPS at Runtime – A Case Study in the UAV Domain. Unmanned aerial vehicles (UAVs) are becoming increasingly pervasive in everyday life, supporting diverse use cases such as aerial photography, delivery of goods, or disaster reconnaissance and management. UAVs are cyber-physical systems (CPS): they integrate computation (embedded software and control systems) with physical components (the UAVs flying in the physical world). UAVs in particular, and CPS in general require monitoring capabilities which allow them to detect and possibly mitigate erroneous and safety-critical behavior at runtime. Existing monitoring approaches mostly do not adequately address UAV CPS characteristics such as the high number of dynamically instantiated components, the tight integration of hardware and software components, and massive amounts of data to be processed. We report results of a case study in which we extended the REMINDS approach for runtime monitoring, which has so far been used for software monitoring in metallurgical plants, to support monitoring of UAVs. We discuss CPS-specific challenges and present a prototype implementation. Additionally, we demonstrate the applicability of our approach and its scalability by monitoring a real control and management system for UAVs in simulations with up to 30 drones flying in an urban area..

Vierhauser, Michael; Cleland-Huang, Jane; Bayley, Sean; Krismayer, Thomas; Rabiser, Rick; Grünbacher, Paul, “Monitoring CPS at Runtime – A Case Study in the UAV Domain.” In: Proceedings of the 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), IEEE, 2018.

Community Initiatives

A Comparison Framework for Runtime Monitoring Approaches. The full behavior of complex software systems often only emerges during operation. They thus need to be monitored at run time to check that they adhere to their requirements. Diverse runtime monitoring approaches have been developed in various domains and for different purposes. Their sheer number and heterogeneity, however, make it hard to find the right approach for a specific application or purpose. The aim of our research therefore was to develop a comparison framework for runtime monitoring approaches. We present such a framework we developed based on an analysis of the literature and existing taxonomies for monitoring languages and patterns. We use examples from existing monitoring approaches to explain the framework. We demonstrate its usefulness by describing how we applied it to 32 existing approaches and by discussing a detailed comparison of 3 selected approaches in the light of different monitoring scenarios. We also discuss perspectives for researchers.

Rabiser, Rick; Guinea, Sam; Vierhauser, Michael; Baresi, Luciano; Grünbacher, Paul, “A Comparison Framework for Runtime Monitoring Approaches“, In: Journal of Systems and Software, vol. 125, March 2017, pp. 309-321 (DOI: 10.1016/j.jss.2016.12.034).

Rabiser, Rick; Guinea, Sam; Vierhauser, Michael; Baresi, Luciano; Grünbacher, Paul, “Extended Journal First Abstract: A Comparison Framework for Runtime Monitoring Approaches.” Proc. of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2018), Campobasso, Italy, IEEE, 2018.

A domain analysis of resource and requirements monitoring: Towards a comprehensive model of the software monitoring domain. Complex and heterogeneous software systems need to be monitored as their full behavior often only emerges at runtime, e.g., when interacting with other systems or the environment. Software monitoring approaches observe and check properties or quality attributes of software systems during operation. Such approaches have been developed in diverse communities for various kinds of systems and purposes. For instance, requirements monitoring aims to check at runtime whether a software system adheres to its requirements, while resource or performance monitoring collects information about the consumption of computing resources by the monitored system. Many venues publish research on software monitoring, often using diverse terminology, and focusing on different monitoring aspects and phases. The lack of a comprehensive overview of existing research often leads to re-inventing the wheel. We provide a domain model to structure and systematize the field of software monitoring, starting with requirements and resource monitoring. We developed an initial domain model based on (i) our extensive experiences with requirements and resource monitoring, (ii) earlier efforts to develop a comparison framework for monitoring approaches, and (iii) an earlier systematic literature review on requirements monitoring frameworks. We then systematically analyzed 47 existing requirements and resource monitoring approaches to iteratively refine the domain model and to develop a reference architecture for software monitoring approaches. Our domain model covers the key elements of monitoring approaches and allows analyzing their commonalities and differences. Together with the reference architecture, our domain model supports the development of integrated monitoring solutions. We provide details on 47 approaches we analyzed with the model to assess its coverage. We also evaluate the reference architecture by instantiating it for five different monitoring solutions. We conclude that requirements and resource monitoring have more commonalities than differences, which is promising for the future integration of existing monitoring solutions.

Rabiser, Rick; Schmid, Klaus; Eichelberger, Holger; Vierhauser, Michael; Guinea, Sam; Grünbacher, Paul, “A Domain Analysis of Resource and Requirements Monitoring: Towards a Comprehensive Model of the Software Monitoring Domain“, In: Information and Software Technology, vol. 111, pp. 86-109, 2019.

Tools

Please refer to our tool page at http://mevss.jku.at/reminds for details about the tools we have been developing to support monitoring of systems of systems. We also have published two tool demo papers:

Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Thanhofer-Pilisch, Jürgen, “The REMINDS Tool Suite for Runtime Monitoring of Systems of Systems“, In: Proceedings of the 30th IEEE/ACM Int’l Conference on Automated Software Engineering, IEEE, 2015, pp. 777-781.

Kritzinger, Lisa Maria; Krismayer, Thomas; Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “Visualization Support for Requirements Monitoring in Systems of Systems“, In: Proceedings of the 32th IEEE/ACM International Conference on Automated Software Engineering, IEEE; 2017, pp. 1017-1022.

Krismayer, Thomas; Kronberger, Peter; Rabiser, Rick; Grünbacher, Paul, “ReMinds-CMT: An Interactive Tool Supporting Constraint Mining for Requirements Monitoring“, In: Joint Proceedings of REFSQ-2019 Workshops, Doctoral Symposium, Live Studies Track, and Poster Track co-located with the 25th International Conference on Requirements Engineering: Foundation for Software Quality, Essen, Germany, pp. 1-5, 2019.

Team

Team Members

	Thomas Krismayer	Researcher
	Lisa Maria Kritzinger	Student Researcher
	Rick Rabiser	Senior Researcher
	Dario Romano	Student Researcher
	Jürgen Thanhofer-Pilisch	Student Researcher
	Michael Vierhauser	Researcher
	Paul Grünbacher	Head of Laboratory

Cookie	Duration	Description
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.