Requirements-based Monitoring and Diagnosis in VLSS Evolution
Due to the scale, complexity, and heterogeneity of VLSS engineers face significant challenges when determining the compliance of systems with their requirements during development, staging, and operation. This is particularly challenging after evolving a large-scale component of an existing VLSS, a scenario that is very common in the automation software industry. The scale and heterogeneity of VLSS imply significant challenges for requirements monitoring: for instance, different technologies and architectural paradigms need to be considered. VLSS also operate in a complex environment and interact with third-party and legacy systems. Furthermore, requirements in VLSS are owned by diverse stakeholders and exist at different levels, for different systems, and are defined in different artifacts.
Requirements are often overlapping, conflicting, or cross-cutting and cannot easily be allocated to the involved systems. Still, when different systems work together, their requirements need to be aligned to ensure the correct behavior of the VLSS. VLSS also have to satisfy requirements regarding the variability of components as systems are frequently tailored to specific customer needs as well as legal and environmental conditions.
We have been developing the REMINDS, a REquirements Monitoring INfrastructure for Diagnosing Systems of Systems.
REMINDS comprises a flexible runtime monitoring infrastructure providing support for different roles and a requirements monitoring model covering the requirements to be monitored, the constraints checking adherence of a system’s behavior to its requirements, the events produced by systems at runtime, and the probes instrumenting systems to intercept events and data at runtime.
State of the Art and Practice Analysis
Case Study on Testing, Commissioning, and Operation of VLSS. We conducted an exploratory case study involving engineers and technical project managers of an industrial automation VLSS for metallurgical plants. We report empirical evidence on how VLSS are tested, commissioned, and operated in practice. The paper discusses practical challenges and reports industrial requirements regarding process and tool support.
Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “A Case Study on Testing, Commissioning, and Operation of Very-Large-Scale Software Systems“, In: Proceedings 36th Int’l Conference on Software Engineering (ICSE 2014), Hyderabad, India, pp. 125-134, 2014.
Requirements Monitoring Frameworks: Systematic Literature Review. Through a systematic literature review (SLR) we identified 330 publications, which we used to produce a comprehensive overview of the landscape of runtime monitoring approaches. An analysis of these publications regarding their support for Robinson’s requirements monitoring layers resulted in the identification of 37 frameworks. We discuss how well these frameworks support requirements monitoring in SoS. We conclude that most existing approaches are restricted to certain kinds of checks, particular types of events and data, and mostly also limited to one particular architectural style and technology. This lack of flexibility makes their application in an SoS context difficult. Also, systematic and automated variability management is still missing. Regarding evaluation, existing frameworks focus on measuring the performance overhead, while only few frameworks have been assessed in cases studies on real-world systems.
Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “Requirements Monitoring Frameworks: A Systematic Review“, In: Information and Software Technology, Volume 80, December 2016, pp. 89-109.
REMINDS Framework and the underlying Requirements Monitoring Model
A flexible monitoring infrastructure for VLSS. Our REMINDS infrastructure is split into a core — providing functionality independent of actual technologies or application areas — and extensions containing domain-specific and technology-specific implementations. Specifically, the Probing & Instrumentation Layer directly interacts with the actual systems to be monitored. Probes are used to retrieve data from diverse data sources. The Aggregation & Distribution Layer contains capabilities for collecting data from different systems, persisting data, and distributing events and data to registered clients for visualization or analyses. The Processing & Analysis Layer provides capabilities for data analysis and evaluation. Finally, the Views Layer allows developing different user interfaces, views, and visualizations for different types of users. Variability Management allows managing the variability of probes, constraints, and views wrt the variability of the systems to be monitored.
Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “Monitoring Requirements in Systems of Systems“, IEEE Software, vol. 33(5), pp. 22-24, September/October 2016.
Vierhauser, Michael, Rabiser, Rick, Grünbacher, Paul, Seyerlehner, Klaus, Wallner, Stefan, and Zeisel, Helmut: “ReMinds: A Flexible Runtime Monitoring Framework for Systems of Systems,” Journal of Systems and Software, vol. 112, pp. 123-136, 2016.
Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Danner, Christian; Wallner, Stefan; Zeisel, Helmut, “A Flexible Framework for Runtime Monitoring of System-of-Systems Architectures“, In: Proceedings 11th Working IEEE/IFIP Conference on Software Architecture (WICSA 2014), Sydney, Australia, 2014. (best paper award)
Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “A Requirements Monitoring Infrastructure for Very-Large-Scale Software Systems“, In: Proceedings 20th Int’l Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ 2014), Essen, Germany, 2014.
Vierhauser, Michael, “A Requirements Monitoring Infrastructure for Systems of Systems“, In: Doctoral Symposium at 29th IEEE/ACM Int’l Conference on Automated Software Engineering (ASE 2014), 2014.
A Requirements Monitoring Model for Systems of Systems. We developed a three-dimensional requirements monitoring model (RMM) for SoS: (i) it allows modeling the monitoring scopes of requirements with respect to the SoS architecture; (ii) it employs event models to abstract from different technologies and systems to be monitored; and (iii) it can be instantiated at runtime depending on the actual SoS configuration. To evaluate the feasibility of our RMM we created a RMM for a real-world SoS from the automation software domain. We evaluated the model by instantiating it using our monitoring infrastructure and a simulator running parts of this SoS. The results indicate that the model is sufficiently expressive to support monitoring SoS requirements of a directed SoS. It further facilitates diagnosis by discovering violations of requirements across different levels and systems in realistic monitoring scenarios.
Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Aumayr, Benedikt, “A Requirements Monitoring Model for Systems of Systems“, In: Proceedings of the 23rd International Requirements Engineering Conference (RE’15), Ottawa, Canada, IEEE 2015, pp. 96-105.
REMINDS Constraint DSL and Usefulness Study
A DSL-Based Approach for Event-Based Monitoring of Systems of Systems. Today, diverse approaches exist to define and check runtime behavior and performance characteristics. However, existing approaches often focus on specific types of systems and address certain kinds of checks, thus impeding their use in industrial SoS. Furthermore, as many SoS need to run continuously for long periods, the dynamic definition and deployment of constraints needs to be supported. We describe experiences of developing, applying and evolving a DSL-based approach for monitoring an SoS in the domain of industrial automation software. We evaluate the expressiveness of our DSL as well as the scalability of the constraint checker. We also describe lessons learned. When evolving our approach, we also conducted a systematic mapping study to systematically identify, classify and analyze existing research on DSL evolution.
Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Egyed, Alexander, “Developing a DSL-Based Approach for Event-Based Monitoring of Systems of Systems: Experiences and Lessons Learned“, In: Proceedings of the 30th IEEE/ACM Int’l Conference on Automated Software Engineering IEEE, 2015, pp. 715-725.
Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Egyed, Alexander, “A DSL-Based Approach for Event-Based Monitoring of Systems of Systems“. Software Engineering Konferenz (SE 2016), Wien, Austria, Lecture Notes in Informatics (LNI), Gesellschaft für Informatik, 2016, pp. 93-94.
Thanhofer-Pilisch, Jürgen; Lang, Alexander; Vierhauser, Michael; Rabiser, Rick, “A Systematic Mapping Study on DSL Evolution”, In: 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), IEEE, Vienna, Austria, 2017.
Assessing the Usefulness of a Requirements Monitoring Tool: A Study Involving Industrial Software Engineers. Requirements monitoring approaches support defining and checking the run-time behavior and performance characteristics of complex software systems. However, although numerous monitoring tools have been described in the literature, hardly any empirical studies exist on their usefulness for software engineering practitioners. Empirical data on usefulness, however, is important for practitioners to select and adapt capabilities of monitoring tools for their application context. In our paper we first describe common capabilities of requirements monitoring tools and then empirically assess the usefulness of these capabilities as implemented in our REMINDS monitoring tool. We report findings from an initial assessment of the tool performed using the Cognitive Dimensions of Notations Framework. We then present results of a usefulness study involving software engineers of a large company from the domain of automation software. Finally, we discuss implications for developers of requirements monitoring tools.
Rabiser, Rick; Vierhauser, Michael; Grünbacher, Paul, “Assessing the Usefulness of a Requirements Monitoring Tool: A Study Involving Industrial Software Engineers“, In: Proceedings 38th Int’l Conference on Software Engineering (ICSE 2016), ICSE Companion, Austin, USA, 2016, ACM, pp. 122-131.
Support for Evolution and Variability
Event Capture and Compare for Runtime Monitoring of Systems of Systems. Today’s complex software systems are highly variable systems of systems (SoS) that evolve continuously to meet new customer requirements or to address technological changes. Despite thorough testing of the different parts of an SoS, the full behavior often emerges during operation only. The systems and their interactions thus need to be continuously monitored and checked during operation to determine compliance with requirements. In particular, after changes to one system, it is necessary to check whether the overall SoS still behaves correctly and as intended. Based on an existing monitoring framework we are thus developing support for capturing and comparing event traces in SoS. Our tool-supported approach facilitates, and partly automates, identifying differences in event traces, which often indicate undesirable behavior introduced during evolution.
Thanhofer-Pilisch, Jürgen; Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul, “Event Capture and Compare for Runtime Monitoring of Systems of Systems“, In: Proceedings 1st Int’l Workshop on Variability and Complexity in Software Design (VACE 2016), in conjunction with ICSE 2016, Austin, TX, USA, 2016, ACM, pp. 1-4.
Thanhofer-Pilisch, Jürgen; Rabiser, Rick; Krismayer, Thomas; Vierhauser, Michael; Grünbacher, Paul; Wallner, Stefan; Seyerlehner, Klaus; Zeisel, Helmut, “An Event-based Capture-and-Compare Approach to Support the Evolution of Systems of Systems“, In: Proceedings 11th ACM International Conference on Distributed and Event-Based Systems (DEBS 2017), Barcelona, Spain, ACM, 2017, pp. 261-270.
Variability Management for a Runtime Monitoring Infrastructure. We use a decision-oriented variability management approach to support the reconfiguration of a monitoring infrastructure at runtime. Our approach provides configuration support for setting up the MI to reflect system variability. It also supports runtime reconfiguration of the MI to reflect the different monitoring tasks of users and to support evolution.
Rabiser, Rick; Vierhauser, Michael; Grünbacher, Paul, “Variability Management for a Runtime Monitoring Infrastructure“, In: Proceedings 9th International Workshop on Variability Modelling of Software-intensive Systems (VaMoS 2015), Hildesheim, Germany, ACM, 2015, pp. 35-42.
Quinton, Clément; Rabiser, Rick; Vierhauser, Michael; Grünbacher, Paul; Baresi, Luciano, “Evolution in Dynamic Software Product Lines: Challenges and Perspectives“, In: Proceedings 19th International Software Product Line Conference (SPLC 2015), Nashville, USA, ACM, 2015, pp. 126-130.
Automated requirements monitoring model creation and maintenance
Mining Constraints for Event-based Monitoring in Systems of Systems. Independent of the monitoring approach, deep domain knowledge is required to specify the desired behavior. This knowledge is often not available in SoS environments with multiple development teams independently working on different, heterogeneous systems. We thus developed an approach that automatically mines constraints for runtime monitoring from event logs recorded in SoS. Our approach combines specification mining and machine learning techniques to mine different types of constraints on event occurrence, event timing, and event data. The approach further presents the mined constraints to users in an existing constraint language and it ranks the constraints using different criteria. We demonstrated the feasibility of our approach by applying it to event logs from a real-world industrial SoS.
Krismayer, Thomas; Rabiser, Rick; Grünbacher, Paul, “Mining Constraints for Event-based Monitoring in Systems of Systems”, In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, 2017 (accepted for publication).
Extended support for diagnosing and fixing violations
From Requirements Monitoring to Diagnosis Support in System of Systems. The focus of existing monitoring approaches is mainly on detecting violations of expected behavior, while support for subsequent diagnosis of violations is rather limited and often even neglected. Diagnosis is particularly challenging in systems of systems (SoS), which are characterized by complex heterogeneous architectures and a slew of different development and testing tools. In particular, it is difficult for developers to uncover the root cause of a requirements violation and to track it back to source code or requirements documents without detailed domain knowledge. We work on diagnosis support in SoS, i.e., a tool-supported monitoring framework supporting diagnosis in SoS.
Vierhauser, Michael; Rabiser, Rick; Cleland-Huang, Jane, “From Requirements Monitoring to Diagnosis Support in System of Systems“, In: 23rd International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ 2017), Essen, Germany, Springer, 2017, pp. 181-187.
A Comparison Framework for Runtime Monitoring Approaches. The full behavior of complex software systems often only emerges during operation. They thus need to be monitored at run time to check that they adhere to their requirements. Diverse runtime monitoring approaches have been developed in various domains and for different purposes. Their sheer number and heterogeneity, however, make it hard to find the right approach for a specific application or purpose. The aim of our research therefore was to develop a comparison framework for runtime monitoring approaches. We present such a framework we developed based on an analysis of the literature and existing taxonomies for monitoring languages and patterns. We use examples from existing monitoring approaches to explain the framework. We demonstrate its usefulness by describing how we applied it to 32 existing approaches and by discussing a detailed comparison of 3 selected approaches in the light of different monitoring scenarios. We also discuss perspectives for researchers.
Rabiser, Rick; Guinea, Sam; Vierhauser, Michael; Baresi, Luciano; Grünbacher, Paul, “A Comparison Framework for Runtime Monitoring Approaches“, In: Journal of Systems and Software, vol. 125, March 2017, pp. 309-321 (DOI: 10.1016/j.jss.2016.12.034).
Please refer to our tool page at http://mevss.jku.at/reminds for details about the tools we have been developing to support monitoring of systems of systems. We also have published a tool demo paper:
Vierhauser, Michael; Rabiser, Rick; Grünbacher, Paul; Thanhofer-Pilisch, Jürgen, “The REMINDS Tool Suite for Runtime Monitoring of Systems of Systems“, In: Proceedings of the 30th IEEE/ACM Int’l Conference on Automated Software Engineering, IEEE, 2015, pp. 777-781.
Current Team Members
|Rick Rabiser||Senior Researcher/