RTE

Robust Testing Environments

Currently much time is spent in setting up system tests, trying to understand exactly what is being tested, understanding errors that occur during tests and verifying the test results. These issues can be especially difficult for distributed enterprise applications made up of a large number of software components that run on a collection of heterogeneous servers or cloud environments. Load generation for accurate simulation of complex user behaviour is a growing challenge when testing large-scale critical systems. For example, with more and more applications moving to cloud-based infrastructures, typical load generation designed for applications residing on a local area network may be ineffective for robust testing.

Research is required to develop new load generation techniques that take into consideration new cloud-based infrastructures where applications may be physically distributed across different infrastructures in different geographical locations and where network bandwidth may be unknown or at different quality levels at various points. This research will develop a set of prototype tools to allow for more efficient systems testing and of large enterprise applications.

What is the question that this WP addresses?

Identifying critical system production level issues requires robust testing environments with appropriate loads. This research proposes to develop these expert tools, techniques and algorithms to allow for large scale testing environments to be easily created. Expert knowledge will be applied to develop these tools with the ultimate aim of problem determination in critical systems.

The objective is to develop tools, techniques and algorithms suitable for accurate, robust and easily understandable test generation suitable for critical systems such that system problems and bugs likely to arise in production can be more easily identified during testing. The aim is to develop a test creation framework that allows for the production of robust load for accurate testing of critical systems, as well as robust data sets for testing and for visualisation of the load produced such that communication of test plans can be more easily understood.

Why is this question significant?

Problem determination in large scale critical systems is challenging as there are large numbers of components physically distributed and multiple layers of abstraction (OS, VMs, Frameworks). The current monitoring and analysis tools are limited and expert knowledge is required to understand the outputs. A major issue is that replicating robust production level environments is difficult and is currently largely manual and time consuming.

How will the question be addressed?

Some aspects of this research include using advanced analysis techniques (e.g. data mining) to extract relevant information from large volumes of data, advanced monitoring techniques (e.g. run-time correlation of events from different sources) that can be applied during testing, automatic diagram creation and diagram to script transformations, as well as the categorisation and collection of real issues that occur throughout the testing process.

Bayesian graphical models provide a flexible and principled framework for processing uncertainties in large networks of interacting nodes such as arise in this context. A major difficulty is the enormous number of loading scenarios (modes) that need to be modelled, and the paucity (or, indeed, absence) of data to model these modes in the Bayesian graphical model. This presents a major challenge in terms of test generation that is robust to the probability models chosen for the system interactions. Bayesian non-parametrics relax the requirement for specific parametric model choices, allowing flexible functional classes to be assessed. They provide a robust and reliable framework for modelling in these data-poor large-scale critical systems.

Following on from the modelling challenge, the generation of datasets (simulations) that allow for robust testing of search algorithms, database queries and application transactions can be equally challenging. Markov chain Monte Carlo methods are available for simulation from the high-dimensional and nonparametric system models proposed above. They are expected to permit representative test datasets to be generated for specific large-scale critical systems in a numerically efficient way.

Understanding test scripts such that the purpose and effect of tests can be easily communicated at various levels of abstraction is a further important issue addressed here. Test scripts are today generally recorded using test automation frameworks and represented in vendor-specific scripting languages. Modifications to such languages can be challenging and understanding recorded tests can be difficult especially for non-experts. The aim is to develop an approach to visualise tests scripts such that they can be represented at various levels of abstraction for different potential end users (e.g. management, technical experts etc.).

The work programme will create a test development framework for critical systems for problem determination purposes. The research will focus on (a) load generation for critical systems (b) data set creation for testing of complex systems and (c) development of visualisation of tests such that a better understanding of system testing actions can be more easily achieved. The test generation will make use of expert knowledge to generate robust and production-level test environments.