Performance Measurement: Indicators and Metrics

We conducted a structured literature review (SLR) to find papers dealing with performance measurement in the business process literature. SLR can be defined as "a means of evaluating and interpreting all available research relevant to a particular research question, topic area, or phenomenon of interest". An SLR is a meta study that identifies and summarizes evidence from earlier research or a way to address a potentially large number of identified sources based on a strict protocol used to search and appraise the literature. It is systematic in the sense of a systematic approach to finding relevant papers and a systematic way of classifying the papers. Hence, according to Boellt and Cecez-Kecmanovic, SLR as a specific type of literature review can only be used when two conditions are met. First, the topic should be well-specified and closely formulated (i.e., limited to performance measurement in the context of business processes) to potentially identify all relevant literature based on inclusion and exclusion criteria. Secondly, the research questions should be answered by extracting and aggregating evidence from the identified literature based on a high-level summary or bibliometric-type of content analysis. Furthermore, King and He also refer to a statistical analysis of existing literature.

Informed by the established guidelines proposed by Kitchenham, we undertook the review in distinct stages: (1) formulating the research questions and the search strategy, (2) filtering and extracting data based on inclusion and exclusion criteria, and (3) synthesizing the findings. The remainder of this section describes the details of each stage.

Formulating the research questions and search strategy

A comprehensive and unbiased search is one of the fundamental factors that distinguish a systematic review from a traditional literature review. For this purpose, a systematic search begins with the identification of keywords and search terms that are derived from the research questions. Based on the research questions stipulated in the introduction, the SLR protocol for our study was defined, as shown in Table 2.

Table 2 The structured literature review protocol for this study, based on Boellt and Cecez-Kecmanovic

Table 3 shows the degree to which the initial sample sizes varied, with 433 resulting papers for the most permissive search query (TOxTO) and 19 papers for the most restrictive one (TIxTI). The next stage started with the most permissive search query in an effort to select and assess as many relevant publications as possible.

Table 3 The number of papers in the web of science per search query (until November 2015)

Filtering and extracting data

Figure 3 summarizes the procedure for searching and selecting the literature to be reviewed. The list of papers found in the previous stage was filtered by deleting 35 duplicates, and the remaining 398 papers were further narrowed to 153 papers by evaluating their title and abstract. After screening the body of the texts, 76 full-text papers were considered relevant for our scope and constituted the final sample.

Fig. 3 Exclusion of papers and number of primary studies

More specifically, studies were excluded if their main focus was not business process performance measurement or if they did not refer to indicators, measures or metrics for business performance. The inclusion of studies was not restricted to any specific type of intervention or outcome. The SLR thus included all types of research studies that were written in English and published up to and including November 2015. Furthermore, publication by peer-reviewed publication outlets (e.g., journals or conference proceedings) was considered as a quality criterion to ensure the academic level of the research papers.

Synthesizing the findings

The analysis of the final sample was performed by means of narrative and descriptive analysis techniques. For RQ1, the 76 papers were analyzed on the basis of bibliometric data (e.g., publication type, publication year, geography) and general performance measurement issues by paying attention to the methodology and focus of the study.

For RQ2, all the selected papers were screened to identify concrete performance indicators in order to generate a comprehensive list or checklist. The latter was done in different phases. In the first phase, the structured literature review allowed us to analyze which performance indicators are mainly used in the process literature and how they are concretized (e.g., in a question or mathematical formulation), resulting in an unstructured list of potential performance indicators. The indicators were also synthesized by combining similar indicators and rephrasing them into more generic terms.

The next phase was a comparative study to categorize the output of phase 1 into the commonly used measurement models in the performance literature. For the purpose of this study, we specifically looked for those organizational performance models, mentioned in "Theoretical background" section, that are cited the most and that suggest categories, dimensions or performance perspectives that can be re-used. Since the BSC is the most commonly used of these measurement models, we began with the BSC as the overall framework to categorize the observed indicators related to business (process) performance, supplemented with an established view on process performance from the process literature. Subsequently, a structured list of potential performance indicators was obtained.

In the third and final phase, an evaluation study was performed to validate whether the output of phase 2 is sufficiently comprehensive according to other performance measurement models, i.e., not included in our sample and differing from the most commonly used performance measurement models. Therefore, we investigated the degree to which our structured list covers the items in two variants or concretizations of the BSC. Hence, a validation by other theoretical models is provided. We note that a validation by subject-matter experts is out of scope for a structured literature review but relates to an opportunity for further research.