How to Find Value in KPI Lifecycle

By John Horst, MESA member, National Institute of Standards and Technology (NIST)

and Barry Ezell,Virginia Modeling, Analysis and Simulation Center (VMASC) 

Key Performance Indicators (KPIs) are part of a lifecycle because KPIs are continuously defined, redefined, and are even sometimes abandoned. Explicitly defining a KPI Lifecycle provides an impetus for the development of methods and tools that enhance performance.  

NIST and VMASC have designed two methods to define KPI Lifecylce: one for performing KPI Set Composition, and the other for KPI Assessment.  

Both methods need to produce accurate results. But how can we know that the methods are producing accurate results --thereby providing real value to the manufacturer?

TESTING NOT SIMULATION

Given the nature of the models, KPI Set Composition and KPI Assessment cannot be effectively modeled via simulation software. Therefore, in order to verify the efficacy of both methods, live tests with process stakeholders need to be conducted. And, the tests must be performed at real plant operations with dedicated involvement of process stateholders.

To date, we have conducted three tests on three target processes at three separate plants. Two out of three tests involved the KPI Assessment method only.

We did this because all three organizations are mature in their use of KPIs in the chosen target processes. Two are chemical plants (one large and one small), and one is a micro-chip fabrication plant. Each testing event took less than one day to perform and included full involvement from expert stakeholders.

Note that KPI Set Composition was never fully tested because the majority of KPIs examined by the methods were already implemented in the target processes that were tested. In the future, we will examine a process and test the KPI Set Composition method on a target process that currently has no KPIs assigned to it.

Output of KPI Assessment Method
KPI	Criterion	Improvement Score
KPI #10	Quantifiable	5.4
KPI #10	Documented	5.2
KPI #10	Accurate	5.2
KPI #10	Buy-in	5
KPI #10	Independent	4.8
KPI #5	Buy-in	4.5
KPI #8	Buy-in	4.5
KPI #7	Buy-in	4.5
KPI #8	Buy-in	4.5
KPI #2	Buy-in	4.5
KPI #6	Buy-in	4.5
KPI #1	Buy-in	4.5
KPI #4	Buy-in	4.5

GOALS FOR KPI ASSESSMENT METHOD

1. The first goal for testing events using the KPI Assessment method is to generate a measurement of the general effectiveness of each KPI with respect to a set of KPI performance criteria from ISO 22400 Part 
2. The second goal is to generate KPI/criterion pairs and rank them based on the level of priority that the KPIs need to be improved in the domain of the criterion.
3. The third goal is to examine how well the KPIs satisfy the critical objectives. For example, scores were generated in answer to the question ”How well is the KPI under test performing with respect to having buy-in from the process stakeholders?”

RESULTS FOR KPI ASSESSMENT METHOD

Example test results for KPI effectiveness and KPI-criterion priorities are shown in the table labeled ‘Output of KPI Assessment Method.’   The KPI scores for the 10 candidate KPIs give a general effectiveness score across all the criteria. The improvement scores for {KPI, criterion} pairs are high when the KPI is important to the process but is performing poorly with respect to the criterion. 

For more information on the KPI Assessment Method, visit MESA's Resource Library.

Field Tests of KPI Effectiveness

Key Performance Indicator Assessment Model