7 observations from a review of performance measurement research

What drives the use of performance measurement (PM) systems in the public sector? And what are the effects of PM use? In a recent study in Financial Accountability & Management (Van der Kolk, 2022, open access), I address these questions. In this blog, I highlight seven observations from the review.

Observation 1: Public sector PM seems here to stay

Fuelled by the new public management movement in the end of the 20th century, and crises in the beginning of the 21st century, performance measurement use and research has gained a strong foothold in practice and academia. In the examined survey studies, some evidence of the benefits of mobilizing PM were listed (e.g. enhanced efficiency, motivation, commitment) as reasons to implement PM systems, while respondents from one UK survey indicated that the use of key performance indicators was the single most successful accounting innovation in their organization.

Observation 2: Three categories of PM antecedents

One goal of the review was to investigate what drives the use of PM in public sector organizations. Three categories of antecedents were identified: 1) context variables (e.g. competition, political constraints), 2) organization variables (e.g. organizational values, financial stress), and 3) individual-level variables (e.g. respondent role, age, tenure, educational background). All reported antecedents and main effects have been visualized in four maps based on the empirical context of the study (map A: Government, map B: Health, map C: Education, map D: Other settings). Map A is included below (scroll down) and shows antecedents and effects of using PM in government settings (for the meaning of the various colors and arrows, see table 1 in the original paper, available here).

Observation 3: Three categories of PM effects

Another goal of the review was to map what sorts of effects public sector PM generates. Some studies find evidence that the adoption of a PM system impacts performance, in line with prior research. In the review, a distinction is furthermore made between performance-related effects (e.g., a more detailed PM positively impacts performance in a hospital), other organization-level effects (e.g., ease of use of PM impacts the perceived usefulness of PM practices) and individual level effects (e.g., PM subjectivity negatively impacts trust in a supervisor). It is important to note, however, that there are inherent limits to what can be concluded based on cross-sectional survey studies. Aspects that are not measured or expected do not become visible in quantitative survey studies, and other methods such as qualitative field studies should be used to complement this review to gain a more complet understanding of PM and its effects.

Source: Van der Kolk (2022, map A). More maps are available in the review paper.

Observation 4: Geographical bias

Looking at the geographical distribution of the empirical work conducted, it is clear that there is a strong bias towards certain geographical areas. For instance: 69% of the papers include evidence from Europe, 22% from Australia, and 9% from North America and Asia. Countries such as The Netherlands and Norway are well represented, while entire continents such as Africa and South America are absent. Given that culture can impact the effectiveness of management control instruments such as PM (cf. Malmi et al., 2020), a call for more research in “non-Western public sector contexts” (Van der Kolk, 2022, p. 723) therefore seems warranted.

Observation 5: Where are surveys on public sector PM published?

Although all 27 accounting journals that are rated 3 or higher in the ABS 2018 journal list were included in the initial search, only 8 journals published survey papers on public sector performance measurement between 1999 and 2018. From these journals, Financial Accountability & Management (17 papers), Accounting, Organizations & Society (4 papers), Management Accounting Research (3 papers) and the European Accounting Review (3 papers) were most strongly represented in the sample. Furthermore, it seems that scholars from the US who study PM publish less often in accounting journals, but rather send their work to outlets in the public management discipline, which may be a reason why also the US is not not strongly represented in this sample.

Observation 6: Task characteristics matter

When comparing this review to extant literature reviews from the discipline of public administration, it struck me that accounting scholars often point to task characteristics (e.g., output measurability, contractibility) to explain the effects of PM in different settings, while in the discipline of public administration this factor received considerably less attention. Comparing the two disciplines, it is (mostly) the accounting discipline that continuously stresses the importance of task characteristics to explain why PM yields different results in different settings.

Observation 7: Gaps and possibilities for future research

Based on the reviewed survey papers, a few gaps and directions for future research are identified. Among them are: 1) examining non-linear relations among PM and other variables (given that currently most survey studies only test linear relations while there are reasons to believe that these relations may not be linear in reality), 2) examining combinations of PM and other forms of management control ‘as a package’ (given the various calls and the limited evidence available on this topic in public sector organizations), and 3) studying PM in non-Western public sector settings.

Conclusion: PM, a blessing and/or a curse

In line with prior research within and outside of the accounting discipline, the findings from this review suggest that there is no easy answer to the question whether PM in the public sector is a blessing or a curse. PM can yield both favorable and unfavorable effects, and this review highlights that PM characteristics such as its perceived fairness, subjectivity and ease of use matter to a great extent for the effects PM can generate. And in line with prior research, much depends on the way how a PM system is used (cf. Van Elten et al., 2021, Gerrish, 2016). In short, what you measure isn’t always what you get.

This essay was also published as a blog of the European Accounting Association, and is based on the paper “Performance measurement in the public sector: Mapping 20 years of survey research”, published in Financial Accountability & Management (https://doi.org/10.1111/faam.12345).

References
Gerrish, E. (2016), The Impact of Performance Management on Performance in Public Organizations: A Meta-Analysis. Public Administration Review, 76: 48-66. https://doi.org/10.1111/puar.12433

Malmi, T., Bedford, D. S., Brühl, R., Dergård, J., Hoozée, S., Janschek, O., Willert, J., Ax, C., Bednarek, P., Gosselin, M., Hanzlick, M., Israelsen, P., Johanson, D., Johanson, T., Madsen, D. Ø., Rohde, C., Sandelin, M., Strömsten, T., & Toldbod, T. (2020). Culture and management control interdependence: An analysis of control choices that complement the delegation of authority in Western cultural regions. Accounting, Organizations and Society86, [101116]. https://doi.org/10.1016/j.aos.2020.101116 

van der Kolk, B. (2022). Performance measurement in the public sector: Mapping 20 years of survey research. Financial Accountability & Management, 38, 703– 729. https://doi.org/10.1111/faam.12345

van Elten, H. J., van der Kolk, B., Sülz, S. (2021). Do different uses of performance measurement systems in hospitals yield different outcomes? Health Care Management Review, 46, 3, 217-226. https://doi.org/10.1097/HMR.0000000000000261

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s