PITCH Process evaluations – why do we do them, how do we get the most out of them, and do we all agree?

Problem

Primary care interventions are often complex, and it is therefore increasingly considered important to undertake process evaluations in conjunction with their outcome evaluations. Despite a growing body of guidance and methodological literature for process evaluations, there is a range of apparently conflicting opinions about their purpose and value. There are some conflicting methodological recommendations, and a range of reported challenges to their planning, conduct and dissemination. Different understanding of what is meant by ‘high-quality’ and ‘useful’ process evaluations add to this confusion.

By synthesising the diverse range of perspectives in the methodological literature, this review provides a novel conceptual framework for researchers and other stakeholders to reflect on the potential value of process evaluations and how this may be realised.

Approach

Critical narrative review of literature relating to methodology, guidance, or opinions about process evaluation. 94 papers were systematically identified for inclusion. Literature sources included methodological reflections by researchers, PhD theses, editorials, and protocols for process evaluation.

Findings

We identified 14 potential values and 4 potential harms of process evaluation. Perceived values included ‘supporting implementation into practice’, ‘increasing the credibility of the outcome evaluation’, and ‘financial value’. Harms included ‘biasing the outcome results’ and ‘providing erroneous evidence’. Values are associated with how process evaluation knowledge is used, the perceived qualities of this knowledge, and the act of undertaking the process evaluation. The achievement of certain values may also cause harms or negate other values, highlighting a need to balance priorities.

Perceptions about how values may be achieved related to methodological recommendations, dissemination, and contextual influences. There are diverse opinions about how certain values are best achieved (especially whether process evaluations should be formative or summative) and differing perspectives on causality, complexity, truth and bias. Some of the values and harms are in themselves subjective. It is therefore important that stakeholders in process evaluations reflect upon and discuss the values that they hope to achieve and how these will be realised, to avoid potential conflict and disappointment.

Consequences

We propose a conceptual framework for stakeholders to reflect on and discuss their expectations of the value they hope to gain from process evaluations, and how this will be achieved. This will assist identification of potentially problematic differences in expectations. The framework also raises awareness of potential values and harms that may not have previously been considered and draws attention to potential conflicts between values to guide informed decisions.