The e-Health Implementation Toolkit (e-HIT): Reflections on its Application in the Evaluation of a national digital health deployment (‘dallas’).

Problem

The dallas (Delivering Assisted Living Lifestyles at Scale) programme is unique in its ambition to deliver a broad range of digital health services at scale. Large heterogeneous partnerships involving businesses, third sector organisations and statutory services came together to implement and deliver dallas products. A longitudinal evaluation of the dallas program has been ongoing in parallel since its inception in May 2012. One of the evaluation methods used was the eHit (e-Health Implementation Toolkit), a questionnaire derived from normalisation process theory (NPT), a sociological framework which helps examine how interventions become implemented and integrated into healthcare settings. The aim of this study was to assess the suitability of eHit as a tool to evaluate the implementation processes in a national implementation programme.

Approach

We conducted a series of in depth, e-HIT – led qualitative Research Interviews at baseline, mid-point and end-point of the dallas programme. The e-HIT consists of a series of 21 validated research questions covering aspects related to the Collective Action construct of NPT. e-HIT scores and additional interview comments were recorded from a sample of key stakeholders from each dallas consortium. Responses were analysed descriptively with comparisons made from each timepoint. Furthermore, transcripts were coded for areas of eHIT misalignment whereby the stakeholder indicated that some questions did not correspond exactly with the multi-agency, evolving nature of the dallas partnership development work.

Findings

Interim analysis of 21 e-HIT scores between baseline and mid-point indicated an underestimation of the workload involved in the initial stages of the implementation. Furthermore, qualitative data analysis revealed some stakeholders had difficulties in answering eHIT questions at the time of the interview, for example, when products were still under iterative development or there was at times a lack of joint understanding of national or local policy. At times, stakeholders commented that it was difficult to assign a number on a scale to an eHIT question when they had been involved implementing multiple dallas products with varying results. However, the e-HIT did provide a valuable starting point for a comprehensive qualitative interview addressing implementation issues.

Consequences

The e-HIT tool kit proved effective and acceptable in the evaluation of the dallas programme and enabled us to understand the changes related to implementation at baseline, mid-point and end point, however a note of caution is required. As a generic tool and depending on the stage of implementation, some of the questions may have been deemed unsuitable or premature. However it is essential to note that the e-HIT toolkit was sufficiently flexible and elicited rich qualitative data allowing scores to be embedded in the context. In addition, it was one of a number of evaluation tools employed to gather a comprehensive dataset across the entire dallas programme.