Automatic auditing of out-of-hours consultation records for safety-netting advice using natural language processing techniques

Problem

It is recommended that ‘safety-netting advice’, (defined as, “information shared with a patient or their carer, designed to help them identify the need to seek further medical help if their condition fails to improve, changes, or if they have concerns about their health”), is provided to patients consulting during out-of-hours primary care consultations. As such, the documentation of safety-netting advice is an auditable standard in the Royal College of General Practitioners endorsed 'Urgent and Emergency Care Clinical Audit Toolkit'. However, clinical audits are often time-consuming, conducted sporadically, and frequently require highly trained clinicians to conduct them instead of utilising their clinical skills seeing patients. Additionally, safety-netting advice is often spoken in the format of “if X symptom(s) happen then seek medical help at Y", but medical notes frequently contain unexplained abbreviations (for example, “tcb inb” = to come back if not better).

Approach

As part of a previous project, 1886 out-of-hours consultation notes from the Birmingham Out-of-hours general practice Research Database (BORD) from July 2013 to February 2020 with adult (≥18 years) patients have been manually coded for the presence and type of safety-netting advice. 1472 (78.0%) of records contained documented safety-netting advice. This manual coding will be used to train multiple machine learning models using natural language processing techniques to automatically predict which consultations contain documented safety-netting advice. In addition, a ‘rules-based’ model will also be generated searching for common safety-netting phrases, and abbreviations, allowing for spelling errors.

Findings

The four key metrics for assessing natural language processing classification models will be reported for each model including: accuracy, recall (sensitivity), precision (positive predicted value) and F1 score (a combination of recall and precision).

Consequences

If the models can automatically detect safety-netting advice with sufficient accuracy, this could save a lot of clinician time auditing records manually and allow for enhanced monitoring and governance of clinical practice. Furthermore, the models could be used to generate real-time in-consultation feedback to clinicians, promoting them to record their safety-netting advice, or if this has been omitted, communicate this information post-consultation, for example using SMS text messaging systems.