TURF: An EHR Usability Assessment Tool

The following is a guest post by Carl Bergman from EHR Selector.

To paraphrase Mark Twain, everyone talks about EHR usability, but no one does anything about it, at least until now. Led by Dr. Jiajie Zhang, the University of Texas Health Science Center at Houston’s National Center for Cognitive Informatics and Decision Making (NCCD) has developed several tools for measuring usability.

Now, Zhang’s team at NCCD has put several EHR usability tools into a Windows based app, TURF, an acronym for Task, User, Representation, Function. Funding for the project comes from ONC’s Strategic Health IT Advanced Research initiative.

TURF’s Tools. TURF has two major tools, Heuristic Evaluation and User Testing:

• Tool One. Heuristic Evaluation: Expert Screen Capture and Markup. This tool takes EHR screen snapshots and let you compare them to usability standards. You can markup the screen and document the problem.
  
  For example, you can note if the error is minor, moderate, major or catastrophic. The system has a review function, so others can look at your markup and comment. The system also compiles your edits and can generate various statistics.
  • Administration. To work with groups, the system has several preset admin template forms and a template editor. The furnished templates cover these areas:
    • Demographics
    • Expert Review
    • Performance Evaluation, and
    • System Usability. This form asks 10 questions about the EHR, such as:
      • I think I would like to use the system frequently,
      • I thought there was too much inconsistency in this system,
  • Standards. The system uses the National Institute of Standards and Technology’s (NIST) EHR usability protocol, NISTIR 7804. You may also add your own rules to the system. (Also, see EMRandEHR.com, June 14, 2012.)
  • EHR Sections. Using the NIST protocol, the system’s review areas are:
    • Clinical Decision
    • Clinical Information Reconciliation
    • Drug-drug, drug-allergy interactions
    • Electronic Medical Administration
    • ePrescribing
    • Med – Allergies
    • Medications list
    • Order Entry
    • User defined

• Tool Two. Live Session Testing. TURF’s user test tool sits on top of an EHR and recording each movement. TURF’s designers have created a system that not only tracks use, but also adds these major functions:
  • User Sessions. TURF captures live screens, keystrokes, mouse clicks and can record a user’s verbal comments in an audio file.
  • Administration. The tool is designed for testing by groups of users as well as individuals. It captures user demographics, consent forms, non disclosures, etc. All of these can be tailored.
  •  Testing for Specifics. TURF allows managers to test for specific problems. For example, you can see how users eprescribe, or create continuity of care documents.
  • Comparing Steps. Managers can set up an optimum selection path or define the steps for a task and then compare these with user actions.
  • Reporting. TURF builds in several counting and statistical analysis tools such as one way ANOVA.

• Running TURF. TURF isn’t your basic run and gun app. I downloaded it and then tried to duff my way through, as I would do with most new programs. It was a no go. Before you can use it, you need to spend some time setting it up. This applies to both its tools.
  
  Fortunately, TURF has about 30 YouTube tutorials. Each covers a single topic such as Setup for Electronic Data Capture and runs a minute or so. Here’s what they cover:
  
• Hands On. Installing TURF was straightforward with one exception. If you don’t have Microsoft’s .Net Framework 4.5 installed, put it up before you install TURF. Otherwise, the install stops for your to do it. TURF will also want the Codex that it uses for recordings installed, but the install deals with that.
  
  TURF is a Windows program, so I ran it in a virtual Win 7 session on my iMac. Given the environment, I kept the test simple. I ran TURF on top of a web based EHR and had it track my adding an antibiotic to a patient’s meds. TURF stayed out of the way, recording in the background.
  
  Here’s how TURF captured my session:
  
  The left side screen played back my actions click for click. It let me run the screen at various speeds or stop it to add notes. The right screen lists each move’s attributes. You can mark any notable actions and document them for review by others. You can save your sessions for comparisons.

I found TURF to be a versatile, robust tool for EHR usability analysis. Its seeming complexity masks an ability to work in various settings and tackle hosts of problems.

If you aren’t happy with your EHR’s interface, TURF gives a remarkable tool to show what’s wrong and what you want. Indeed, with some adaptation you could use TURF to analyze almost any program’s usability. Not bad for a freebie.