Written by: John Lynn
As much as I’d like to visit DC (I’ve never been), I wasn’t able to make it out there to attend the NIST workshop on EHR usability. However, Carl Bergman from EHR Selector did make it to the event and sent the following notes on EHR usability according to NIST. Most of the speakers name link to their slides in PDF format.
National Institute of Standard and Technology’s Workshop on EHR Usability
This week I went to a NIST workshop examining the state of EHR usability. The workshop was at its administrative headquarters, a large 60s building on its sprawling Gaithersburg, MD campus about 20 miles outside Washington.
You might wonder what NIST is doing in the EHR business? I certainly did. NIST’s mission is to promote commerce and technical innovation including methods to determine, independently, the safety and security of a broad range of technologies including software. (It’s part of the Department of Commerce.) Since WW II, this has involved looking at the human factors involved in operation of every thing from nuclear plants to robotics. Interestingly, it’s not a regulatory agency, such as, the FDA or FCC. NIST’s standards work is through consensus building among manufacturers, consumers, regulators, etc.
The workshop, attended by about 200 persons, had two parts:
• A review of the state of EHR usability studies by academics, practioners and system administrators and,
• Introduction of NIST’s draft for a usability standard.
Part I. EHR Usability Today. There were many speakers, here’re the ones that had the most new information for me:
• Mat Quinn of NIST covered its approach and work with ONC on the issue. Notably, NIST has published several documents in the area such as, NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, (NISTIR 7741) which promotes a user centric approach to design and development.
• I was really taken by Muhammad Walji’s study using a unified framework for EHR testing. The study compared user experience with the VA’s Vista program and a prototype system. It looked at:
o What percent of an operation was substantive and what was overhead?
o How long it took users to reach various performance levels.
o How much memorization tasks took.
o How many steps tasks required.
o Error and recovery occurrence.
o Time to complete defined tasks.
The study then applied its findings to rework the EHRs’ structure and workflow showing potential time and effort savings.
• Anjum Chagpar of Toronto’s University Health Network. A human factors manager for this large healthcare network, she discussed the problems of integrating various vendor products into their system and their approach to usability and user satisfaction.
• Buckminster Fuller famously declared, “I am a verb.” Dr. Lyle Berkowitz may not be a verb, but he is at least a gerund. His presentation swiftly covered several topics from HIMSS’ EHR Usability Task Force to usability definitions to stakeholder roles, and applying metrics to see how much of the problem was the system and how much the user.
• The VA’s Dr. Jorge Ferrer provided several key references on usability studies.
Part II. NIST’s Proposed Protocol. If the first part took a broad and free ranging approach to usability, NIST’s staff approach was more focused. After an outline of the study’s setting and approach, the study director, Lana Lowery, outlined the protocol’s goal: prevention of unacceptable medical errors. These include errors of both omission and commission, for example:
• Writing an order for the wrong patient.
• Prescribing the wrong dosage.
• Omitted information causing an error.
• Critical delays in delivery due to system design errors.
• Errors due to incorrect sequencing of actions.
Next, came examples of EHRs allowing errors. Unfortunately, several of the examples weren’t well thought out. For example, a patient ID error showed two patient records on the screen. One had the first patient’s x-ray, but the second patient’s name. Most likely, this would be a database problem or an x-ray production error not an EHR problem.
Robert Schumacher of User Centric, outlined how the protocol would be tested. For example, review and update of a problem list or replacement of one medication with another. The plan included testing several of ONC’s meaningful use functions that had usability factors.
Part III. Workshop Reactions. The workshop finally broke into two discussion groups: one for the draft protocol and the other on consensus building. In both cases, the discussion quickly went off script. Participants were quick to criticize the staff’s error oriented protocol as too narrow. Why, for example, did the protocol focus on internal EHR processes to the exclusion of workflow generated errors?
I understand NIST has a high interest in eliminating catastrophic errors, but I think there is not enough solid evidence on the kind and extent of the problem. No one discounts the need to prevent catastrophic errors, however, much of the EHR error focus is due to anecdotal reports of computer prescribing errors. From what I read, many of these reports are both old and recycled. Does anyone know the actual extent of major errors?
The FDA has developed several systems for dealing with medical device errors. These now include the software that the devices use. Even if the FDA does not regulate EHRs, it may step up its efforts to record important errors. I’d sure like to know FDA’s findings before I started an effort to shape EHRs.
This is not to say that safety is not important in EHRs, obviously the types of errors that are outlined by the staff are major. However, I think there are three points that are missing in the NIST approach:
• Design for Success. You can’t design for failure. You have to design for success. The object of EHRs, as with any system, must be to accomplish certain ends. If you loose sight of that, you may not make mistakes, but you also will fail your objective.
• Risk Analysis. Risk analysis measures the impact on a given population of an action, its potential and costs broadly defined. It also specifies mitigation efforts. I’d be far more comfortable about the protocol if there were a risk analysis behind it.
• Error Handling. There should be more thought to error handling. For example, when the stall warning alarm goes off on a plane, it doesn’t grab the stick and take control. It’s a warning, just that. Physicians should be warned if they are about to prescribe beyond the recommended dose, but they may have good clinical reason to do it.
NIST put on a worthwhile workshop. My guess is that the draft protocol is not going to survive without modifications that take into account a broader range of usability issues and approaches.