Free EMR Newsletter Want to receive the latest news on EMR, Meaningful Use, ARRA and Healthcare IT sent straight to your email? Join thousands of healthcare pros who subscribe to EMR and HIPAA for FREE!!

TURF: An EHR Usability Assessment Tool

Posted on October 22, 2013 I Written By

John Lynn is the Founder of the blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of and John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

The following is a guest post by Carl Bergman from EHR Selector.

To paraphrase Mark Twain, everyone talks about EHR usability, but no one does anything about it, at least until now. Led by Dr. Jiajie Zhang, the University of Texas Health Science Center at Houston’s National Center for Cognitive Informatics and Decision Making (NCCD) has developed several tools for measuring usability.

Now, Zhang’s team at NCCD has put several EHR usability tools into a Windows based app, TURF, an acronym for Task, User, Representation, Function. Funding for the project comes from ONC’s Strategic Health IT Advanced Research initiative.

TURF’s Tools. TURF has two major tools, Heuristic Evaluation and User Testing:

  • Tool One. Heuristic Evaluation: Expert Screen Capture and Markup. This tool takes EHR screen snapshots and let you compare them to usability standards. You can markup the screen and document the problem.
    Turf Expert Markup Tool - Showing Problem and Documentation
    For example, you can note if the error is minor, moderate, major or catastrophic. The system has a review function, so others can look at your markup and comment. The system also compiles your edits and can generate various statistics.

    • Administration. To work with groups, the system has several preset admin template forms and a template editor. The furnished templates cover these areas:
      • Demographics
      • Expert Review
      • Performance Evaluation, and
      • System Usability. This form asks 10 questions about the EHR, such as:
        • I think I would like to use the system frequently,
        • I thought there was too much inconsistency in this system,
    • Standards. The system uses the National Institute of Standards and Technology’s (NIST) EHR usability protocol, NISTIR 7804. You may also add your own rules to the system. (Also, see, June 14, 2012.)
    • EHR Sections. Using the NIST protocol, the system’s review areas are:
      • Clinical Decision
      • Clinical Information Reconciliation
      • Drug-drug, drug-allergy interactions
      • Electronic Medical Administration
      • ePrescribing
      • Med – Allergies
      • Medications list
      • Order Entry
      • User defined
  • Tool Two. Live Session Testing. TURF’s user test tool sits on top of an EHR and recording each movement. TURF’s designers have created a system that not only tracks use, but also adds these major functions:
    • User Sessions. TURF captures live screens, keystrokes, mouse clicks and can record a user’s verbal comments in an audio file.
    • Administration. The tool is designed for testing by groups of users as well as individuals. It captures user demographics, consent forms, non disclosures, etc. All of these can be tailored.
    •  Testing for Specifics. TURF allows managers to test for specific problems. For example, you can see how users eprescribe, or create continuity of care documents.
    • Comparing Steps. Managers can set up an optimum selection path or define the steps for a task and then compare these with user actions.
    • Reporting. TURF builds in several counting and statistical analysis tools such as one way ANOVA.

  • Running TURF. TURF isn’t your basic run and gun app. I downloaded it and then tried to duff my way through, as I would do with most new programs. It was a no go. Before you can use it, you need to spend some time setting it up. This applies to both its tools.

    Fortunately, TURF has about 30 YouTube tutorials. Each covers a single topic such as Setup for Electronic Data Capture and runs a minute or so. Here’s what they cover:
    Turf Tutorials Screen
  • Hands On. Installing TURF was straightforward with one exception. If you don’t have Microsoft’s .Net Framework 4.5 installed, put it up before you install TURF. Otherwise, the install stops for your to do it. TURF will also want the Codex that it uses for recordings installed, but the install deals with that.

    TURF is a Windows program, so I ran it in a virtual Win 7 session on my iMac. Given the environment, I kept the test simple. I ran TURF on top of a web based EHR and had it track my adding an antibiotic to a patient’s meds. TURF stayed out of the way, recording in the background.

    Here’s how TURF captured my session:
    Turf Playback Screen
    The left side screen played back my actions click for click. It let me run the screen at various speeds or stop it to add notes. The right screen lists each move’s attributes. You can mark any notable actions and document them for review by others. You can save your sessions for comparisons.

I found TURF to be a versatile, robust tool for EHR usability analysis. Its seeming complexity masks an ability to work in various settings and tackle hosts of problems.

If you aren’t happy with your EHR’s interface, TURF gives a remarkable tool to show what’s wrong and what you want. Indeed, with some adaptation you could use TURF to analyze almost any program’s usability. Not bad for a freebie.

New EHR Selector Website

Posted on March 21, 2013 I Written By

John Lynn is the Founder of the blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of and John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

As long as I’ve been blogging, I’ve been quite interested in the various EHR selection tools that are available out there. For a while, it seems like there was a new EHR selection tool coming out every week. The frenzy of EHR selection tools has mostly stopped and only a few major ones remain. One of those is EHR consultant (Full Disclosure: Has been an advertiser on EMR and HIPAA for many years) which has been doing this as long as anyone and has one of the most comprehensive EHR databases out there. The other that has lasted all this time is EHR Selector. Then, to round out the various EHR selection websites, there are the websites out there like Medical Software Advice and even the AAFP offers a tool to help in EHR selection (only available for AAFP members).

As I suggest in my e-Book on EMR selection, I think you should take advantage of multiple EHR selection tools. I call it triangulating the data since none of them are comprehensive with the EHR world changing so quickly. So, you take all of the data and triangulate down to the best EHR possible.

Many might wonder why I’m talking about EHR selection anyway. Well, if you’re being generous, we’re somewhere around 50% EHR adoption. That’s still a lot of doctors who haven’t adopted EHR. Add to that the number of clinics that are looking to switch EHR software, and there’s still a big need for great EHR selection tools.

The reason I started this look at EHR selection websites is because EHR Selector has rolled out an Open Beta with new features. The best feature they’ve rolled out is that EHR selector is now free for everyone to use. I was pretty harsh on EHR selector in the past for charging doctors to use their service. That business model just didn’t make sense to me. Ironically, Carl Bergman (one of the people behind EHR Selector), has since become a regular reader of this site and we regularly exchange emails about the EHR world. So, I’m glad that he converted the service to a free service for everyone to use.

I tried out the new release of EHR selector and I have some mixed reviews. It is a beta release so that’s partially to be expected. I was a bit overwhelmed by the number of fields I had to complete to get into the tool. I imagine that could be streamlined some to make it a better experience for the user. Although, I expect the data is part of the reason the service can be made free. Plus, the more data you offer, the better customized experience the EHR selection tool can offer you.

What I do love about EHR Selector is that it’s always had a deep set of data available. I wonder if this will backfire for some users thanks to the paradox of choice. Although, as a data lover I really love all the data. Plus, they could help solve this issue with how they choose to display the various EHR and their data with a beautiful UI. I don’t think the UI is to that point yet, but having the data is the first step in that process.

My favorite feature of EHR Selector is the Compare EHR feature. I love seeing the features of multiple EHR’s compared side by side. This is where all the EHR data points becomes really valuable as well. One thing I do wish is there was more than just a check mark for each EHR data point. I’d love to have some qualitative description or images of each data point so you could really compare the EHR features that matter most to you.

One of the other shortcomings of EHR Selector is they haven’t yet gotten the data for all 300+ EHR vendors. No doubt that’s a daunting task and you have to start somewhere. I expect they’ll work to resolve this over time. Plus, they need the support of the EHR vendors to be able to get the data as well (not always an easy task).

Certainly EHR Selector isn’t perfect yet, but with its new Free feature it’s worth taking a look at if you’re in the market for an EHR. It’s another nice data point in the EHR selection process.

Redesigning The Patient Medical Record, the Healthcare Challenge’s Results

Posted on January 28, 2013 I Written By

John Lynn is the Founder of the blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of and John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

The following is a guest post by Carl Bergman from EHR Selector.

The Obama administration’s, site encourages the public to submit suggestions that solve specific, public policy questions. To do this, it’s set up dozens of contests or challenges. For example, the FTC has a $50,00 challenge for a solution to illegal robo calls that often come from off shore.

In healthcare, the VA and the ONC recently ran a Health Design Challenge for a better patient health record announcing the winners a few days ago.

The challenge asked for a record that:

  • Improves the visual layout and style of the information from the medical record
  • Makes it easier for a patient to manage his/her health
  • Enables a medical professional to digest information more efficiently
  • Aids a caregiver such as a family member or friend in his/her duties and responsibilities with respect to the patient

The entries were judged by a twelve person panel ranging from Wired Magazine’s Executive Editor, Thomas Goetz to Facebook’s Product Designer, Nicholas Felton to Dr. Sophia Chang, the director of the Chronic Disease Care program of the California Health Care Foundation. They looked at several features of a revamped record from overall appeal to how readily it shows important information and how accessible it is for physicians, patients, etc.

The Winners

The judges picked three big winners and three winners in the Problem History, Medication and Lab Summaries areas. Here’s a brief look at the top entries, but the submissions should be looked at more as a resource than a race result, as I’ll discuss.
First place went to Nightingale an anonymous group that won $16,000. Others won smaller amounts. In the next few months, elements of the winning designs will be put together and put up on Github.

Nightingale’s design stressed that health was a continuing concern and that a user should be able to see an improving or declining trend without having to dig for the data. They did this by integrating the often disparate information in visits, exams and lab results. You can see this emphasis in their lipid panel screen. Sliders place each test result for each test’s in a range. Good results slide to green while poorer result move to red.
Second place StudioTACK took a somewhat similar approach to creating a problem history, which they call a medical strategy rather than a record. They did this by bringing their findings into a body map with references to location and organ.

Matthew Sanders’ CCD scored the best Problem History section award. Sanders rearranged and redesigned the traditional note not by condition nor by past chronology, but into a timeline of past, present and future actions. While he admits that his approach is somewhat redundant for meds, he emphasizes that this arrangement helps all the users maintain a focus on the most important areas for action. Sanders presentation notably describes how he implemented his approach. To do this, he stripped out standard label text, clarified terms and gave the remaining items visual emphasis. This type of analysis makes going through the submissions worth it.
Sanders CCD
This isn’t to say that the way the contest was run and the approach of many submissions  — including some prize winners — were without shortcomings. There were some notable problems.

The Contest’s Problems

The contest’s operators needed to be far more specific about what they wanted and how they judged the results.

The challenge’s purpose was far from clear:

The purpose of this effort is to improve the design of the medical record so it is more usable by and meaningful to patients, their families, and others who take care of them. This is an opportunity to take the plain-text Blue Button file and enrich it with visuals and a better layout. Innovators will be invited to submit their best designs for a medical record that can be printed and viewed digitally.

A medical record is an on going repository of a person’s health context, status, prognosis, plans, etc. It has many contributors and users. The VA’s Blue Button is a snapshot of the person’s status for their use. However, the contest uses these terms interchangeably. Due to this muddle, many of the submissions sent in designs for a medical record, while others, a minority, only redid the Blue Button’s outline. Thus, not all submissions were developed on the same basis. Indeed, the judges seem to acknowledge this since they gave first place to Nightingale, which claims, “to be a new take on health records.” The contest would have done much better if it asked for particular types of screens putting everyone on the same page, as it were.

The contest judging panel while distinguished, had no practicing physicians, nurses or practice managers, a significant failing. While three of the twelve judges are MDs, not one is a practicing physician.

Finally, if you’re going to hand out $50,000 in public funds, you might just want to say why you thought the winners stood out.

The Submissions

The contestants almost universally got one thing right. They designed their entries for desktops/laptops, pads and phones. They showed a great understanding that we don’t work on just one platform, but move from one to the other almost continuously. In this, they deserve much praise. However, all this cross platform awareness is done in by an appalling over, under and misuse of font color, and size. As one post noted about Nightingale:

The text is too small and medium gray on light gray is very hard to see, especially for older people and people on cheap computers with low contrast displays. How can this possibly be the first place winner?

The comment is generous. Nightingale’s gray on gray font is almost unreadable. Granted their submission is a PDF of a prototype, nonetheless the possibility of staring at their screens all day would give me a headache.

They are not alone in color misuse. Second place winner, Studio TACK, goes to excess the other way with a white text on red iPhone screen. It’s more suited to public safety than health.
StudioTack Mobile
Going through the submissions, however, can be most rewarding. I found a gem of a summary page in Uncorkit’s submission. Their infographic approach puts not only labs and weight history on timelines, but also includes BP, conditions and meds. It gives you a great overview and a logical place to drive down for detail information without overwhelming your senses.

The Health Challenge submissions have much to recommend them. Just remember how they came about and what they may or may not include.

What Keeps EMR and HIPAA Running and New Healthcare Scene Service

Posted on January 12, 2012 I Written By

John Lynn is the Founder of the blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of and John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

I’m sure that many of you probably don’t know the story and history of EMR and HIPAA. Some day I’ll have to share the entire story, but EMR and HIPAA has been around for about 6 years and about two years ago during a panel at HIMSS I announced that I was going to quit my job and become a full time blogger. I must admit that it is pretty much a dream for me that I’m able to be a full time blogger doing something that I love. A long time ago I chose advertising as my model for making this happen. Thankfully there have been a lot of great companies that have supported me and this blog. Many of them for many years.

As I do every couple months I like to recognize the new EMR and HIPAA advertisers and recognize those advertisers that have renewed their ads. Many of the ad renewals have been doing so for a lot of years. I might have to go back and research how long many of these advertisers have been renewing on EMR and HIPAA and recognize them for this in the future.

Also, we have a NEW service we’re offering, so read to the bottom to learn more.

New EMR and HIPAA Advertisers
Medical Mastermind – I’ve been talking with Medical Mastermind for a while now about advertising, so I’m glad that not only did they decide to advertise on EMR and HIPAA, but they’re advertising across the entire Healthcare Scene blog network. They are offering a special Buy 2 EMR licenses, Get 1 Free offer right now for readers of this blog. One thing I really like about Medical Mastermind is that over the past year or so they’ve come under new management that’s really taking them to the next level as a company. Plus, they have a bunch of clients that have been with them for a long time. It’s always great for a company to have a base of existing users that have been with them for a while.

PTS Consulting Group – PTS Consulting Group is a multi-national consulting group with offices in Boston, New York and London (to name a few). I know a number of the consultants out of Boston and they are some of the most pleasant and professional people you can meet. They have some great war stories and you might remember the meaningful use tool they created. If you’re looking for an EMR consultant, they offer a free phone consultation so you can see if they’re a good fit for your organization. Plus, one of their best resume items is that they’re regular readers of EMR and HIPAA (Sorry, I couldn’t help myself).

Valant Medical Solutions – Valant Medical Solutions offers an EHR focused solely on the mental health profession. No doubt they agree with me that there’s a huge opportunity available to specialty specific EHR software. I think few would argue that the mental health space has unique needs where they can benefit from a specialty specific EHR. Turns out investors seem to agree since Valant Medical recently closed an almost $1 million funding round. This is a good sign for mental health professionals looking for a mental health specific EHR that’s going to be around for a while to come.

Dynamic Vault – One of the things that I think clinics do incredibly poorly is making sure that they have a quality backup of their EHR system. I’ve done a whole series on EHR backups before and I should do it again. Dynamic Vault offers a service to help solve a lot of the EHR backup problems. They handle Access Controls, Audit Controls, Data Integrity and provide a contingency plan. I’d encourage every clinic to evaluate their EHR backup plan and make sure it is sound. Otherwise, you should check out a service like Dynamic Vault.

Renewing EMR and HIPAA Advertisers
Now a big thanks to the following advertisers that have renewed their ad on EMR and HIPAA: Cerner, Practice Fusion, SOAPware, EMR Consultant, and Sequelmed. There are a bunch more in the process of renewing too. Watch for another batch coming soon.

New Healthcare Scene Service
Over time Healthcare Scene has gotten a lot of requests from companies to create content for them. My normal response was that I really didn’t have time to create content for other places. As I’ve grown Healthcare Scene to now something like 10 writers (depending on how you count), I realized that we do have a bunch of talented people who could create content for other companies as well.

With this as inspiration, I got together with one of my writers, Anne Zieger to put together a new Healthcare Scene Content Service. In fact, we already have 3 clients we’re working with now on content creation with more in discussion. While we certainly are great at creating blog posts, Anne Zieger also has years of experience creating whitepapers, case studies, surveys, webinars and other custom content.

Besides our experience in the healthcare and healthcare IT space, our content creation service has the unique ability to also be able to distribute the content that’s created across the Healthcare Scene network. This makes for a really compelling opportunity for companies with a story to get their voice heard.

We’re putting the finishing touches on a PDF describing the various services we can offer and as webpage describing those services as well. Until then, if you’re interested in more details, drop me a line on our Contact Us page and we’ll get right back to you with more details.

The NIST Workshop on EHR Usability

Posted on June 14, 2011 I Written By

John Lynn is the Founder of the blog network which currently consists of 10 blogs containing over 8000 articles with John having written over 4000 of the articles himself. These EMR and Healthcare IT related articles have been viewed over 16 million times. John also manages Healthcare IT Central and Healthcare IT Today, the leading career Health IT job board and blog. John is co-founder of and John is highly involved in social media, and in addition to his blogs can also be found on Twitter: @techguy and @ehrandhit and LinkedIn.

As much as I’d like to visit DC (I’ve never been), I wasn’t able to make it out there to attend the NIST workshop on EHR usability. However, Carl Bergman from EHR Selector did make it to the event and sent the following notes on EHR usability according to NIST.  Most of the speakers name link to their slides in PDF format.

National Institute of Standard and Technology’s Workshop on EHR Usability

This week I went to a NIST workshop examining the state of EHR usability. The workshop was at its administrative headquarters, a large 60s building on its sprawling Gaithersburg, MD campus about 20 miles outside Washington.

You might wonder what NIST is doing in the EHR business? I certainly did. NIST’s mission is to promote commerce and technical innovation including methods to determine, independently, the safety and security of a broad range of technologies including software. (It’s part of the Department of Commerce.) Since WW II, this has involved looking at the human factors involved in operation of every thing from nuclear plants to robotics. Interestingly, it’s not a regulatory agency, such as, the FDA or FCC. NIST’s standards work is through consensus building among manufacturers, consumers, regulators, etc.

The workshop, attended by about 200 persons, had two parts:

•      A review of the state of EHR usability studies by academics, practioners and system administrators and,

•      Introduction of NIST’s draft for a usability standard.

Part I. EHR Usability Today. There were many speakers, here’re the ones that had the most new information for me:

•      Mat Quinn of NIST covered its approach and work with ONC on the issue. Notably, NIST has published several documents in the area such as, NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, (NISTIR 7741) which promotes a user centric approach to design and development.

•      I was really taken by Muhammad Walji’s study using a unified framework for EHR testing. The study compared user experience with the VA’s Vista program and a prototype system. It looked at:

o   What percent of an operation was substantive and what was overhead?

o   How long it took users to reach various performance levels.

o   How much memorization tasks took.

o   How many steps tasks required.

o   Error and recovery occurrence.

o   Time to complete defined tasks.

The study then applied its findings to rework the EHRs’ structure and workflow showing potential time and effort savings.

•      Anjum Chagpar of Toronto’s University Health Network. A human factors manager for this large healthcare network, she discussed the problems of integrating various vendor products into their system and their approach to usability and user satisfaction.

•      Buckminster Fuller famously declared, “I am a verb.” Dr. Lyle Berkowitz may not be a verb, but he is at least a gerund. His presentation swiftly covered several topics from HIMSS’ EHR Usability Task Force to usability definitions to stakeholder roles, and applying metrics to see how much of the problem was the system and how much the user.

•      The VA’s Dr. Jorge Ferrer provided several key references on usability studies.

Part II. NIST’s Proposed Protocol. If the first part took a broad and free ranging approach to usability, NIST’s staff approach was more focused. After an outline of the study’s setting and approach, the study director, Lana Lowery, outlined the protocol’s goal: prevention of unacceptable medical errors. These include errors of both omission and commission, for example:

•      Writing an order for the wrong patient.

•      Prescribing the wrong dosage.

•      Omitted information causing an error.

•      Critical delays in delivery due to system design errors.

•      Errors due to incorrect sequencing of actions.

Next, came examples of EHRs allowing errors. Unfortunately, several of the examples weren’t well thought out. For example, a patient ID error showed two patient records on the screen. One had the first patient’s x-ray, but the second patient’s name. Most likely, this would be a database problem or an x-ray production error not an EHR problem.

Robert Schumacher of User Centric, outlined how the protocol would be tested. For example, review and update of a problem list or replacement of one medication with another. The plan included testing several of ONC’s meaningful use functions that had usability factors.

Part III. Workshop Reactions. The workshop finally broke into two discussion groups: one for the draft protocol and the other on consensus building. In both cases, the discussion quickly went off script. Participants were quick to criticize the staff’s error oriented protocol as too narrow. Why, for example, did the protocol focus on internal EHR processes to the exclusion of workflow generated errors?

I understand NIST has a high interest in eliminating catastrophic errors, but I think there is not enough solid evidence on the kind and extent of the problem. No one discounts the need to prevent catastrophic errors, however, much of the EHR error focus is due to anecdotal reports of computer prescribing errors. From what I read, many of these reports are both old and recycled. Does anyone know the actual extent of major errors?

The FDA has developed several systems for dealing with medical device errors. These now include the software that the devices use. Even if the FDA does not regulate EHRs, it may step up its efforts to record important errors. I’d sure like to know FDA’s findings before I started an effort to shape EHRs.

This is not to say that safety is not important in EHRs, obviously the types of errors that are outlined by the staff are major. However, I think there are three points that are missing in the NIST approach:

•      Design for Success. You can’t design for failure. You have to design for success. The object of EHRs, as with any system, must be to accomplish certain ends. If you loose sight of that, you may not make mistakes, but you also will fail your objective.

•      Risk Analysis. Risk analysis measures the impact on a given population of an action, its potential and costs broadly defined. It also specifies mitigation efforts. I’d be far more comfortable about the protocol if there were a risk analysis behind it.

•      Error Handling. There should be more thought to error handling. For example, when the stall warning alarm goes off on a plane, it doesn’t grab the stick and take control. It’s a warning, just that. Physicians should be warned if they are about to prescribe beyond the recommended dose, but they may have good clinical reason to do it.

NIST put on a worthwhile workshop. My guess is that the draft protocol is not going to survive without modifications that take into account a broader range of usability issues and approaches.