EVALUATING USABILITY OF AN ELECTRONIC HEALTH RECORD SMARTPHRASE: TRIAGE NURSES USE AND PERCEPTIONS

Electronic Health Record (EHR) systems have the potential to be beneficial, but healthcare vendors are challenged to find ways to make them easier to work with and more productive. Systems and tools should be designed to enable nurse-patient engagement, seamlessly fit into practice, and be based on workflow needs. The usability of EHR systems is probably the key factor in making the design of the system fit healthcare providers’ workflow and display patient information clearly. A quantitative descriptive, non-experimental study was conducted to evaluate the usability of a new SmartPhrase interface tool embedded into the Epic EHR system. Twenty-three telephone triage nurses in a multi-site cancer center, located within the largest healthcare system in Rhode Island, were involved in designing and testing the interface tool. The task of the SmartPhrase tool is to permit quick and easy insertion of pretexted symptom assessment cues or phrases into a patient’s EHR. The SmartPhrase tool evolved as a basic unit of the Epic EHR in the Lifespan system, specifically as an electronic documentation feature that permits nurses to use pre-formatted text, statements, or structure note templates for cuing documentation. The specific SmartPhrase tool was developed in the context of a larger parent study to improve standardized telephone-triage symptom assessment for cancer patients. Generalized Linear Mixed Model was used to evaluate the usability of an EHR SmartPhrase tool to (1) determine the relationship between telephone triage nurses’ years of experience (in nursing, telephone triage, and oncology) and their perception of the usability of the SmartPhrase tool; (2) assess changes in self-perceived job performance six months following SmartPhrase implementation; and (3) determine the relationship between the telephone triage nurses’ evaluation of the SmartPhrase tool usability and actual tool utilization. The SmartPhrase tool, as evaluated by telephone triage nurses, was not perceived as particularly useful. Although the usability score was at an acceptable level, this may be indicative of usability problems requiring improvement. The variable contributing to the negative perception was years of oncology experience. The study results supported the premise that whereas high usability would increase self-perceived job performance, low usability would not do so or would decrease job performance. That is, more experienced oncology nurses are less likely to use the SmartPhrase tool. This result is useful and supports findings on fitting technology the workflow task in literature. For a tool to increase job performance, the tool must be highly useful.

system fit healthcare providers' workflow and display patient information clearly. A quantitative descriptive, non-experimental study was conducted to evaluate the usability of a new SmartPhrase interface tool embedded into the Epic EHR system.
Twenty-three telephone triage nurses in a multi-site cancer center, located within the largest healthcare system in Rhode Island, were involved in designing and testing the interface tool.
The task of the SmartPhrase tool is to permit quick and easy insertion of pretexted symptom assessment cues or phrases into a patient's EHR. The SmartPhrase tool evolved as a basic unit of the Epic EHR in the Lifespan system, specifically as an electronic documentation feature that permits nurses to use pre-formatted text, statements, or structure note templates for cuing documentation. The specific SmartPhrase tool was developed in the context of a larger parent study to improve standardized telephone-triage symptom assessment for cancer patients. Generalized Linear Mixed Model was used to evaluate the usability of an EHR SmartPhrase tool to (1) determine the relationship between telephone triage nurses' years of experience (in nursing, telephone triage, and oncology) and their perception of the usability of the SmartPhrase tool; (2) assess changes in self-perceived job performance six months following SmartPhrase implementation; and (3) determine the relationship between the telephone triage nurses' evaluation of the SmartPhrase tool usability and actual tool utilization.
The SmartPhrase tool, as evaluated by telephone triage nurses, was not perceived as particularly useful. Although the usability score was at an acceptable level, this may be indicative of usability problems requiring improvement. The variable contributing to the negative perception was years of oncology experience. The study results supported the premise that whereas high usability would increase self-perceived job performance, low usability would not do so or would decrease job performance. That is, more experienced oncology nurses are less likely to use the SmartPhrase tool. This result is useful and supports findings on fitting technology the workflow task in literature. For a tool to increase job performance, the tool must be highly useful.
ix  outcomes, quality of care, and nurse and patient satisfaction. The problems include the increasing complexity in utilization of EHR systems, and the need for design changes to keep pace with rapid advances taking place in technology. This is often accompanied by a lack of testing of the usability of these systems prior to implementation. Usability testing involves making computer software and systems easier to use and matching them more closely to user needs and requirements. The international standard, ISO 9241-11, defines usability as the extent to which a computer system can be used by specified users "to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use" (International Organization for Standardization [ISO], 1998, p. 3).

LIST OF TABLES
A continuous increase of EHR system complexity and the capacity of nurses to use its tools will not ensure the fundamental requirement of healthcare settings to deliver what its patients need. In Health Information Technology (HIT), usability has become a central issue for EHR systems. Previous studies have identified challenges with an EHR's design, resulting in poor utilization and less effective use of the system (DeLucia, Ott, & Palmieri, 2009). Poor utilization has become an ongoing concern to nurses and service users (American Medical Association [AMA], 2014). Also, with continued advancement of EHR systems, it can be difficult to ascertain whether nurses are satisfied with the current EHR system, and if not, where concerns exist (Arrowood et al., 2013). Further, a new EHR tool design might be incompatible with nursing practice and result in unanticipated errors, lead to over or under-documentation of services, or make it a useless tool (Palabindala, Pamarthy, & Jonnalagadda, 2016).
A growing body of literature highlights the importance of EHR systems as the standard of documentation in health care organizations (Steinfeld & Keyes, 2011).
Automation is an important component in the EHR system and plays a key role in standard of documentation. Automation offers multiple potential benefits, including the ability to pull in a pre-defined text, statements, or structure note templates for documentation. In EHR systems, these are known as smart phrases (Cryts, 2016). The smartphrase tool has evolved into an elemental unit of the EHR and is an easy way to incorporate a standard of care into the documentation system in order to make the patient's record complete (Lamba et al., 2016). This can be advantageous for healthcare providers who write similar notes or repeated statements. For example, this tool can be used for creating smart phrases for repeated clinical assessment or for establishing a policy which provides standard guidance and best practices. Overall, SmartPhrase tools play a vital role in the fulfillment of documentation, coordination, and standardization in EHR documentation. The end result is an improvement in quality, safety, and efficiency (Lamba et al., 2016). However, the development and implementation of new SmartPhrase tools is often a neglected area, and little attention is paid to evaluating the usability of these SmartPhrase tools before and after implementation in an EHR system.

Study as a Subset of a Larger Project
In consideration of this evidence, the investigator of this study joined a larger project, the aim of which was the development and implementation of a new SmartPhrase tool for telephone triage nurses in a multisite cancer center. The project's purpose was to assist nurses with assessing cancer symptoms over the phone and triaging cancer patients to the most appropriate level of care. This study's purpose was to contribute to the larger project by evaluating the usability of the new SmartPhrase tool for telephone triage nurses. This researcher worked directly with the Principle Investigator (PI) of the larger project, who is a clinical manager at one of the cancer center sites, to facilitate communication with 23 telephone triage nurses. Using Dufault's translating-research-into-practice model (Dufault et. al, 2010), 23 Lifespan Comprehensive Cancer Center clinical nurses were involved in translating evidencebased telephone triage nursing assessment SmartPhrases into standard nursing practice. The aim of the parent project was to demonstrate the SmarPhrase tool's effectiveness in improving patient satisfaction and nurse-sensitive safety and quality outcomes related to symptom management. Prior to integrating the SmartPhrase tool into the EHR system, 21 of the 23 triage nurses used Dufault's 6-step model in designing the tool. This preliminary work is described in Appendix A.
While integrating a SmartPhrase tool in the EHR system holds promise for improving symptom management tailored to the patient's individualized needs, evaluating the effectiveness and usability of SmartPhrase tools has been neglected (Tariq, Westbrook, Byrne, Robinson, & Baysari, 2017). Despite the significant need to develop clearly understandable, standardized, and timely SmartPhrase tools, studies have not examined usability, efficiency, ease of use, and design effectiveness of these unique tools for managing cancer patients' symptoms remotely over the telephone (Tariq et al, 2017). Similar to other studies in the field of telephone-health, originally the parent project did not plan to test usability or delineate barriers to optimal use of the SmartPhrase tool. Upon this researcher joining the parent project team, the need for examining usability became a significant focus. It was recognized that if the SmartPhrase tool was designed correctly, it can be beneficial in facilitating documentation requirements, improving the quality of assessments, enhancing EHR documentation, and, ultimately, improving patient outcomes.

Statement of the Problem
Due to the lack of literature on usability, there was a need to address usability problems, especially by its targeted users, and to close this gap in reference to use in a multi-site cancer center. The specific objective of this dissertation was to evaluate the usability of a new EHR SmartPhrase tool and to explore the relationship between usability of an EHR SmartPhrase tool and telephone triage nurses' perceived job performance as well as their utilization of the tool itself.
According to the Agency for Healthcare Research and Quality (AHRQ), EHR systems should significantly reduce errors and be effective in transforming the quality, safety, and efficiency of healthcare. Issues with usability and information design, however, can actually facilitate errors and decrease the efficiency gains made possible by HIT software (Johnson, Johnston, & Crowle, 2011). In practice, testing the usability of EHR systems is recognized as critical for identifying design features of EHR systems that result in poor utilization and ineffective use (Page & Schadler, 2014). However, studies on system testing often overlook functionality, technical requirements, software and security aspects of an EHR system. Thus usability or, more broadly, information design is ignored (Ong, 2016). In addition, healthcare organizations and professionals usually pay little attention to nurse contributions and perceptions when implementing high-performing healthcare delivery systems (Dubois, D'Amour, Pomey, Girard, & Brault, 2013). The aims of this study were to conduct a usability evaluation of a new EHR SmartPhrase tool to (1) determine the extent to which this tool is easy to use or user-friendly; and (2) to determine if it improves telephone triage nurses' self-perceived job performance and their utilization of this new SmartPhrase tool.

Justification for and Significance of the Study
In recent years, the need to streamline processes and to improve quality of healthcare has been met with an increase in the growth of EHR system use (Edwards, Moloney, Jacko, & Sainfort, 2008). Kim (2015) has argued that the EHR system plays a crucial role in the health care delivery system. The utilization of an EHR system changes the way nursing actions (i.e. nursing assessment, medication administration, communication, or documentation) can be performed. Use of an EHR offers potential benefits for health care providers and patients, such as timely access to clinical data, alerts to avoid medical error, care coordination, and improved billing and coding.
Electronic documentation in an EHR is a meaningful system of realizing these benefits (Murphy, 2017).
Ineffective communication errors are the most frequent cause of sentinel events in United State (U.S.) healthcare. Communicating and assessing risks of cancer treatment related symptoms is significantly burdensome for telephone triage nurses.
These nurses are the frontline contact for channeling patients to the most appropriate level of care, from self-management to the emergency room. Making over-the-phone assessments is different than face-to-face encounters (Purc-Stephenson & Thrasher, 2010;Tariq et al, 2017). Immediate access to user-friendly, real-time online information in the EHR (i.e. patient history, lab values, and predictors of toxicity-risk) during a telephone conversation poses a significant challenge for the nurse who is simultaneously trying to provide emotional support to the patients and their family caregivers. This is especially true in the case of managing cancer patients' symptoms remotely over the telephone while accessing their EHR.
User-friendly approaches, such as SmartPhrases, to improve symptom management through cuing nurses' telephone-triage assessments are not widely used in practice (Teriq et al. 2016). For example, at this project site, telephone nurses usually bypass the EHR assessment tool and free text their symptom management calls. Such workarounds can dilute efforts to improve patient safety. Toggling backand-forth to multiple computer screens to retrieve data while maintaining patient rapport in real time may result in missed communication. This can be especially dangerous to vulnerable cancer patients (often with multiple co-morbidities), and often results in omissions of relevant patient care and leads to dissatisfaction of patients, families, and nurses. There is a significant need to develop and test user-friendly, clearly understandable, standardized and timely SmartPhrase tools if they are to be widely disseminated and deemed reliable. Integrating a SmartPhrase telephone EHRinterface holds much promise for improving symptom management.
Usability evaluations are not commonly performed, and those done focus more on adoption and less on usability (Page & Schadler, 2014). Zahabi and colleagues (2015) emphasized that the usability of EHR systems is a critical paradigm not adequately researched or tested. Zhang and Walji (2011) noted, in a usability lab at the National Center for Cognitive Informatics and Decision Making in Healthcare, that usability is a human performance issue. The way EHR's structure information, present patient information, process data, and generate clinical reminders (e.g. alert notification, popups message) too often detracts from healthcare provider's time with a patient and has a direct effect on clinical decision-making (Edwards et al., 2008).
The AHRQ considers usability as one of four current HIT priorities in the US.
To ensure the EHR system is designed to optimize usability, a healthcare organization needs to "test, test, and then test some more" the usability of EHR systems (AHRQ, 2013). Bowman (2013) indicated that EHRs systems are complex, and the usability evaluation of these systems is crucial to ensure safety and to enable clinical staff to focus on their patients rather than the technology. contributions.

Theoretical Framework
The theoretical framework for this study is the Task-Technology Fit (TTF) model by Goodhue and Thompson (1995). It was developed to describe, explain, and predict user performance and provide understanding of relationships between technology and user evaluation. The general model is based on the outcomes of user evaluations, which are assessments of various characteristics of an information system as perceived by the user. The TTF model forms the base for evaluating the SmarthPhrase tool for usability and potential impact on nurses' job performance.
In this study, 23 telephone triage nurses in a four-site cancer center within the largest healthcare system in Rhode Island evaluated the usability of a new EHR SmartPhrase interface tool. A survey approach using self-reporting instruments, and perceptions of the usability of the SmartPhrase tool from the telephone triage nurses' viewpoint were used to determine if there was a relationship between usability and its perceived impact on their nursing care performance.

REVIEW OF LITERATURE
This chapter presents a summary of relevant literature on usability and nursing practices regarding use of a SmartPhrase tool embedded in EHR systems, especially in the context of telehealth and, more specifically, in remote symptom assessment by telephone triage nurses. The following sections describe the literature search strategy, the concept of usability in the context of the EHR, and usability principles in an EHR system. The role of the telephone triage nurse is described followed by the advantages offered by use of the SmartPhrase tool in documentation. Lastly, the methods of usability evaluation are discussed.

Search Strategy Key Terms and Databases
The terms and keywords used in the literature search were usability, usability principles, usability attributes, electronic health record, electronic medical record, SmartPhrase tool, electronic documentation tool, telephone triage nurse, and telehealth nursing. Because the keyword "usability" was too broad, the additional keywords listed were used to retrieve articles that assisted in narrowing the search to relevant articles. The inclusion criteria specified that the articles a) be in English language only, b) include disciplines of nursing, medicine, and computer science, and c) consist of studies in which usability was applied in EHR systems.
The Cumulative Index of Nursing and Allied Health Literature (CINAHL) with full text, MEDLINE, Web of Science, and PubMed databases were searched for relevant articles using the keywords identified. The articles for review were expanded through exploring the references of the selected articles to identify additional research studies.

Results of the Literature Research
The literature search resulted in the retrieval of a total of 319 articles regarding usability in terms of EHRs and nursing practice. The researcher read the abstracts and identified the most relevant articles and narrowed the search to 25 articles that met the criteria for inclusion. Full-text versions were obtained. The majority of the full-text articles were found on EBSCOhost, while others were accessed from Google Scholar and University of Rhode Island (URI) Interlibrary Loan service.

Background of Usability Concept
The origins of usability are grounded in engineering, the social sciences, and computer science. Usability engineering is a discipline that combines computer science with behavioral aspects of interactive systems and first emerged during World War II, when the United States (U.S.) government began studying the ways in which soldiers interacted with machines. The goal was to design simpler, safer equipment that could save lives and help win the war. By the 1950s, scientists had begun to employ similar usability testing on civilian products, such as the telephone and refrigerator, to make them easier to use. The early definitions of usability meant ease of use (Heradio, Fernández-Amorós, Cabrerizo, & Herrera-Viedma, 2012). Shackel (1991) attempted a formal definition by stating the usability of a system could be defined as "the capability in human functional terms to be used easily and effectively by the specified range of users, given specified training and user support, to fulfill the specified range of tasks, within the specified range of environmental scenarios" (p.

24).
The American Heritage Dictionary of the English Language defines usability as "fit for use" (Morris, 1970). According to the Usability Professionals Association, usability is the extent to which software or hardware is easy to use and is a good fit for users (Soegaard, 2018). In the literature, a widely used definition is from the International Organization for Standardization (ISO 9241, 1998), which defines usability in terms of users effectively, efficiently, and satisfactorily achieving set goals in a specified context of use. The three attributes efficiency, effectiveness, and satisfaction often are used to describe the outcome of usability and as such, are inherent in the definition. Another widely cited definition is by Nielsen (1994), who defines usability in terms of the following attributes: learnability (pertaining to the ease of learning use of a tool), efficiency, memorability, errors, and satisfaction. A search of the literature revealed that usability stands for more than just "ease of use".
From the perspective of a software developer, usability has multifarious attributes.
Regardless, all definitions emphasize the relationship between usability and context of use, wherein the level of usability achieved depends on its use in specific situations.

Definition of Usability in EHR Context
The term "usability" is frequently discussed in the computer science discipline and is used in relation to any computer program that is employed to accomplish a task.
One example of this is seen in healthcare information technology (IT), of which the EHR system is the center of computerized clinical systems. The Healthcare Information and Management Systems Society defines EHR as an electronic record of patient health information that gives a longitudinal view of the patient's medical encounters in a care delivery setting (Belden, Grayson, & Barnes, 2009). Generally, the recorded information includes patient demographics, progress notes, medical problems, medications, vital signs, past medical history, immunizations, laboratory data, and radiology reports. The EHR aids clinicians by automating data for access, which often includes evidence-based decision support, quality management, and outcomes. This definition is more inclusive than some, as the definition of the EHR has varied over time due to the complexity of summarizing the numerous and diverse inputs.
Different scholars in the health disciplines have devised definitions of usability based on their perspectives on informed research and experience. Their general definition of usability is that an EHR system facilitates the achievement of health care goals within a clinical setting. The retrieval of information should be in such a way as to improve the quality of healthcare (Iakovidis, 1998).
The National Institute of Standards and Technology (NIST) provide a definition of EHR usability as a way in which professionals can efficiently use the system to accomplish tasks, given the context of use of a specific product (Schumacher & Lowry, 2010). The difference between this definition and the ones provided by other authors is that it emphasizes meaningful use and widespread adoption as the major tenets that define usability. An electronic record must fit the specific clinical setting of a healthcare institution. In the context of the definition provided by NIST, an EHR must facilitate team collaboration among healthcare workers, reduce medical errors, improve efficiency, and have a positive impact on the cognitive load of health care clinicians.
In an EHR usability lab at the National Center for Cognitive Informatics and Decision Making in Healthcare (NCCIDM), Zhang and Walji (2011) presented a unified framework of task, user, representation, and function (TURF). The researchers considered usability as a human performance issue. They defined usability through the user's perspective as to how the users find the system useful, usable, and satisfying for accomplishing work domain goals through task performance, preferably in sequence.
The TURF framework, as postulated by NCCIDM, is based on the fact that an EHR system should satisfy the needs of the users that provide medical care. If nurses, clinicians, and other medical practitioners do not derive the intended benefit of adopting EHR technologies, then the NCCIDM considers such a system is not usable and is unsatisfactory.
Key features that the Agency for Healthcare Research and Quality (AHRQ) considers vital for an EHR system to be usable are in concurrence with those of Scarlat (2012), who acknowledges aspects of the cognitive load of clinicians. When judging the level of usability of an EHR, AHRQ considers direct applicability to be the rule of thumb for high quality human-computer interaction. Essentially, the usability of an EHR system accounts for data density, data link/ratio, time series and small multiples, and missing data, and it also has efficient icons and navigation apparatus . Scarlat (2012) defines usability in terms of an EHR as easing the cognitive load of health care clinicians. Usability provides a measure of satisfaction of different stakeholders using a healthcare institution's EHR system. According to Scarlat, nurses and other clinicians perform their duties in an environment that has potential distractions. These distractions can compete with the clinicians' attention. That is, the core aspects of the cognitive load affect the quality of health care in many instances.
Thus usability, using Scarlat's perspective, is the ability of an EHR system to support the cognitive processes of its users (e.g. nurses, physicians, nurse practitioners [APRNs], pharmacists, and physician assistants [PAs]) .

Usability Attributes in EHR Systems
One of the key, and perhaps the most important feature of usability of an EHR system, is the ease of use. The Standard ISO 9241 (1998) defined usability as a set of attributes that focus on the effort needed for use, and on the individual assessment of such use, by a stated or implied set of users. It further describes usability as the extent users can attain their goals with some degree of efficiency. An expansion of the ISO definition defines the key attributes in relation to a particular product. These attributes are described as follows: Effectiveness. This denotes the accuracy and pace at which a user attains a selected goal. It is assessed by weighing whether the user objective and aims are met and whether the system works correctly (ISO 9241, 1998). The nature of user assistance inherent in the system has a profound effect on effectiveness.
The effectiveness of the EHR system depends on the presentation of choices in a way that is understandable to nurses and other health care providers.
Efficiency. Efficiency denotes the total resources (human and non-human) expended to accomplish a particular task (ISO 9241, 1998). It is aligned with accuracy, completeness, and speed of the system. Jee and Kim (2013) considered usability as an aspect of efficiency where users meet their clinical goals within the shortest time possible and with the least amount of mental effort. Therefore, more mental energy can be used in providing services to patients.
Satisfaction. This refers to the comfort and positive attitudes of the work system in relation to its users (ISO 9241, 1998). According to Peikari, Zakaria, Yasin, Shah, and Elhissi (2013), an EHR system should be capable of meeting subjective perceptions of the user by means of its features. An EHR system that satisfies the subjective perceptions of users is likely to have a positive impact on its general likeability and increases the likelihood that the users will consider it a vital aspect of service delivery (Doll, Xia, & Torkzadeh, 1994). A health institution should survey users as instruments are being developed.

Engaging.
A system is termed engaging if the user is satisfied and pleased when using it. The design of the system is the most vital element that determines the degree of engagement (ISO 9241, 1998).
Error tolerant. The number of errors should be reduced as much as possible.
However, because programs are developed by humans, no system is perfect (ISO 9241, 1998). Nonetheless, an error tolerant system should be designed to detect user errors and, if the interface is to be useful, permit the user to correct such errors. Examples of measures of system error are the recovery rate of error and the frequency of user errors of an EHR system (Zhang &Walji, 2011).

Easy to learn.
A system that is easy to learn enables users to build on their skills and knowledge with ease (ISO 9241, 1998). An interface that is easy to learn allows users to build on their knowledge, and, as such, access a new functionality, change the workflow, or explore options. Some of these changes may be dictated by factors in the environment (Charlton & O'Brien, 2002). A system that is easy to learn facilitates learning the system (Zhang & Walji, 2011). A novice user, therefore, can more rapidly learn how to use the EHR's interface. The interface should have exploratory features, such as "undo" and "cancel" functions that allow the user to correct or make changes (Middleton et al., 2013).
Usefulness. Usefulness is measured in terms of the percentage of the system's domain functions (e.g. terminology, hierarchy of items, feature descriptions, and icon usage) that users find relevant. According to Zhang and Walji (2011), an EHR system must have domain features that are relevant to the essential functions of a healthcare institution. For example, a pediatric center's EHR system should possess domain features that are relevant to the provision of pediatric care (Charlton & O'Brien, 2002). Viitanen, Kuusisto, and Nykänen (2011) state usability of an electronic nursing record system should have the following attributes: 1) Fluency of reporting practices in terms of efficiency of documentation, simplicity of the system, and ease of use; 2) Accuracy of documentation, including a system's support for error recovery; 3) Learnability, or the intuitive use of the system; 4) Support for nurses' work, including exchange of information and the manner of representation (content and layout); and 5) Support for collaborative care among healthcare professionals, including accessibility and readability of documented information, information exchange, and the way that data is presented relevant to professional needs. Johnson et al. (2011) narrows usability to features relatable to a primary care setting. According to the authors, an EHR system that works in a primary care setting is likely to be effective in many different healthcare environments. The attributes of a system for primary care include the following: Usable by diverse users. Users of an EHR system in primary care settings could include nurses, APRNs, physicians, PAs, administrative staff, and office staff. The usability of an EHR system is, therefore, determined by what users in primary care settings are able to collect, input, select, and interpret the information retrieved from it. Thus, an EHR must accommodate the working models of different practitioners in healthcare settings for it to be considered efficient. Johnson et al. (2011) consider primary care settings as the basic benchmark for assessing the usability of an EHR system.

Accommodate varied encounters and patients.
In the context of a primary care setting, Johnson and colleagues (2011) define EHR usability as one that has retrievable information in various areas concerning undifferentiated symptoms as well as preventive, acute, and chronic care.
Facilitate the performance of complex tasks. Johnson et al. (2011) consider usability as directly related to the complex tasks performed in primary health care settings, such as obtaining information about past data points, carrying out lab tests to determine future data points, obtaining personalized medical evidence about a particular condition, and reviewing the cost of formulary coverage for different medication options. The authors recognize usability as the ability to make complex tasks simple so that the quality of care improves.
Conceivably, making tasks simple contributes to reduced training costs and limits user risk.
Allow healthcare users to share workflows. Primary care settings put significant pressure on clinicians. This is due to the multiple demands which require their attention. Thus, Johnson et al. (2011) consider usability as the ability of an EHR system to help clinicians and other users to meet the demands of their patients with effectiveness and efficiency in order to provide quality of care. The system can only enable clinicians to meet demands by making it possible for them to complete tasks accurately and within the shortest time possible.
Simplify tasks in high-pressure and interruptive healthcare settings. Johnson et al. (2011) contend that an EHR system should actually be risk adverse as well as intolerant of errors that are likely to compromise the quality of care. Health care workers in primary care settings are usually under intense pressure, and they are likely to rely on the accuracy and effectiveness of an EHR system. Thus, rather than allow novice workers to learn how to retrieve information on a trial and error basis, a better alternative is to train potential users to accurately use an EHR system. Table 1 shows usability attributes in health IT.

Table 1. Usability Attributes in Health Information Technology. Effectiveness
The accuracy and pace at which a user attains a selected goal. It is assessed by weighing whether the user objective and aims are met and whether the system works correctly (ISO 9241, 1998).

Efficiency
The total resources (human and non-human) expended to accomplish a particular task (ISO 9241, 1998)

Satisfaction
The comfort and positive attitudes of the work system in relation to its users (ISO 9241, 1998)

Engaging
The user is satisfied and pleased when using it. The design of the system is the most vital element that determines the degree of engagement (ISO 9241, 1998).

Error tolerant
The number of errors should be reduced as much as possible. However, because programs are developed by humans, no system is perfect (ISO 9241, 1998).

Easy to learn
A system that is easy to learn enables users to build on their skills and knowledge with ease (ISO 9241, 1998).

Usefulness
Usefulness is measured in terms of the percentage of the system's domain functions that users find relevant (ISO 9241, 1998). Usable by diverse users The usability of an EHR system is determined by what users are able to collect, input, select, and then interpret information retrieved from it .

Accommodate varied encounters and patients
EHR usability as one that has retrievable information in various areas concerning preventive care, acute care, chronic care, and undifferentiated symptoms . Facilitate the performance of complex tasks.
Usability as the ability to make complex tasks simple so that the quality of care improves . Allow healthcare users to share workflows Usability as the ability of an EHR system to help clinicians and other users to meet the demands of patients with effectiveness and efficiency that helps them provide quality of care .

Simplify tasks in highpressure and healthcare settings
Usability as the ability of an EHR system to be risk adverse as well as intolerant of errors that are likely to compromise the quality of care .

Usability in Relation to the EHR as it Pertains to Nursing
The use of informatics is seen in a multitude of processes within the clinical setting. Kennedy and Hussey (2015) defined healthcare informatics as an interdisciplinary field of health-care science, computer science, information science, and cognitive science to assist in the management of healthcare information. Nursing informatics is a subset of informatics, specific to the field and role of the nurse in the healthcare setting. The American Nurses Association (ANA) identified nursing informatics as a specialty that integrates nursing, computer science, and information science to manage and communicate data, information, and knowledge in nursing practice (American Nurses Association, 2001). Healthcare and nursing informatics are fundamentally changing the clinical practice environment and the way health information is documented, stored, viewed, retrieved, shared, managed, and consumed (Rojas & Seckman, 2014). Increased implementation of EHR systems has a considerable impact on nursing (Rojas & Seckman, 2014). For instance, an EHR system can reduce, even prevent, medical errors, improve patient safety, and support better patient outcomes. In 2017, the National Coordinator for Health Information Technology (ONC) for the U.S. Department of Health and Human Services (HHS), pointed out the impact of informatics on EHR systems and subsequently on care.
Essentially, informatics permitted EHR systems to improve patient care, increase patient participation, and improve care coordination Additionally, EHR systems impact diagnostics and patient outcomes, practice efficiencies, and cost savings (ONC Organization, 2017).
Although EHR systems are tools used for improving the quality, safety, and coordination of care, Rojas and Seckman (2014) found that nurses and other clinicians have traditionally been passive users of EHR technology. For nurses, earlier EHR systems were originally intended for finance, laboratory, or other ancillary functions and they did not support nursing practice at the point of care. The current EHR systems such as Epic and Cerner attempted to facilitate nursing practice at the point of care. In today's reality, the complexity of their designs necessitates the development of user-friendly tools such as SmartPhrases to assist clinicians. To redefine the reality of usage, nurses must first understand the significance of usability. Usability increases the ease of use in EHR systems (Staggers & Troseth, 2010). According to DuLong (2008), nurses must educate themselves on usability and key clinical application design principles. This education, along with strong advocacy from nursing professionals, determines how well informatics and subsequent EHR functions are integrated into day-to-day nursing practice. Many nurses have learned to expect that some things just do not work in an EHR system. In the domain of HIT, usability of an EHR system is guided by the need for the system to be used effectively. This requires an evaluation of usability before and after implementation. According to a Telmediq team (2017), the most immediate issue of an EHR system has been to decrease the time required for documentation and order entry. Chandrasekaran, Anand, Ward, Sharma, and Moffatt-Bruce (2017) found that usability is about getting the right information in the easiest way. Therefore, when an EHR system has high usability, health care staff will be able to quickly and safely access pertinent information about their patient(s) anywhere within the health institution (Lopez & Fahey, 2018).
In healthcare organizations today, the use of EHR systems are likely to be mandatory and part of the standard of care (The Joint Commission, 2018). Currently, there is a gap in the literature on studies evaluating EHR system use. The ONC Organization (2017) found that there is an abundance of information about the "why" of an EHR system use instead of "how" EHR systems impact heath care staff performance. While much certainty exists about the benefits of EHR systems, (as in allowing health care staff to provide comprehensive, quick, safe, and evidence-based care to patients) (ONC Organization, 2017), previously published studies have been limited to investigating the use of an EHR system for routine documentation of history and physical examination findings, retrieval of laboratory results, and such tasks (Savoy et. al, 2018). The how of use focuses on whether or not nurses accomplish their tasks and work effectively.
In a review article by Zahabi, Kaber, & Swangnetr (2015), they found that EHR system usability evaluation is a phenomena that has not been adequately researched. Studies of EHR usability issues mostly compare paper-based systems to electronic based systems. Few studies include evaluation and comparisons among multiple EHR designs in order to identify the advantages of one system over another.
In addition, little attention has been given to usability evaluation of an EHR after implementation, especially in relation to validating EHR usability from the nurse's perspective. This type of evaluation could help to identify critical issues in EHR systems, such as patient safety issues (e.g. medical errors). Cresswell and colleagues (2013) identified causes of clinical decision support (CDS) malfunctions. One of the most common causes was a defect in the EHR software. The authors found an error that caused the EHR to function other than as designed or documented, and this error led to a CDS malfunction. There are few studies that have tested an EHR interface in terms of the principles of usability. Harrington, Kennerly, and Johnson (2011) identified a gap in the published research in their review of EHR usability. They found that descriptive or qualitative analysis methods had been primarily used, while there is very little published research using quantitative methods. They suggested more quantitative research is needed particularly with larger and more representative samples.
The next section of this literature review addresses usability in the context of the Practice Domain, as identified by Kim's domains typology used to generate nursing knowledge (Kim, 2010;.

Nursing Practice Domain and EHR Systems Overview
Kim (2010) identifies a typology of four domains as a structure for organizing the content of nursing knowledge. This typology categorizes nursing phenomena as either client, client-nurse, practice, or environment. These four domains are used to identify and generate knowledge of phenomena, concepts and theoretical frameworks from a nursing perspective. The ultimate purpose of this organizing scheme is to systematize classes of phenomena, concepts and theories thereby identifying what is developed in the science of nursing and what knowledge needs to be generated.

Definition of Nursing Practice Domain
Nursing practice is often used interchangeable with "clinical practice", "nursing acts", "nursing skills, "nursing work", or simply "nursing". The domain of practice, as conceptualized by Kim (2015), includes phenomena specifically related to the nurse who is engaged in delivering nursing care. It includes what nurses do and experience on behalf of clients. Kim (2010) offers a generic definition of nursing practice that includes activities that are goal-directed, scientific, and deliberate.
Nursing practice is action-oriented, fulfills societal responsibilities, and provides service for specific healthcare needs. Services are coordinated for and with clients, and practice involves human-to-human engagement and technological problem solving. In general, Kim (2015) views nursing practice as acceptable when activities are toward the good of the client.

EHR System Fits in the Nursing Practice Domain
From a nursing perspective, Kim (2015) describes the practice domain as what and how nurses carry out and perform nursing actions. Explicitly, EHR is changing the way nursing actions can be performed. Nurses use the EHR as their primary tool to document, store, synthesize, view, communicate, consume, retrieve, share, and/or manage patient health information. Usability of the EHR denotes the ease with which nurses can accomplish a task accurately and efficiently. It also offers a solution to problems important to healthcare institutions, such as reducing medical errors and redundancy as well as supporting nurses to perform tasks quickly, efficiency, and with a minimum of cognitive load. The intent is that the high level of usability of an EHR can improve quality patient care outcomes and increase nurses' performance. Kim (2015) specified two human processes, deliberation and enactment, as a way of organizing the practice domain. Deliberation is concerned with using cognitive processes to develop a program of action involving what the nurses should do, or need to do, in anticipation of actual delivery of nursing care. Deliberation means making choices in practice. Nurses need to be aware of and take into consideration the consequences of their actions. Examples of deliberation include clinical decision making, care planning, information processing, clinical judgment, and diagnosing. Kim (2015) proposed nursing practice as the doing and acting that occurs as nurses are engaged in actions. The process of enactment is the phase in which the nurse actually performs nursing activities. From a nursing perspective, enactment occurs as actions are carried out and performed within the arena of human services practice. In this process, nursing actions are bound by time, space, and physical locality in the context of nursing care. Examples of enactment include caring, nursing communication, nursing documentation, and tailoring nursing actions.
In the deliberation process, the interaction of the nurse and client, nursing goals, and nursing care practices lead to decision-making about diagnosis and subsequent intervention(s). The diagnosis is then documented, most commonly into the patient's record in an EHR system. The documentation is thus part of an enactment during which nurses' record decisions and actions. Thus, usability can be viewed as an opportunity to transform nursing actions in ways that increase their utility. Kim (2015) specified that the use of nursing tools (documentation, information management, and care management) permit nurses to fulfill their professional role responsibilities in day-to-day practice. Accordingly, electronic recording systems have become a part of health care within which nurses provide documentation. Kim (2015) suggests two levels of information management. One is at the client level for collecting, recording, storing, and communicating data regarding patients and clinical work performed for patients. The second is at the decision support level. EHR systems are designed to support clinical decision making in nursing practice, and nurses at this level need to provide safe and effective patient care. The American Association of Colleges of Nursing (2008) indicated that knowledge and skills in information management are essential to delivering quality patient care. In addition to documentation and information management, care management as a nursing tool ensures continuity of care and prevention of fragmentation of services while promoting the best possible patient outcomes (Kim, 2015). The major goals of these nursing tools are to 1.) provide integrated, coordinated nursing care to patients, 2.) mobilize the best health care possible for patients, and 3.) ensure a high level of quality of care (Kim, 2015). Belden, Grayson, and Barnes (2009) state that the usability of EHR will reduce error and redundancy, provide efficacy, and maintain confidentiality. In short, what is desired in an EHR system is that it can improve the nurse's performance resulting in higher quality of patient care.

Overview of SmartPhrase Tool
The generalizability of published research on EHR systems is problematic.
There is little consensus on usability evaluation and information about the benefits of using EHR systems. Whereas a study might compare the benefits of EHR systems in terms of clinical, organizational, and societal outcomes (Menachemi & Collum, 2011), there is little information as to the effects of electronic documentation tools, or, in this case, usability of the SmartPhrase tool. The SmartPhrase tool (also called Smart Form, Smart Set, Smart List, or Smart Text) is a customizable documentation tool that allows the use of pre-designed smart phrases that rapidly provides assistance, as in cuing and standardizing symptom assessments (American Health Information Management Association [AHIMA], 2013). The National Institute of Standards and Technology defined smart phrases as structured text, which is an auto-build documentation feature (Schumacher & Lowry, 2010). Schnipper et al. (2008) defined the SmartPhrase tool as primarily a documentation tool integrated within an EHR system that permits adding, editing, and deleting patient information or structured clinical information, such as medical conditions, problems, diseases, medications, allergies, vital signs, and laboratory values. In addition, the SmartPhrase tool also organizes clinical data in a focused manner to facilitate decision-making for clinicians.

Benefits of SmartPhrase Tool in EHR System Documentation
The use of a SmartPhrase tool saves time in documentation as well as provides a method for achieving standardized assessment (Perez, 2014). Schnipper et al. (2008) used a SmartPhrase tool known as Smart Forms in an EHR to improve disease management and found that the tool had potential to improve the care of patients with both acute and chronic conditions. Essentially, the tool is a clinical workflow tool that helps organize data for specific problems, facilitates effective and efficient data capture, and serves as a clinical decision support system that is integrated in a single environment. The tool has evolved into an elemental unit of the EHR and incorporates a standard of care in order to make the patient's record accurate and comprehensive (Clements, 2018). Perez (2014) found that the use of smart phrases (pre-structured text) as an automatic-build documentation feature aided healthcare providers in two ways. First, it can insert information necessary for devising a plan of care for the treatment of the patient's condition and secondly, can augment the management of patient information necessary for clinical decision making. Thaker et al. (2016) (Perez, 2014). In addition, use of the tool had the potential to reduce costs and immediate workload of healthcare providers (Clements, 2018).
EHR systems are increasingly sought as the standard of documentation in health care organizations, and there is evidence that electronic documentation tools play a crucial role in standardizing EHR documentation (Steinfeld & Keyes, 2011).
The SmartPhrase tool offers multiple benefits, including the ability to insert predefined text or structure note templates for documentation. Ideally, the SmartPhrase tool could replace the nurses' usual note-writing tools, including standard free text within the EHR. For example, the tool can be used for creating smart phrases for repeated clinical assessments and for establishing a policy that provides standard guidance and best practices. However, despite the tool's apparent usefulness, little published research has evaluated usability.

Barriers of SmartPhrase Tool in Telehealth Oncology Nursing
Assessing risks of cancer treatment-related symptoms is significantly burdensome for telephone-triage nurses, who serve as the frontline contact for prioritizing patients for the most appropriate level of care. Immediate access to userfriendly, real-time online information in the EHR (i.e. patient history, lab values, and predictors of toxicity-risk) during a telephone conversation poses significant challenge for the nurse. The use of high usability smart phrases in the EHR may help telephonetriage nurses expedite prioritizing while still providing emotional care and treatment.
One caution is the use of such tools could, if used inappropriately, result in either the over or under-documentation of services. This could lead to unanticipated errors and render the tool useless (Clements, 2018). However, Hurria et al. (2016) stated that the SmartPhrase tool plays a vital role in ensuring the completeness and accuracy of documentation, coordination, and standardization in EHR documentation and can improve quality, safety, and efficiency of clinical data integrity and management.
Although the development and implementation of smart phrases is often a neglected area, published studies support the view that the effectiveness of smart phrases have been less than expected (Clements, 2018). This could hold true in assessing symptoms of cancer patients. Schnipper et al. (2008) has shown that the main barriers to use of smart phrases included lack of workflow integration, software usability issues, and relevance of the content to the patient.

Research Gap in Literature
A SmartPhrase tool integrated within an EHR holds the promise of improving healthcare quality. However, few researchers have evaluated the use and the usability of the SmartPhrase tool in telehealth nursing. One aim of this study is to assist telephone-triage nurses with assessing symptoms of cancer patients over the phone and then triage the patients to the appropriate level of care. Telephone-triage nurses working in this dynamic area are required to maintain accurate and concise documentation of all interventions they propose in order to meet the compliance of standards outlined by the medical staff and the institution's financial department.
Without this accurate and timely record, nurses could place themselves at risk for financial, legal, and medical penalties (Clements, 2018). Unless the SmartPhrase tool is used appropriately, the integrity of data may be questioned, and the information could be deemed inaccurate or perceived as a fraudulent activity (AHIMA, 2013).
Arrowood and colleagues (2013) studied guidelines for EHR documentation to prevent fraud. The authors found that SmartPhrase tools have potential documentation practices that could create concerns regarding patient safety, quality of care, and compliance all of which may leave an organization vulnerable to patient safety errors and medical liability. Arrowood and colleagues (2013) discussed that a SmartPhrase tool may not exist for a specific problem or visit type. This issue can occur if the structure of the tool is not a good clinical fit and does not accurately reflect the patient's condition and the clinical services offered (Arrowood et. al, 2013). The automatic generation of sections or the completion of SmartPhrase templates in advance of a patient encounter can not only detract from quality care, it can also increase the clinician's exposure to liability claims and to accusation of fraud due to over-documentation that causes a higher level of service to be billed for than was actually performed (Dawson, 2017).
According to AHIMA (2013), health care staff must review and edit all default data to ensure that only patient-specific data is recorded, while all irrelevant data pulled in by the default SmartPhrase template is removed. For example, the SmartPhrase automatic generation of common negative findings within a review of systems for each body area or organ system may result in a higher level of service delivered, unless the health care staff documents any pertinent positive results and deletes the incorrect auto-generated entries (Arrowood et al., 2013). These unintentional practices may involve repeated billing and coding errors that over time may be considered fraudulent if patterns of continued practice are found upon external review (AHIMA, 2013). However, the leadership and management of the healthcare setting should determine system functionality, and system usability that potentially results in fraudulent entries into the EHR (Arrowood et al., 2013). Usability evaluation must be in place to ensure compliant nursing care when electronic documentation tools such as SmartPhrase tool within an EHR system are used to promote effective data management and documentation (AHIMA, 2013).

Definition of Telephone Triage Nurses
Telephone work is an increasingly important way of remotely managing the workloads of healthcare professionals. The field is known as telehealth nursing or telepathology. In recent years, there has been a growth in the use of telephone-triage services to reduce the immediate workload in the healthcare settings (Giesen et al., 2007). The majority of healthcare professionals that work in this field are registered nurses (RN). A telephone-triage RN uses the phone to help determine what type of care the patient will need. This is designed to help patients who are unable to get to a doctor's office or hospital to determine the level of care they may need. These nurses are trained to ask specific questions to help the patient decide if they need to seek emergency treatment, or make an appointment with a healthcare provider, or selfmanage their care at home (Campbell et al., 2013;Gallagher, Huddart, & Henderson, 1998).

Advantages and Challenges of Telephone Triage Nurses
Telephone-triage services have several advantages over traditional healthcare services. First, telephone-triage nurses help patients determine the level of care they may need and can assess the severity of the health problem(s), without the patient having to visit a clinic or emergency room (Campbell et al., 2013). This is especially helpful to patients that find it either difficult to get to a medical facility or pay for medical services. Telephone-triage nurses also help healthcare providers reduce their patient load by helping patients with minor health issues and, if emergency medical attention is not needed, can aid in reducing overcrowding and waiting time in emergency departments (ED) (McLean et al., 2013).
Telephone-triage nurses are often the frontline contact for assessing symptom severity and triaging patients to the most appropriate levels of care, ranging from selfmanagement at home or to the ED (Stacey, Macartney, Carley, & Harrison, 2013).
Evidence-based approaches to improve standardized telephone-triage symptom assessment are embedded in nursing practice, and standardized symptom assessment permits accurate documentation in the EHR system. Thus, telephone triage systems need to be highly reliable, sustainable, and have the ability to widely disseminate information to patients (i.e. reducing chemotherapy and radiation treatment toxicity risks) and enhance patient/family engagement and comfort.
Nurse-driven evidence-based algorithms have been used in home and primary care models to safely, effectively, and efficiently manage patient symptoms (Dufault & Willey-Lessne, 1999). However, they have limited use in telephone triage nursing (Flannery, Phillips, & Lyons, 2009). Limited attempts to interface telephone-triaging within the EHR show promise for oncology models. In this sense, telephone-triaging improves care transition, handoff communication, nurse-patient relationships, and patient or family education. It also aids in decreasing ED visits, delays in care, and helps to avoid hospital-acquired infections (Gleason, O'Neill, Goldschmitt, Horigan, & Moriarty, 2013;Waters et al., 2015).
Despite these advantages, significant challenges remain for implementing high quality, cost effective telephone triaging. This suggests that nurses may not apply empirical evidence about best telephone-triage practices. Four major barriers for telephone triage nurses include 1) lack of knowledge required for symptom treatment; 2) lack of confidence in the ability to assess, triage, and guide patients in self-care; 3) time management; and 4) discomfort in not recommending use of the ED (Gleason et al., 2013;Hawley, Loney, & Wiece, 2011;Maloney et al., 2013;Stacey et al., 2007;Stacey et al., 2015). Phone assessments differ from face-to-face encounters. Therefore, telephone-triage nurses need immediate access to user-friendly, real-time online resources embedded in the EHR (i.e. patient history, lab values, functional assessments, and predictors of toxicity risk) without having to simultaneously toggle multiple EHR computer screens. This can be especially challenging for the telephonetriage nurse who is also trying to provide over the phone emotional support to the patient at the same time (Purc-Stephenson & Thrasher, 2010;Tariq et. al., 2017).

Research Gap in the Literature
Whereas significant advances have been made in developing valid and reliable clinical assessment tools in computer and telephone-interface technology, these advances have not been widely used or empirically evaluated for their impact on improving symptom management for cancer patients. A multi-center study looking at the ability to quickly predict chemotherapy toxicity risk in older adults was conducted (Hurria et al., 2016). The evaluation of the usability of SmartPhrases and their impact on telephone triage nurses' ability to gain beneficial use when working with cancer patients was recognized as a first step towards reducing barriers and gaining efficiency of use. A Lifespan Health System Outpatient Oncology Report (7/1/2016 -9/30/2016) briefly noted that embedding a SmartPhrase tool in the organization's EHR system may predict hospitalization in nearly 70% of patients treated (Lifespan Rhode Island Hospital, 2017). However, validation of such predictions is not currently reported in literature.

Formative Usability versus Summative Usability
The two different types of usability testing, formative and summative, have different aims. Formative testing includes expert evaluation as to whether or not an interface is usable, whereas summative usability includes evaluation of end-user perception toward effectiveness, efficiency, and satisfaction of use. Table 2 summarizes differences between formative usability and summative usability.

Table 2. Summary of Differences between Formative and Summative Usability. Formative Usability Summative Usability Aim
Computer expert evaluation (Rosenbaum, 1989).

Number of participants
Nielsen (2017) estimated that 5 reviewers are adequate. Nielsen (2017) recommends 20 endusers. Methodology Qualitative. The researcher lets experts describe the process or what they are trying to do and give feedback (Travis, 2012).
Quantitative. The researcher main interest is the statistics of enduser's behavior (Travis, 2012).

Data analysis
Data analysis tends to be descriptive and non-parametric (qualitative) in nature (Travis, 2012).
Data analysis requires tests of significance and calculations such as time on task and measures of success rate (Travis, 2012).

Frequency and timing
Conducted on a given time frame (monthly) or at scheduled intervals. Requires the smallest investment of time and money (Bevan, 2008).
Requires fewer tests. Conducted on pilot or pre-release basis. Difficult to carry out after system release (Travis, 2012).
The advantages of a formative approach is that it can be used to identify usability problems and this can inform and improve the system design during the development process. Also, it allows feedback about problems of a given system interface design as it is being developed and pilot tested. A limitation is that it requires usability specialists or software developers to examine and judge each element of a user interface, and as such, a formative approach does not necessarily assess what a real user in actual time can or cannot do (Bevan, 2008).
An advantage of summative measures for usability of EHR system is that it can be used to obtain a more complete understanding of the nurses' needs, likes, and dislikes, as well as feedback on issues with a given system interface design (Schumacher & Jerch, 2012). This affects its implementation, adoption, and utilization. Bevan (2008) noted that summative evaluation measures should be used with an adequate sample of representative users in a real-world context.

Summative Usability Evaluation
This investigator's study used a summative approach to evaluate the usability of a SmartPhrase tool implemented in an EHR system. The end-users were telephone triage nurses in a multisite cancer center, where nurses in several hospitals participated in the study. The summative usability method permitted achieving the study's primary goal. The goal was to evaluate nurses' perception of the attributes of SmartPhrase tool usability. In nursing practice, performance difficulties and errors that occur when using EHRs are highly variable and contextual, so simply counting failures is insufficient to understand the usability of the system. Thus, a summative evaluation involves describing the usability principals of the EHR system as well as establishing a baseline to assess whether usability requirements were achieved. Folmer and Bosch (2004) stated that summative usability falls into two main categories. One is a usability testing approach, in which real users are required to work on typical tasks. Then the researcher examines the results to see how the user interface of a system supports the users in doing their tasks. Usability testing can be used throughout the product development cycle and can be conducted quickly and allow retesting to check the validity of solutions to fix any usability problems. However, it requires time and resources, including personnel trained in research design and statistics, to conduct usability testing (Tan & Gencel, 2009).
The second category is the usability inquiry approach, in which researchers are required to attain information about real users' perceptions and satisfaction (i.e. likes, dislikes, comfort, opinions, needs, and understanding of the system) (Folmer & Bosch, 2004). It can be used to evaluate the usability of an EHR that has been used consistently by the same nurses over a period of time. The researcher can sample the same user population (Tan & Gencel, 2009). The inquiry approach has the potential to allow usability comparisons across interface designs. Therefore, the usability inquiry approach was applied in this study.

Usability Inquiry Approach: User Administered Questionnaires
Various tools, techniques, and questionnaire response types of user-based evaluations employ different classifying and identifying strategies. Sauro and Lewis (2016) noted that a short questionnaire can be used to obtain a quick response of user responses, usually when they have just used a product for the first time. A longer questionnaire may be divided into more specific subscales to gain additional information. However, in relation to questions to measure usability, there is a concern of whether or not the questions measure what they intend to measure (i.e. validity) (Folmer & Bosch, 2004). The researcher can either develop a questionnaire that will be used with other evaluation methods or as a stand-alone questionnaire. For the latter, researchers focus on a numerical measure of the usability of a product that is independent of its relationship to any other evaluation method (Folmer & Bosch, 2004).
According to Folmer and Bosch (2004), a questionnaire designed to assess perceptions of usability can be administered in one of two different ways. A questionnaire can be administered following participation in a scenario-based usability test (post-task). The researcher asks the participant to perform a task on the computer and then administers a questionnaire immediately following the completion of a usability test task. Another way is the administration of a questionnaire at the completion of the test scenario (post-study), where the researcher focuses on the measurement of computer usability without asking the participant to complete a task or scenario (Folmer & Bosch, 2004).

Measuring Usability
Measuring usability has long been an issue in the engineering field. Reviewing the literature on measuring usability revealed that usability is a broad concept that is best measured by consideration of its individual aspects. For example, Good, Spine, Whiteside and George (1986) showed that determining the usability needs of a system or measuring whether or not the finished system fulfils those needs, cannot be done without measuring specific usability features or attributes. Nielsen and Levy (1994) reported that usability attributes can be measured by two methods (subjective preference measurements or objective performance measurements) depending on the reason for testing the usability of a system.
The System Usability Scale (SUS) developed by Brooke (1996) is an example of a short questionnaire. The questionnaire requires the subjective opinions of participants and has been widely used to evaluate usability in both commercial and research studies for over 30 years. It contains 10 questions that are scored on a 5-point Likert scale and are based on strength of agreement. The SUS items test for system usability effectiveness, efficiency, and satisfaction. The SUS is quick to administer, and responses are numerical, which permits statistical analysis. The scale consists of five positive and five negative statements and provides a single summated rating for the system being evaluated. Brooke (1996) states that this questionnaire can be used as a stand-alone evaluation instrument. The SUS is the most frequently used questionnaire for measuring the perceptions of usability and has become an industry standard and is cited in over 4000 publications. For example, Hodgson, Magrabi, and Coiera, (2018) conducted a usability study on ED personnel (n = 35) to determine the value of using speech recognition for clinical documentation tasks within an EHR system. Significant difference in SUS scores between EHR system use, with and without speech recognition, were observed. Similarly, a usability evaluation was employed to examine how user perceptions toward a telemedicine system changed over the course of everyday use (Lemon et al., 2018). The usability findings indicated that a temporary period of positive user perceptions occurs when new telemedicine systems are used in the first few months. Sauro and Lewis (2016) reported a reliability coefficient of 0.85. Another estimate using a larger sample (n = 2,324) reported a reliability coefficient of 0.91 (Bangor, Kortum, & Miller, 2009).
The Software Usability Measurement Inventory (SUMI) developed by Kirakowski (1996) consists of 50 statements on a three-point scale (Agree, Undecided, Disagree). The SUMI provides an overall usability rating as well as ratings on five subscales: efficiency, affect, helpfulness, control, and learnability. The subscales have 10 items each. Efficiency is a measurement of the users' perception on how the software helps them complete their work. Affect measures how well the users like the software. Helpfulness measures the degree to which the software is self-explanatory and the adequacy of help facilities and documentation. Control measures the extent to which the users feel like they are in control of the software. Learnability measures the speed and how users master the system or learn to use new features. A factor analysis conducted during the development and evaluation of the SUMI provided evidence of construct validity. The Cronbach's alpha as a measure of SUMI's internal consistency was 0.89 and the reliability of the SUMI Scales ranged from 0.71 to 0.92 for each of the questionnaire's five subscales (Kirakowski, 1996).

Measuring Nurse Performance
The Task Technology Fit (TTF) model, developed to evaluate individual user performance of an Information System (IS), is a significant user evaluation construct in understanding and predicting the utilization of a technology (Goodhue & Thompson, 1995). The model can be used to predict utilization and performance of nurses using EHR systems. It consists of eight factors. The eight factors include quality, locatability (ease of finding information), authorization, compatibility, ease of use/training, production timeliness, systems reliability, and relationship with users.
Each factor is measured using between two and ten questions for a total of 25 statements used to measure TTF. For each statement, respondents indicated, on a Likert scale of 1 to 7, whether they strongly disagreed or strongly agreed. The TTF has high reliability and discriminant validity and also exhibits strong predictive validity (Goodhue, 1998). Reliability of the TTF scales ranges from 0.89 to 0.99 for each of the questionnaire's constructs (Lee, Lee, & Kim, 2005). All the constructs have a construct reliability of 0.7 or greater (Tam & Oliveira, 2016).

Summary of Chapter Two
A literature review of the concept of usability, the SmartPhrase tool, and telephone triage nurse roles was conducted. This chapter focused on the usability evaluation of EHR systems, specifically that of a SmartPhrase tool in an EHR system. Generally speaking, little is known about how usability of EHR systems are positively related to nurse performance. More specifically, this review of the literature showed that there is a gap regarding the usability of the SmartPhrase tool, suggesting that nurses do not apply empirical evidence in the use of the SmartPhrase tool in telephone-triage practice.
The usability of EHRs fits well in the nursing practice domain. Kim (2010) organizes nursing knowledge into a typology of four domains that includes the Practice Domain. Within this domain, Kim recommends that nursing generates knowledge on how nurses act in their practice. Knowledge in this domain is aided in theorizing nursing practice in terms of EHR usability evaluation. Further, nursing tools provide documentation, information management, and care management, all which are essential aspects of the EHR system.
Usability is a complex concept, with multiple dimensions, elements, and attributes, that has not been defined homogeneously, either by researchers or professional organizations. The term "usability" refers to multiple sub concepts, such as execution time, performance, user satisfaction and ease of learning (Abran, Khelifi, Suryn, & Seffah, 2003). Scholars in the health disciplines have given definitions of usability based on their perspectives influenced by informed research and experience.
Healthcare has been slow to adopt usability features and principles, and the result is tools within EHRs that fit poorly into nurses' work. This, in turn, influences nurse performance and productivity.
The SmartPhrase tool is an EHR-based documentation feature that allows the use of templates and smart phrases to assist with documentation. The success of using the SmartPhrase tool in an EHR has spread from medicine to other healthcare disciplines, including nursing and rehabilitation services. However, in support of evaluating the usability of an EHR's interface to increase the fit between nursing duties and the EHR, it is necessary to have comprehensive knowledge about the attributes of an EHR system and to evaluate its usability in reference to the professional role of telephone triage nurses who deal with accessing and managing data relevant to cancer patients.

Background
This chapter describes the Task Technology Fit (TTF) model and the framework developed by Goodhue and Thompson (1995). The TTF framework can be used to describe, assess, and predict user performance of information technology (IT) systems and provide understanding of relationships between technology and user evaluation. For all purposes, user satisfaction is no longer the main reason why a system is adopted, the impact on performance may be a larger consideration (El-Gayar, Deokar, & Wills, 2010).
In the past decade, researchers evaluating specific IT constructs at the individual level have adopted TTF as a theoretical framework. The framework has been empirically tested and provides a strong diagnostic tool to evaluate whether IT meets user requirements (Cresswell, Bates, & Sheikh, 2013;El-Gayar, Deokar, & Wills, 2010). The TTF model aids in understanding concepts that are relevant to the issue of fitting technology to the tasks to be performed (Goodhue & Thompson, 1995). (2013) argue that TTF is an established theoretical framework in IT research that enables the investigation of issues of fit of technology to tasks as well as performance.

D'Ambra, Wilson and Akter
Components of an EHR, such as a SmartPhrase tool, is seen as a technology fitted to a task. Information about the usability of the SmartPhrase tool from the telephone-triage nurses' viewpoint and its application to the workflow is the type of usability information that can be gained using the TTF framework. The intent of the study is to test whether or not there is a relationship between usability and the user's actual use of the Smartphrase tool and to what extent usability may impact healthcare performance. The TTF model permits studying the relationship between IT and selfperceptions of individual performance. The TTF model explicitly predicts overall selfperceived performance and its potential impact. If it is indeed true that healthcare administrators/decision-makers and IT professionals usually pay little attention to nurse contributions and perceptions when implementing high-performing healthcare delivery systems (Dubois, D'Amour, Pomey, Girard, & Brault, 2013), then use of the TTF model to structure usability evaluation serves as a measure less of interest and more of value.

Task-Technology Fit Framework
In 1995, Goodhue and Thompson, both business theorists, introduced the TTF model. The TTF model reflects a logical-positivist perspective, which is characterized as traditional science and includes concepts such as correspondence truth theory and empiricism. The rationale of a logical-positivist position is based on hypotheses testing, where the intent is to describe, explain, and predict user performance to promote an understanding of relationships between technology and user evaluation.
Goodhue and Thompson's research efforts were deductively driven using quantitative methodology. They used the TTF model to develop tools to assess characteristics of an information system as perceived by the user.
The TTF model has its roots in contingency theory, a theory that argues specific situational factors effect direct relationships within organizations (Donaldson, 2001). Goodhue and Thompson (1995) developed the model to study the relationship between IT and individual performance for a best fit contingent upon task demands.
The authors proposed the TTF model as a user evaluation construct that is "defined within a theoretical perspective that can usefully link underlying systems to their relevant impacts" (p. 1827).
A significant focus of the TTF model has been to assess and explain the success and impact of IT on individual performance. A model premise is that IT uses, and performance benefits are attained when IT is well-suited to the tasks that must be performed.

Assumptions and Key Concepts of the TTF Framework
The TTF model is comprised of six key constructs, as shown in Figure 1. An overview of these concepts, as well as a review of the applicable theoretical and empirical approachs using this model, is provided in the following sections. According to Goodhue (1995), evaluation of information systems deals with the match between task requirements and individual abilities. Factors that determine the task-technology fit are the interactions between the individuals, the task, and the technology. Characteristics of these factors jointly effect TTF and represent the core construct of the model. Utilization and performance impact are dependent variables and are indicative of outcomes. The main independent variables are individual characteristics, task characteristics, and technology characteristics. Individuals can use technology to assist themselves in the performance of their tasks. However, their individual characteristics may play an important role in their technology choice.
According to Goodhue and Thompson (1995), these characteristics included prior experience in computer use, previous computer training, and personal motivation, which may affect how easily and well the user will utilize the technology.
The TTF framework permits measurement of the match among task requirements of the user, an individual's abilities, and the functionality of the system. Thus, the strongest link between IT and performance impact is due to the correspondence between task needs and system functionality (Goodhue, 1998).
Goodhue found that measures are higher when the task requirements of the individual and the functionality of the technology match, and measures are lower as tasks become more demanding or technologies offer less functionality to meet the task demands of the individual. Users are more likely to use technology to complete a task if the technology fits the task at hand, and, as such, TTF can be considered an antecedent to system utilization. The TTF model suggests that various kinds of fit should contribute to performance.
The general model is based on the outcomes of user evaluation, which are assessments of various characteristics of an information system as perceived by the user. Users generally rate the system on a continuum from positive to negative. If users give the system a positive evaluation, then the system is likely being used in such a way to improve their performance (Goodhue, 1995).
Several instruments for user evaluations have been reported, including the Bailey and Pearson User Information Satisfaction instrument (1983), the Davis Perceived Usefulness and Perceived Ease of Use instrument (1989), and Goodhue TTF instrument (1998). User evaluations of TTF must be linked to the characteristics of the system being evaluated to confidently apply the results for diagnostics or measures of success (Goodhue, 1995). In other words, Goodhue utilized deductive reasoning by which specific dimensions were identified and categorized into factors.
In addition, Goodhue (1995) proposed that evaluations of TTF will be affected by characteristics of the 1) information system and services, 2) task, 3) individual's skills and abilities, and 4) interaction among the task, the technology, and the individual.
Individual characteristics have been identified as influential in an individual's assessment of TTF. These are defined as the individual skills and abilities that an individual brings to bear on the task (Goodhue, 1995). Individuals who have more relevant experience and skills to perform a task would be expected to perform better on the task. Assessment of the task then should show if the technology used increases the performance. Goodhue and Thompson (1995) proposed that at any level of utilization, a high-test score leads to increased performance because the system more closely fits the task needs of the user. Individual performance is linked with the completion of tasks (Goodhue & Thompson, 1995). Higher individual performance implies improved efficiency and effectiveness and may result in higher quality.
Indeed, it has been noted that the TTF model is concerned with explanation and prediction of the utilization of IT.

The TTF Framework in Testing the Usability of EHR
In nursing practice, EHR systems with effective usability features have significant impact on task performance. Positive outcomes are seen when usability promotes patient care through nurses making sound and appropriate decisions and performing tasks effectively. This can ensure optimal quality of care in a cost-saving manner, where the performing or functioning of the EHR system's tasks will be in the best possible manner with the least waste of time and effort (Kilmon, Fagan, Pandey, & Belt, 2008). On the other hand, compromised or poor EHR system usability can have negative implications in a clinical setting. User error can potentially cause patient harm, and negative outcomes may culminate in an attenuation of EHR adoption (Kilmon et al., 2008). For an EHR user interface to be effective, a systematic evaluation of its usability about the fit between nursing performance and the technology is required. For a systematic evaluation to be relevant, it is necessary to focus on each professional role of nursing and its relationship to tool use in nursing practice.
Health care relies on an information intensive practice. To provide high-quality care services to patients, nurses need the ability to identify, access, interpret and integrate relevant data within the EHR system (Fossum, Ehnfors, Fruhling, & Ehrenberg, 2011). Schumacher and others found that health care workers face usability challenges caused by several factors. These may include inefficient workflows that fail to match clinical processes, confusing popup messages that can be ignored (and sometimes not), poorly designed screens overloaded with data disrupting potential critical issues, alert fatigue (both visual and audio), and frustration with too many clicks to perform common tasks (Schumacher & Jerch, 2012;Schumacher & Lowry, 2010). Thus, it is essential to identify and rectify these issues to improve usability of EHR interface and tool use. Taiwo, Awodele, and Kuyoro (2016) confirm that when usability is effective, it results in a reduction of medical errors, better clinical decision making, improved patient safety, and lower healthcare costs. For example, poorly designed screens can result in user frustration. This may cause the nurses to work around the problem, which in turn, could introduce medication errors. Dawson (2017) believes that one major reason to adopt an EHR system is to reduce medical errors. Unfortunately, EHR systems can result not only in new types of errors, but also in more errors. In this light, a cohort study was conducted during the two-year period between January 1, 2012 and December 31, 2014 by Graber and colleagues (2015). The aim was to study the role that IT played in malpractice claims.
Data were obtained from an insurance database containing more than 300,000 cases.
An important observation from this study was that EHR-related errors were classified as medical (31%), diagnostic (28%), and due to complications from treatment (31%).
For example, in one case, an infant died from a drug overdose caused by a transcription error that arose when a handwritten order was entered incorrectly into the computer. In another case, critical ultrasound results were routed to an incorrect tab in the EHR, causing a yearlong delay in treatment for a cancer patient. A patient's death, from a subarachnoid hemorrhage, resulted when a physician was unable to access critical information from the ED notes. The information would have changed the treatment plan.
Another source of error can be system-related. Bowman (2013) pointed out that some EHR systems will auto-complete certain fields in the record based on specific patient characteristics or on other data entries, even if the default data does not apply to the actual patient. These examples highlight that while efficient use of an EHR aids in promoting preventive medicine and improved coordination of health care services, as well as reducing waste and redundant tests, poor EHR system design and improper use can cause EHR-related errors impacting health care outcomes (Dawson, 2017).
The TTF model is key to understanding the impact of technology tools on individual performance. For example, pre-texted medical short phrases that can be readily and rapidly entered into the patient's health record could be useful, if the user is familiar with the phrases and knows how to retrieve them. Kilmon and colleagues (2008) evaluated whether Goodhue's TTF model would serve as a useful diagnostic tool for assessing implementation of a healthcare EHR information system. Kilmon et al. (2008) surveyed 140 nurses and 80 physicians who used the system during its implementation phase. The authors hypothesized that user responses to survey questions would be greater than the mean (4 on a Likert scale of 1-7). This proved true for five of the seven survey questions. The authors concluded that the TTF model and its associated instrument appeared to be a useful diagnostic tool for evaluating a health care information system.
El-Gayar, Deokar, and Wills (2010) conducted a study that focused on user evaluation on three important areas. They included how well the EHR fit the users' tasks, how the users thought use of the system impacted their performance, and the relationship between dimensions of TTF and individual performance. The study tested the hypothesis that the constructs of TTF model would predict perceived performance (El-Gayar et al., 2010). This study pointed to the importance of shifting focus from evaluating the "why" of system use to "how" such system use impacts user performance.
Wills, El-Gayar, and Deokar (2012)  results highlighted barriers and facilitators associated with CDSS use. In summary, on a positive side, personnel who were familiar with using computers gave higher ratings compared to those who were less computer literate. On the negative side, those familiar with computers did not think the CDSS interfaced well with the existing EHR system.

Strengths and Limitations of TTF Framework in
Testing the Usability of EHR

Strengths of TTF Model
The TTF model has been applied in different domains and locales, such as group support systems, knowledge management systems, healthcare settings, ecommerce, and mobile information systems. The TTF model addresses both voluntary and mandatory use situations, has a strong theoretical foundation, and is accompanied by a validated instrument. The TTF has proven to be a valid model and instrument to measure and predict performance impact in a healthcare setting. In addition, the TTF model was developed to be a diagnostic tool to evaluate whether IT services in a given organization will meet user needs (Goodhue & Thompson, 1995). Goodhue and Thompson (1995) reported that research corroborated the relevance of the TTF model in explaining and predicting IT success for individual performance in a healthcare context. Although no systematic bias has been identified regarding the relevance of TTF for different types of information systems, the working premise is that TTF is a valid construct to explain user evaluation of EHR systems. In addition, unlike other theoretical frameworks (e.g. Unified Theory of Acceptance and Use of Technology and Technology Acceptance Model), TTF explains and predicts user performance within an information system.
Results of various studies have confirmed that TTF can be used to explain the impact of IT on individual performance and is a better indicator than usage alone (El-Gayar et al., 2010;Gebauer, Shaw, & Gribbins, 2010;Kilmon et al., 2008). The TTF model developed by Goodhue (1995) studied the relationship between IT and individual performance to predict overall performance and its impact on outcomes.
Thus, the TTF model has been increasingly used to assess user satisfaction and acceptance of IT, including its use in health care domains. The TTF model provides a strong diagnostic tool to evaluate whether IT meets user requirements when performance tasks are broken down into detailed components. The focus on the individual level permits consideration of impact on performance.
In many healthcare organizations today, use of EHR systems is standard of care. Thus, it becomes less central to evaluate the "why" of EHR system use and more relevant to direct research at evaluating "how" EHR systems impact the user performance. The why of system use was examined in earlier models, such as the In contrast with the "why" models, the TTF model addresses utilization from a different perspective and attempts to explain user performance within information systems based on the fit of the task to the technology. The model focuses on the concepts that are most likely to predict performance impact and measures the match between task requirements of the user, an individual's abilities, and the functionality of the system.

Limitations of TTF Model
Although the TTF model has been applied in healthcare settings, studies examining the TTF model have been limited, leaving gaps that need further investigation. Dishaw, Strong, and Bandy (2002) focused on self-efficacy constructs and pointed out that Computer Self-Efficacy (CSE), which examines users' beliefs regarding their ability to perform specific tasks using such a EHR system, had not been linked to the TTF model. Ammenwerth, Iller, and Mahler (2006) reported that there was a lack of knowledge concerning the interaction between the user and the task. Because TTF focuses on the fit between user and technology, and between task and technology, the model does not account for the interaction of user and task, an important factor for the success of IT introduction projects. In addition, TTF does not address the dynamics of introduction projects. Because of the attributes of users, task and technology frequently change over time in a clinical environment, thus, interactions and fit also change (Ammenwerth et al., 2006). Yen (2010) noted that although the TTF model incorporated the essential concept of user-tool-task interaction, it did not address environmental factors that could be crucial to the healthcare context.
Other than the dynamic nature of changing systems, technology, and individual characteristics that a model must contend with, other limitations can be examined from a methodological viewpoint. These include a lack of quantitative study designs, small sample size, convenience sampling, and use of one target population or one health care setting that limits generalizability of results. In addition, the healthcare field is constantly changing and upgrading, so nurses and other health care staff must stay current and aware of new technologies (Chandrasekaran, Anand, Ward, Sharma, & Moffatt-Bruce, 2017). Kilmon et al. (2008) conducted one of the first studies to evaluate whether the TTF model and its associated instruments would provide useful diagnostic tools for assessing the implementation and use of EHR systems. While the results indicated the model was a success in terms of the task-technology fit, the study did not validate the TTF instrument within the healthcare context. Moreover, the study did not evaluate performance impact, or the relationship between TTF and performance impact. It also failed to address user interaction with the task.

El-Gayar et al. (2010) closed this gap by using the TTF framework to evaluate EHR systems. The results confirmed that the TTF is a valid model and instrument that
can be used to predict performance impact in a healthcare setting. In addition, the results highlighted the importance of the TTF dimensions of data quality, ease of use and training, and commutability. However, the recommendation was that further work was needed concerning timeliness and locatability dimensions, which are features to be considered when implementing an EHR (El-Gayar et al., 2010). Despite the results of validating the instrument in the healthcare domain, further work is needed to adapt the instrument to the needs of decision makers and health care providers (e.g., nurses, APRNs, physicians, and PAs) in terms of their job characteristics and information needs.
In the literature reviewed, an article related to health care information systems suggested the TTF framework lacked an adequate consideration of the interaction between the user and the task (Ammenwerth et al., 2006). Future studies about interaction between the user and the task are needed to validate the application among various decision makers within different health care contexts. The importance of the fit between the selected technology and the task to be accomplished is often overlooked in the development and implementation of health care information systems.
Research is shifting from evaluation of the behavioral aspects of adoption and use towards performance impact. As the adoption of EHR systems and other health IT increases, it is imperative that IT research also shifts from evaluation of the behavioral aspects of adoption and use to performance impact. This shift is critical because of the mandatory use of EHR systems. It becomes less important to evaluate the "why" of system use and more important to direct resources to evaluating "how" such IT use impacts user performance.

Conclusion
The purpose of this chapter was to describe the Task Technology Fit (TTF) model in support of its applicability to the study undertaken. The model is widely used in IT and serves as a framework for dealing with the actual fit of technology to the task at hand. Use of the model permits the validity of user preferences and perceptions in the assessment of usability prior to an all system implementation. According to Grant and Osanloo (2014), a framework is a necessity that serves as a blueprint for discovery.

METHODOLOGY
The research methodology and design of the study are presented in this chapter. Included in the chapter is the purpose of the study, the research questions and hypotheses, variables of interest, rationale for the research design, sample procedure, setting, data collection procedure and instruments, human subjects protection and confidentiality, and the data analysis plan.

Purpose of the Study
The purpose of this descriptive, non-experimental survey study was to evaluate the usability of a new SmartPhrase interface tool embedded into the Epic EHR system used by 23 telephone-triage nurses for symptom management in a multi-site cancer center located within the largest healthcare system in Rhode Island. The study was designed to examine nurses' perceptions of how SmartPhrase utilization and job performance are related to usability evaluation of the SmartPhrase tool. Additionally, the impact of demographic characteristics (i.e. years of experience) will be explored.
The study's specific aims are to (1) determine if there is a relationship between telephone triage nurses' usability evaluation of the SmartPhrase tool and their selected demographics; (2) determine if changes in the telephone triage nurses' self-perceived job performance occurred six months following SmartPhrase implementation; and (3) determine if there is a relationship between the telephone triage nurses' usability evaluation of the SmartPhrase tool and their tool utilization.

Rationale for Research Design
This research study was a quantitative non-experimental survey design, primarily descriptive in nature. According to Creswell (2013), this approach is a postpositive perspective for developing knowledge that uses strategies such as surveys and predetermined instruments to collect data for statistical analysis. Objectivity is an essential aspect of competent inquiry, while adhering to standards of validity and reliability with quantitative measurement of variables and their relationships to each other. The Task-Technology Fit (TTF) model (Goodhue &Thompson, 1995) was selected as the theoretical framework to guide this research and was focused on hypotheses testing. Goodhue and Thompson (1995) conducted quantitative research based on a deductive research approach to explore, explain, and predict user performance within the IT domain and to provide understanding of relationships between technology and user evaluation. For this study, the main research question dealt with the relationship between usability of the SmartPhrase tool and nurses' selfperceived performance. To examine this relationship, a descriptive quantitative design was used to describe and measure the association (or relationship) between demographics and other variables of interest, including usability, nurses' selfperceived performance, and how well nurses' use of the SmartPhrase tool technology fit their tasks.
According to Schmidt and Brown (2015), a non-experimental design can be used when there is little information known about a particular phenomenon, or when it is not practical to implement an experimental approach. Correlational designs are used when researchers are interested in establishing relationships between two or more variables (Schmidt & Brown, 2015).
This study used a descriptive correlational design to describe the change from pre-to-post implementation of a new SmartPhrase interface tool. The relationship between specific demographic variables (i.e. years of nursing experience situated in nursing practice, oncology, telephone triage, and LifeChart) and usability was examined. The consideration of the nurses' subjective perception of job performance in using the SmartPhrase interface tool, was an exploratory attempt at understanding this relationship. Therefore, this approach was used to describe the extent to which the level of usability of the SmartPhrase tool was related to the nurses' utilization and job performance.

Research Method
For this study, a questionnaire as a stand-alone measure of usability was used to provide a measure of usability and to permit a numerical measure of the usability of an EHR system. This study also used a post-study questionnaire in order to focus on the measurement of EHR usability without asking the nurse to complete a task or scenario in a lab or specific place. The two post-study questionnaires most widely used for assessment of the perception of usability are the SUS (Brooke, 1996) and the SUMI (Kirakowski, 1996). The SUS was selected for this study.
A survey approach can collect a broad range of data (e.g. demographics, opinions, and perceptions about use of tool to task) from the population by just studying a subsample of that population (Creswell, 2013). In order to identify areas for improvement of the EHR system, understanding the nurse-users' perspective on the usability of the SmartPhrase interface as measured by their self-perceived impact on their performance was critical.
The survey method uses as a stand-alone usability evaluation method to obtain numeric data. Binary yes/no responses and a Likert-type rating scale were numerically coded. The survey is designed to examine a sample of 23 telephone triage nurses and testing the use of the SmartPhrase tool. The data collection permits the quantification of the nurses' self-perceptions of their healthcare performance, descriptions of the relationship between the independent variables (socio-demographic characteristics and perceived usability of EHR systems) and the dependent variables (self-perceived performance of telephone triage nurses and utilization).
Using a survey approach has the distinct advantage of testing participants over short periods of time at relatively low cost and supports objectivity in the sense of decreasing researcher bias. That is, data are numerically coded, thus, allowing results other than that of personal judgment by the researcher. This permits generalization of results to the population-at-large. In addition, a survey is relatively easy to administer and usually does not interfere with the participants' tasks (Creswell, 2013;Polit & Beck, 2014). A main disadvantage of this approach could be reliability of survey data.
As reported by DeFranzo (2012), reliability may be affected by factors such as accurate reporting and honest answers. Respondents may answer questions as to how they think they should respond instead of how they truly feel. In addition, some survey answer options could lead to compromised results if answer options are interpreted differently by respondents.

Research Questions and Hypothesis
The research questions and hypotheses are as follows: Research Question 1. Is there a relationship between telephone triage nurses' years of nursing experience and their perception of the SmartPhrase tool usability as measured by the System Usability Scale (SUS) at six-months after implementation? Hypothesis 1. Nurses with more years of nursing experience (nursing practice, oncology practice, telephone triaging, EHR, and LifeChart) will report higher usability scores of the SmartPhrase tool on the SUS than nurses with fewer years of experience (at p<. 05 level of significance).

Research Question 2.
Is there an increase in telephone triage nurses' selfperceived job performance following integration of a SmartPhrase tool into the EHR from baseline pre-implementation to 6-months post implementation? Hypothesis 2. Nurses will show a significant increase in self-perceived job performance following integration of the SmartPhrase tool into the EHR as measured by the Telephone Triage Nurse Survey (TTNS) from pre-implementation to sixmonths post implementation (at p<. 05 level of significance).

Terms and Variables of Interest Defined
The independent variables were sociodemographic characteristics and perceived usability of EHR systems. The dependent variables were self-perceived performance of telephone triage nurses and utilization. Definitions used in the study were as follows: Self-perceived Nursing care performance as an outcome of their nursing practice, refers to the nurses' self-perceived effectiveness of functions that provide the means to achieve nursing system goals (Dubois, D'Amour, Pomey, Girard, & Brault, 2013). The measurement of self-perceived nursing care performance permits examining the contribution that nursing makes to patient outcomes. Evaluation of nursing care performance permits facilitating improvements in nursing quality and patient safety, as well as nursing practice outcomes (Sim, Crookes, Walsh, & Halcomb, 2018). Operationally, selfperceived nursing performance was defined by a score calculated from the

TTNS.
SmartPhrase Tool has evolved as an elemental unit of the Epic EHR in the Lifespan system, specifically as an electronic documentation feature that permits nurses to use preformatted text, statements, or structured note templates for cuing standardized assessment and documentation (Lamba et al., 2016).
Telephone triage nurse is a RN who speaks remotely by telephone to a patient and assesses the patient's symptoms or health concerns, answers his or her health questions, determines what kind of care is needed, and offers advice (National Council of State Boards of Nursing, 2012).
Usability is defined through the user's perspective as how he or she finds the system useful, usable, and satisfactory for accomplishing work goals through task performance (Zhang & Walji, 2011). Operationally, usability is a score calculated from the SUS test.
Utilization is defined as the triage nurses' self-reported use of technology in completing tasks (Attefalk & Langervik, 2001), which, in this case, pertains to SmartPhrase tool use. Operationally, utilization is a score calculated from items of the Socio Technical Approach to Soft Systems Methodology (STSSM) questionnaire and the SUS, measures given post-test only.

Target and Accessible Populations
The target population in this study was comprised of all (23) registered nurses who were employed as telephone-triage nurses in the multi-site Lifespan Cancer Center in Rhode Island. This included experienced and newly appointed telephone triage nurses, who participated in the design of the Smart Phrase tool. This accessible population was chosen because of the large number of patients (~14,000) served by this center. Also, the Epic EHR system has been in operation for approximately two years, and there was a need to evaluate the SmartPhrase tool in a well-integrated system.

Inclusion and Exclusion Criteria
RNs in the Lifespan organization, who were using the SmartPhrase tool in the EHR, were recruited to participate in this study. The inclusion criterion were RNs who had been trained to use the SmartPhrase and had at least one-year experience as a telephone triage nurse. Excluded were telephone triage RNs who were currently working in non-cancer centers (i.e. trauma, poison control and other care facilities).
They were excluded because they worked in settings that had different clinical triage procedures and served patients with different healthcare profiles.

Sampling Method
All 23 RNs were recruited using a purposive sampling approach, namely, the researcher sampled the total population. From the telephone triage nurses employed in the Lifespan Cancer Institute, a total of 23 RNs that met the inclusion criterion were enrolled in this study. A purposive, non-probability sampling approach was used, and this approach was selected based on characteristics of the population and the aims of the study. Purposive sampling is also known as judgmental, selective, or subjective sampling (Schmidt & Brown, 2015). This sampling method was selected because of the limited number of telephone triage RNs in the accessible population that use the SmartPhrase tool in the cancer center. Despite the subjective nature of selecting participants in purposive sampling, it can be useful in situations when the researcher needs to reach a targeted sample quickly to gain insights about the phenomenon being studied when sampling for proportionality is not a main concern (Schmidt & Brown, 2015).

Recruitment of Participants
Following Institutional Review Board (IRB) approval from the Lifespan Cancer Center Institute and the University of Rhode Island (URI), telephone triage RNs, all of whom had participated in the development of the SmartPhrase tool, were first introduced to the study at regularly scheduled staff meetings. The nurses were given a short training session on the SmartPhrase tool, whether or not they elected to participate in this study. Participation was voluntary, responses were confidential and completely anonymous. Subjects were not identified by name or any other demographic identifiers. There was no bearing on their performance evaluations as to whether or not they elected to participate in the study.

Setting
The Lifespan Cancer Institute is Rhode Island's largest center dedicated to the diagnosis and treatment of all types of cancers and blood disorders. It includes four sites that provide state-of-the-art cancer care with the goal of providing "health-withcare" support to over 14,000 patients and their families. The intent of the institute is to

Data Collection: Instruments
A description of the measures used in the current study as well as evidence regarding their reliability and validity is presented in this section. The relationship of each instrument to each of the concepts of the TTF framework is depicted in Figure 2, A form for demographic information and three survey instruments were used to collect data. The surveys administered included items on the SUS, TTNS, and STSSM. These instruments were used to measure the 1) RNs' perception of the overall usability of the new SmartPhrase tool, 2) differences in telephone triage RNs' self-perceptions of their job performance "before" and "after" using the SmartPhrase tool in the EHR, and 3) RNs' utilization of the tool, or the extent to which the SmartPhrase use was integrated into their work processes.

Demographic Form
A demographic data form included in the pretest survey identified six variables of interest. Demographic variables included the subject's age, gender, years in nursing practice, years in oncology nursing practice, years working as a telephone triage RN, years using the EHR in nursing practice, and years using LifeChart in nursing practice.
(See Appendix B). These RN demographic variables have been shown in literature to effect performance (Eo, Kim, & Lee, 2014;Reid, Hurst, & Anderson, 2013). The years of technology use was expected to be a contributing factor in promoting tool usability.

Measuring Usability.
In order to explore nurses' perception of the use of the currently installed SmartPhrase tool, specific questions were asked of the RNs. The information sought was to discover how useful the SmartPhase tool was to the nurses. Use of the EHR system directly affected the nurses' tasks on a daily basis because nurses generally do more documentation and process more information than other health care providers. However, switching from paper to an EHR system for these activities often is perceived as an onerous and frustrating experience. Therefore, it was critical to test the usability of the SmartPhrase tool is in order to consider its benefits and any enhancement of nurse performance.
The SUS, developed by Brooke in 1996, is a user's subjective rating of a product's usability. The instrument is a widely used measure with acceptable ranges of psychometric properties (Bangor et al., 2008). It contains 10 questions that are scored on a 5-point Likert scale based on strength of agreement. (See Appendix C). The SUS was used as a stand-alone evaluation instrument to assess the overall usability of the new SmartPhrase tool. Researchers have reported high Cronbach alpha scores for SUS, with the most comprehensive examination reporting a reliability of .92 (Lewis & Sauro, 2009). Bangor et al. (2008) reported the reliability as 0.9. The SUS also has a significant amount of benchmark data available for a wide range of interfaces allowing researchers to interpret and compare their results with other similar kinds of products (Bangor et al., 2008;Kortum & Bangor, 2013;Sauro, 2011). Bangor et al. (2008) added an interpretation of the SUS score. They found that if the SUS score is over 85 then the software is highly usable. If the score is between 70 to 85, it is characterized from good to excellent and greater than 50 up to 70 shows that the system is acceptable, but it has some usability problems and needs improvement. A score of 50 or less, reflects that the system is considered unusable and unacceptable, and it needs to be fixed fast. The SUS is an effective tool for assessing the usability of a computer system (Bangor et al., 2009) and is the most used questionnaire for measuring perceptions of usability (Peres, Pham, & Phillips, 2013;Xiang & Tussyadiah, 2014).

Measuring Self-Perceived Nurse Job Performance
Preliminary to this study, the TTNS (Dufault, 2017) was specifically developed to measure the difference in telephone triage nurses' self-perceptions of their job performance before and after integrating the SmartPhrase tool into the EHR. Between 87.5% and 100% agreement was obtained on each of the 13 items, with respect to the construct measured and whether it should be included in the survey form. Two qualitative items were later added to obtain additional data for a future qualitative study. Inter-item consistency was conducted and yielded a Cronbach's alpha score of 0.79, indicative of a high level of internal consistency.

Measuring Utilization
Part D of the STSSM questionnaire was developed to measure the extent computer systems have been integrated into an individual's work processes (Attefalk & Langervik, 2001). The authors defined the term "utilization" as users' behavior in the use of technology in completing tasks. Part D focuses on how an individual perceives dependency on the Information System to accomplish his or her work routines. Each statement was rated on a 5-points Likert scale ranging from (1) "Not at all" to (5) "Very dependent." Items were based on the original questionnaire of the TTF model (Goodhue & Thompson, 1995), where the respondents were asked what impact the computer system and surrounding services had on the effectiveness, productivity, and performance of their job. As a measure of utilization, the STSSM, Part D was administered as a posttest only.

Data Collection Procedure
Data collection was from December 2017 through August 2018. The followup, posttest design was selected to test technology use of specific characteristics and conditions relating to usability. The sample of 23 staff RNs who work as telephone triage nurses were asked to complete a demographic data form and the 15-item "Telephone Triage Nurse Survey" at two time points: pre-implementation and sixmonths post implementation of the SmartPhrase tool. This survey was used to measure self-perceived job performance (Appendix D). The nurses also evaluated the usability and utilization of the SmartPhrase tool six months following implementation, using a 13-item Usability of the SmartPhrase Survey (Appendix E). The six-month period included a learning curve of six months post training related to technology changes in computerized information systems.

Human Subjects Protection and Confidentiality
The IRB of both Lifespan and URI designated the study as exempt from review (See Appendix F & G). The decision to participate in this study was voluntary and all potential participants were assured they had the right to accept or refuse to be a part of this study, without any repercussions. During the enrollment phase, the participants were reassured they had the right to ask questions about the study and were informed they could withdraw at any time. All information remained anonymous, and data were stored in a locked file in the locked office of Dr. Marlene Dufault, PhD, RN at the College of Nursing at URI, Kingston, Rhode Island. The completion of the study questionnaire was indicated as consent to participate. Each subject received a cover letter with a detailed explanation of the study (Appendix H).

Risks and Benefits
There were minimal risks and no direct benefits associated with the nurses' participation, although they may have reflected on their own telephone triage skills and identified training needs related to their role as a telephone triage nurse. Using a SmartPhrase tool was expected to help to standardize an approach to symptoms assessment, stream line documentation time, and make the nurse's job easier.
Participating in the study was expected to improve quality monitoring of future symptom assessment and management.

Data Analysis Plan
This section describes the procedures for processing and analyzing the data.
The data were obtained in order to answer research questions and hypotheses and to provide quantifiable, objective, and easy to interpret results (Simpson, 2015). All analyses were carried out using Statistical Analysis System (SAS) software, version 9.4. The following sections provide an overview of the data analysis plan.

Data Entry
Individual responses on each survey instrumentation form were coded numerically, and a code book was created to facilitate data entry. The code book included all items of the collected data at the two time points, and the pre and post responses were identified by number. Initially, all quantitative data were doubleentered into an Excel spreadsheet and a check was made for data entry errors. Then, all the data was imported into a SAS database, where the data was again screened for missing values and outliers and normality evaluation. Prior to carrying out the statistical analyses to answer the research questions and test the hypotheses, the SmartPhrase usability scores, as measured by the SUS, were calculated according to the SUS scoring manual. The scoring formula was as follows: (1) for each of the oddnumbered questions, subtract 1 from the score; (2) for each of the even numbered questions, subtract their value from 5; (3) take these new values and add up the total score; and (4) then multiply this by 2.5.

Descriptive and Inferential Statistics
Descriptive statistics were used to describe, summarize, and synthesize collected data (Schmidt & Brown, 2015). In this case, descriptive statistics were used to organize, simplify, describe and present the data. The data distribution and sample characteristics were summarized, as were responses to individual survey questions, using frequencies and percentages, and means and standard deviations. Histograms were used to display data where the data was continuous (as in Likert scale data).
Also, bivariate analyses were conducted to describe differences in the pre to post-test means of TTNS scores. Bivariate descriptive statistics were run to explore the relationships among the variables (Schmidt & Brown, 2015) including the demographic variables of years of experience and the usability score.
To test hypotheses and potentially generalize results to the population as a whole, inferential statistics were used. The major goal of the inferential statistical analysis was to examine the relationship between usability of the SmartPhrase tool and nurses perceived job performance. Hypotheses testing were based on results from the SUS (post-test only) and TTNS (pre and post-test), with the post test administered 6 months after implementation. Generalized linear models (GLMM) were used to test these hypotheses. GLMM provided a more flexible approach for analyzing data.
Therefore, GLMM accommodated the non-normal distributed responses, handled the possibly non-linear link between the mean of the response and the predictors, and allowed for some forms of correlation in a random effect's covariance data (McCulloch & Neuhaus, 2014).
Each of the measures of years of relevant nursing experience was tested for a relationship to the SUS scale for usability. A hypothesis test for differential changes in perception between pre-implementation and the six-month TTNS survey was accomplished using a generalized estimating equation. For all generalized modes, a binomial distribution was used wherein observed scores were treated as successes, and the maximum range of the instrument was treated as the number of trials after rescaling each score to have a low score of zero. Results were reported as central tendencies and slopes with 95% confidence intervals, depending on the model and hypothesis, and the alpha (p value) was set to 0.05.

RESULTS
The purpose of this study was to evaluate the usability of a SmartPhrase tool integrated into an EHR system and to determine the direction (positive or negative) of telephone triage nurses' self-perceived performance six months following the tool's implementation. Also examined was the relationship between the usability evaluation of the SmartPhrase tool and the nurse subjects' utilization.
This chapter includes the statistical analyses conducted to answer the research questions and hypotheses. The results of the study are presented as follows: 1) the participants' demographical characteristics; and 2) the analyses and results of the study's three research questions and four hypotheses. This chapter presents the data in a meaningful way to facilitate the discussion presented in Chapter 6.

Research Questions and Hypotheses
The three research questions explored in this study and their accompanying hypotheses were: Research Question 1. Is there a relationship between telephone triage nurses' years of nursing experience and their perception of the SmartPhrase tool usability as measured by the System Usability Scale (SUS) at six months after implementation? Hypothesis 1. Nurses with more years of nursing experience in (nursing practice, oncology practice, telephone triaging, using the EHR, and using LifeChart) will report higher usability of the SmartPhrase tool on the SUS than nurses with fewer years of experience (at p< .05 level of significance).

Research Question 2.
Is there an increase in telephone triage nurses' selfperceived job performance following integration of a SmartPhrase tool into the EHR from baseline pre-implementation to 6-months post implementation? Hypothesis 2. Nurses will show a significant increase in self-perceived job performance following integration of the SmartPhrase tool into the EHR as measured by the Telephone Triage Nurse Survey (TTNS) from pre-implementation to sixmonths post implementation (at p< .05 level of significance).

Research Question 3.
Is there a relationship between the nurses' perception of the usability of the SmartPhrase tool and their self-perceived job performance and their self-reported utilization six months post-implementation of the SmartPhrase tool?
Hypothesis 4. Higher self-perceived usability of the SmartPhrase tool is associated with a higher self-perceived job performance six-months post implementation was not upheld.
Hypothesis 4. Higher self-perceived usability of the Smartphrase is associated with greater self-reported utilization of the SmartPhrase tool six-months post implementation was also not upheld.

Sample Demographics
This section presents information on the 1) percentages and frequencies for demographic data, 2) descriptive statistics for the usability of the SmartPhrase tool, and 3) the mean scores and standard deviations for the variables of the telephone triage nurses' self-perceived performance and utilization.

Sample Characteristics
The sample (n = 23) was comprised of telephone triage nurses in the multi-site Lifespan Cancer Center in Rhode Island. Table 3 provides an overview of the sample.
Of the 23 respondents, 69.6 % (n = 16) completed both the pre and post survey questionnaires, 21.7 % (n = 5) completed the pre-survey only, and 8.7% (n = 2) completed the post survey only.
Telephone triage nurses' work experiences in terms of years of nursing practice, oncology practice, telephone triaging, and EHR system use were calculated.
Descriptive statistics for the demographic variables are summarized in Table 4.

Demographic Variables
Years of nursing experience. The average number of years of nursing experience was 17.48 and the range was 1-40 years. The median was 15 years. Figure   3 shows the histogram of the distribution of number of years' experience skewed to the right and peaked at 8 years. The distribution was bimodal with peaks at 8 and 32 years.

Figure 3: Years of Nursing Experience.
Years of oncology experience. The average number of years of oncology experience was 12.59 (SD = 10.08) years, and the range was 0.5 -30 years. The median was 9, and the mode was 5 years. As shown in Figure 4, the histogram shows the distribution skewed to the right and peaked at approximately 6 years. The most representative measure of central tendency for this variable is the median.

Figure 4: Years of Oncology Experience.
Years of telephone triage experience. The average number of years of telephone triage experience was 2.91 (SD = 3.32) years, and the range was 1 -15 years. The median was two years, and the mode was one year. As shown in Figure 5, the histogram shows the distribution skewed to the right. The percent of respondents shows that few with had more than six years' experience in telephone triage. The most representative measure of central tendency of this variable is the median.

Years of EHR system experience.
The average number of years of EHR system experience was 5.54 (SD = 3.41) years, and the range was 1 -13 years. The median was two years and the mode was one year. As shown in Figure 6, the histogram shows the distribution skewed to the right and peaked at 4.5 years of experience. The most representative measure of central tendency for this variable is the median.

Figure 6: Years of Electronic Health Record System Experience.
Years of LifeChart experience. The average number of years of LifeChart experience was three years, and the range was 1 -5 years. The median and mode were three years. As seen in Figure 7, the histogram shows the distribution of number of years' experience, with three years as a central value. Usability score of SmartPhrase tool. The SUS score for the SmartPhrase tool was 58.06 (SD = 12.5), with a range of 42.50 to 82.50. According to Brooke (2013), SUS scores can be transformed to 100 by taking the odd numbered items and subtracting 1 from the users' response, taking the even numbered items and subtracting the user response from 5, summing the resulting values, and multiplying by 2.5. Figure 8 shows the nurses' (n = 18) usability score of the SmartPhrase tool. As seen in Figure 8, one nurse scored a usability score above 80 (good usability), six (33%) scored usability below 50 (unusable and unacceptable), and 11 nurses (61%) scored in the low marginal acceptable range. The median and mode were 56.25 and 57.5, respectively.

Figure 8: System Usability Scale Divisions by Acceptance.
In this study, the average usability score of the SmartPhrase tool was 58.06, a value lower than the SUS Benchmark score of 68 as calculated by Brooke's (2013) conversion to the 100-point scale. Brooke's interpretation of SUS scores was that SUS scores above 85 are considered "excellent," whereas a score of 50 or under indicates the system is considered unusable and unacceptable. Figure 9 is an example of how the SUS scores are measured (Brooke, 2013).

Output Variables
Descriptive statistics for the dependent variables, nurses' self-perceived job performance and utilization of the SmartPhrase tool are summarized in Table 5. Telephone triage nurses' self-perceived job performance mean scores as measured by the Telephone Triage Nurse Survey (TTNS) were essentially the same for both the pre and post surveys. Nurses' self-perceived job performance. Telephone triage nurses' selfperceived job performance scores showed a slight decrease between the mean scores before implementing the SmartPhrase tool (Mean=46.9) and six-months after implementation(Mean=45.9). Figure 10 displays a scatterplot of the telephone triage nurses' self-perceived job performance scores before and after implementation of the SmartPhrase tool. reported on their utilization of the SmartPhrase tool after six months of the tool's implementation. The utilization score average was 7.00 (SD = 2.05), and the range was 3.00 -11.00. The mean and the median were similar.

Research Question Analysis
For the three research questions, a Generalized Linear Mixed Model (GLMM) using the GLIMMIX procedure in SAS, was used for analysis. GLMM accommodated the non-normal distributed responses and the non-linear link between the mean of the response and the predictors. It also permitted a random effects covariance data analysis.

Research Question 1
Research question 1 examined if there was a relationship between telephone triage nurses' years of nursing experience and their usability score of the SmartPhrase tool, as measured by the System Usability Scale (SUS) at six months after implementation.
Hypothesis 1. The hypothesis was that years of nursing experience (nursing practice, oncology practice, telephone triaging, using the EHR, and using LifeChart) related to the usability scores (the greater the experience, the higher the usability score) was not upheld.

Generalized Linear Mixed Model (GLMM).
The GLIMMIX procedure was conducted to assess whether years of nursing experience (nursing practice, oncology practice, telephone triaging, EHR, and LifeChart) significantly predicted the usability score of the SmartPhrase tool.
Results. There were no significant correlations between any pairs of the variables, except for the predictor variable, number of years' experience of oncology practice. Figures 11-15 present the results of the GLIMMIX between all pairs of the variables. Discussion for Figures 11-15 is as follows: 1. Usability summary score as a function of years of nursing experience.
No significant correlation was found between usability summary scores and the years of nursing experience (p = 0.07). As shown in Figure 11, the perception of usability of the SmartPhrase tool slightly decreased as a function of years of nursing experience.

Figure 11: Distribution of Usability Scores with Years of Nursing
Experience.

Usability summary score as a function of years of oncology experience.
The years of oncology practice significantly predicted (p = 0.0311) a direction in the usability score of the SmartPhrase tool. The perception of usability of the SmartPhrase tool negatively correlated with years of nursing oncology practice, as seen in Figure   12. 3. Usability summary score as a function of years of telephone triage nursing. No significant correlation was found between usability summary scores and the years of telephone triage nursing experience (p = 0.6). As shown in Figure 13, the perception of usability of the SmartPhrase tool appeared to be unaffected by the years of telephone triage nursing experience. 4. Usability summary score as a function of years of EHR experience. No significant correlation was found between usability summary scores and the years of EHR experience (p = 0.1). As shown in Figure 14, the perception of usability of the SmartPhrase tool appeared to be slightly increased by the years of EHR experience.

Research Question 2
Research question 2 examined if there was a significant increase in nurse's self-perceived job performance following integration of the SmartPhrase tool into the EHR, as measured by the Telephone Triage Nurse Survey (TTNS) at pre and post implementation.
Hypothesis 2. The hypothesis was that telephone triage nurses would show a significant increase in self-perceived job performance following integration of the SmartPhrase tool into the EHR was not upheld. The GLIMMIX procedure was conducted to examine whether there was a significant difference in nurse's selfperceived job performance six months after implementing the SmartPhrase tool. Table 6, the result for total score analysis was not statistically significant, indicating that using the SmartPhrase tool was not associated with improved nor worsened performance as a telephone triage nurse. The descriptive analyses of telephone triage nurses' self-perceived job performance scores showed a slight decrease from pre(M = 46.9) to post (M = 45.9). Therefore, further investigation was conducted to determine the exact source of a statistically significant interaction.

Results. As shown in
The GLIMMIX procedure was conducted for each individual question of the TTNS.
Questions 10, and 13 were statistically significant, with p-values of .003, and .042 respectively. Question 1, though not statistically significant, approached significance with an F value of 3.52 and a p-value 0.08 which may have clinical significance. In question 10, the telephone triage nurses' self-perceived job performance scores showed a slight decrease from pre (M = 3.6) to post (M = 2.8) as seen in Figure   16.  In question 13, the telephone triage nurses' self-perceived job performance scores showed a slight decrease from pre (M = 3.6) to post (M = 3.0) as seen in Figure   17.

Research Question 3
Research question 3 examined if there was a relationship between nurses' perception of the usability of the SmartPhrase tool and their self-perceived job performance (six-months post implementation) and the relationship between nurses' perception of their self-reported utilization six-months post-implementation of the SmartPhrase tool. This research question was divided into two hypotheses (Hypotheses 3 and 4). The GLIMMIX procedure was conducted to assess whether perceived usability of the SmartPhrase significantly predicted the self-perceived job performance six-months post implementation of the SmartPhrase tool. It was also used to assess whether perceived usability predicted utilization six-months post implementation.
Hypothesis 3. The hypothesis was higher self-perceived usability of the SmartPhrase tool is associated with a higher self-perceived job performance sixmonths post implementation was not upheld.
Results. There were no significant correlations between the nurses' perception of the usability of the SmartPhrase tool and their self-perceived job performance (total score) six-months post implementation. As seen in Table 6, the F value of 1.73 resulted in a p value 0.2.
Hypothesis 4. The hypothesis, higher self-perceived usability of the SmartPhrase would be associated with greater self-reported utilization of the SmartPhrase tool six months post implementation, was not upheld.
Results. Given the p value of 0.09, further investigation was conducted to determine the exact source of the statistically significant interaction. The GLIMMIX procedure was conducted between the usability summary score and self-reported utilization questions to determine which items were statistically related to the summary score. As displayed in Table 7, question 11 and 13 were statistically significant, with p values of 0.035 and <0.0001, respectively.

DISCUSSION
The study was conducted to evaluate the usability of a SmartPhrase tool embedded into the Epic EHR system used by 23 telephone-triage nurses in a multi-site cancer center located within the largest healthcare system in Rhode Island. This chapter provides a discussion of the findings, the limitations of this study, and the implications of the findings for nursing informatics and future research.

Usability Findings
The findings of this study indicated that the SmartPhrase tool evaluated by telephone triage nurses was not perceived as particularly useful. That is, as a task of technology, its usability was less than optimal. The task of the SmartPhrase tool was to permit quick and easy insertion of pre-texted symptom assessment cues or phrases into the patient's EHR. The usability score of the SmartPhrase tool was at an acceptable level but was indicative of usability problems requiring improvement. The only variable contributing to the direction (positive or negative aspect of usability) was the nurses' years of oncology experience. The greater the number of years of oncology nursing experience, the less the nurse perceived the usefulness of the tool and there was less satisfaction with its effect on their performance.
The pre and post mean scores on self-perceived job performance, as measured by the Telephone Triage Nurse Survey, were approximately the same (52.6 [pre] and 51.12 [post]). The hypotheses were not supported. These results were in keeping with the results of the SmartPhrase's usability score (SUS), which indicated that usability was poor. Therefore, the study results supported the premise that whereas high usability would increase self-perceived job performance, low usability would not do so or would decrease job performance.
The study was based on the TTF framework, and results were in keeping with the model's framework. Information on the usability of the SmartPhrase tool from telephone-triage nurses' viewpoint and its application to the workflow was the type of usability information that was expected. The TTF model permitted studying the relationships between health IT and self-perceptions of individual performance, regardless of the extent of usability of the technology.

Research Question 1
The findings of research question 1 as they related to demographic variables are discussed here under headings of the overall usability score of SmartPhrase tool, years of nursing experience in oncology, and nurse satisfaction with the SmartPhrase tool. In this case, overall usability was poor and the years of experience as an oncology nurse was a negative predictor, and nurse satisfaction with the SmartPhrase tool was low.
Overall usability score. The usability test, the SUS, served as a posttest survey. Interpretation of score results were based on a scale devised by Brooks (2013).
Using Brooks' scale, the total mean score (M=58.06) across subjects of the usability score of SmartPhrase correlated to marginal acceptability, which earned an "F" on Brook's grading scale. Marginal acceptability relates to poor usability and a need to redesign for improvement. Figure 20 shows the nurses' usability overall score distribution. None of the telephone triage nurses evaluated the SmartPhrase tool above 85 (excellent usability). However, of the 18 nurses responding to the survey, five (26 %) evaluated the usability above the SUS benchmark of 68. Almost one quarter of the nurses scored in the acceptable range, indicative of a C grade. These nurses had less experience as a triage nurse. The nurses (n=6) scoring lowest, or receiving an F grade, were those with the greater number of years of experience. Based on the Brooke (2013) ranking scores, there is evidence to conclude that the SmartPhrase tool used by the telephone triage nurses was not all that usable, at least for the more experienced nurses. A partial explanation for this finding was that nurses with the most experience in oncology already knew the phrases they wanted to use and did not need to have cues to assist them in their symptom assessment. Therefore, their use of the preselected phrases tool was less than for the nurses with less years of oncology experience. Another interpretation is that the users perceived that they could write in phrases easier and faster than they could using the tool. That is, the technology tool was more of a hindrance than a help. Telephone triage nurses with more years of oncology expressed dissatisfaction with the SmartPhrase tool related to the tool's poor fit with clinical workflow, which caused disruptions in functionality and negatively impacted the nurse-patient communication. This finding supports the importance of using actual end-users (i.e. telephone-triage nurses) to gain perception for usability of technology tools. Summative measures can be used to obtain an understanding of nurses' needs, likes, and dislikes, as well as provide feedback on issues of any given system interface design (Schumacher & Jerch, 2012).
Satisfaction with SmartPhrase tool. Low satisfaction with the SmartPhrase tool suggests that telephone triage nurses were hesitant or not likely to adopt use of the SmartPhrase tool. Consistent with other studies, healthcare stakeholders need to recognize the importance of evaluating usability issues before a system is adopted and implemented. In addition, to ensure that EHR systems and associated tools work within the nurses' needs, EHR usability concerns should be addressed and rectified to support effective and efficient clinical work (Darmon, Sauvant, Staccini, & Letrilliart, 2014;Raglan, Margolis, Paulus, & Schulkin, 2014;Topaz et al., 2017).
Computer experience of telephone triage nurses. In today's healthcare organizations, nurses have some degree of computer literacy. Nurses use computers, smartphones and tablets to manage input and updates into the EHR system (Mugomeri, Chatanga, Maibvise, & Masitha,2016). In this study, the years of experience are indicative of a wide range of chronological age. Study results support that the younger the nurse, the more likely the degree of computer knowledge than among older , more experienced nurses. This is keeping with a study in which health professionals' age influenced their computer knowledge, attitudes, and utilization (Sukums, Mensah, Mpembeni, Kaltschmidt, Haefeli, & Blank, 2014). It is likely that younger, less experienced nurses benefit from SmartPhrase tool use more so than the older, more experienced nurse as the younger nurses are comfortable with the fit of the technology.

Research Question 2
The result of research question 2 is discussed under the heading of comparison of pre to post test scores on self-perceived job performance. The telephone triage nurses completed a 13-item survey, the Telephone Triage Nurse Survey (TTNS) before and 6-months after the SmartPhrase tool was implemented in the EHR system.
Although the pre to post-test analysis was not statistically significant, two items (items 10 and 13 of the survey) were significant and are discussed separately.
Comparison of pre to post scores on self-perceived job performance.
Research question 2 dealt with assessing self-perceived job performance following the integration of the SmartPhrase tool into the EHR. The GLIMMIX procedure conducted to examine whether there was a significant difference in nurses' selfperceived job performance showed this difference was not statistically significant (p = 0.4). However, this result confirmed the SUS score finding that when the SmartPhrase's usability overall score was low, perception of job performance did not change. Conclusions that can be drawn are that the design of the SmartPhrase tool poorly fit with clinical workflow or that the implementation of the SmartPhrase tool into the Epic EHR system was a poor fit. These results further support the idea that usability testing is a useful way to capture performance metrics of a computer system (Soegaard, 2018). With usability testing, insights into the user performance can be obtained. For instance, better self-perceived job performance can be obtained by aggregating the assessments from the end-users to find usability problems (Oztekin, 2011 identified as a usability barrier (Roman, Ancker, Johnson, & Senathirajah, 2017).
Question 13 of TTNS. Question 13 asked if the decision support system and guidelines for assessment could be or was improved by the use of the SmartPhrase tool. Results showed a decrease in the pretest mean of 3.0 (out of five) to a posttest mean of 2.0. Of the 16 nurses who completed the pretest and posttest, six nurses (37%) did not show a change in score, seven nurses (44%) showed a decrease, and only 3 nurses (19%) showed an increase. This indicates that the SmartPhrase tool did not provide a good option and that use of the SmartPhrase could not help the telephone triage nurse to decide which assessment to focus on. Although the SmartPhrase tool was supposed to organize clinical data in a focused manner to facilitate decisionmaking, results showed that the tool use did not, and possibly would not, effect or improve the decision support system. This result revealed that the nurses' usability perception of SmartPhrase tool was related to usability problems, which was not attributable to using the SmartPhrase tool (Clements, 2018).

Research Question 3
The finding of research question 3 is discussed under the heading of the perception of usability of the SmartPhrase tool. Test items 11 and 13 of the Socio Technical Approach to Soft Systems Methodology (STSSM) questionnaire, which were statistically significant, are discussed separately.

Perception of the usability of the SmartPhrase tool. This question dealt
with the relationship between the nurses' perception of usability of the SmartPhrase tool and their self-perceived job performance and tool utilization six-months postimplementation of the SmartPhrase tool. Results of the GLIMMIX procedure was conducted to assess whether perceived usability of the SmartPhrase significantly predicted self-perceived job performance. Results were indicative that self-perceived job performance was not positively effected. Results did not show a significant correlation between the nurses' perception of the usability of the SmartPhrase tool and their self-reported utilization of tool. However, given a significant p value of 0.09, further investigation delineated a probable source of a statistically significant interaction. A GLIMMIX procedure analysis between the usability summary score and the self-reported utilization questions showed that two STSSM questions were significantly related to the perceived usability of the SmartPhrase. Question 11 and 13 were statistically significant, with p-values of 0.035 and <0.0001, respectively.
Question 11 of STSSM questionnaire. Question 11 asked if users were dependent on the SmartPhrase tool in their work routines. The p value was (0.035) and the mean score was 2 (out of 5). The perception of usability of the SmartPhrase tool negatively correlated with the dependency of the nurses on the SmartPhrase tool in their work routines. The interpretation was that the SmartPhrase tool intended use was not realized and was not adoptable as is. Of the 18 nurses who answered this question, ten (56%) of their scores were below 3, four (22%) of their scores were at a neutral score of 3, and four (22%) had scores above 3. The majority of the nurses (56%) did not rely on the SmartPhrase tool to save time in documentation. The tool did not provide a method for achieving a standardized assessment for cancer patients. This is puzzling, as standardization is recognized as an important criterion for assessing quality and safety of nursing assessments. This finding needs further investigation in the future.
The purpose of the SmartPhrase tool was to incorporate a standard of care that would assist telephone triage nurses in achieving complete and comprehensive patient records. However, this purpose was not wholly achieved. Poor usability impacted the nurses' use of the tool. In support of the study's results, previous studies have demonstrated that the nurses' use of the EHR system was linked to the presentation of data in a way that is understandable to them (Kilmon et al., 2008).
Question 13 of STSSM questionnaire. Question 13 dealt with whether or not the SmartPhrase tool shortened the time of nursing documentation. This question focused on how the SmartPhrase tool could enable nurses to complete tasks accurately within the shortest time possible. The p value was statistically significant (<0.0001) and the mean was 2 (out of 5). The interpretation is the perception of usability of SmartPhrase tool correlated with the time saving during nursing documentation. Of the 18 nurses who answered this question, seven (39%) of their scores were 2, six (33%) nurses had scores that were neutral 3, and 5 (28%) of their scores above 3.
Nurses reported an acceptable (albeit not excellent) usability score of the SmartPhrase tool. Thus, there was some belief that use of the tool would streamline documentation time and make the nurses' job easier.
Usability can be viewed as an opportunity to transform nursing actions in ways that increase their utility. This finding is consistent with that of Jee and Kim (2013) According to a Telmediq team (2017), the most immediate issue of an EHR system was to decrease the time required for documentation and order entry. Sharma (2018) stated that usability was getting the right information in the easiest way. Therefore, when an EHR system has high usability, a health care staff (i.e. nursing) will be able to quickly and safely access pertinent information on their patients (Lopez & Fahey, 2018).

Limitations of the Study
The sample size (n = 23) of this study limits generalizing the results on usability. The sample used for this study was drawn from one cancer center, where the SmartPhrase tool was implemented for telephone triage nurses only. Nielsen (2014) recommended at least 20 end-users would be sufficient to achieve power. Nielsen and Landauer reported that 31% of usability problems can be identified with a single user, and more than 80% of usability problems can be identified with a sample of five users (Nielsen & Landauer, 2017). Thus, the premise was that use of a small sample would be sufficient to identify probable interface design problems of the SmartPhrase tool that would affect its usability and adoption. However, because seven respondents transferred to other locations within the 6-month period, the number of respondents who took both the pretest and posttest were reduced to 16.
An additional limitation was that the responses were based on self-perceptions, which are subjective in nature. A notable event that occurred during the data collection phase was that the nurses were involved in a labor dispute and strike. Whether or not this affected the nurse responses cannot be determined. The subjective nature of the perception of usability relies on nurses sharing their opinions openly without incorporating any existing bias (Hodgson et al., 2018).
Future studies might incorporate additional demographic characteristics (gender, age) or computer literacy (keyboarding experience, use of tools and applications on smart phones) factors. This will permit further investigation, as in exploring how younger, and conceivably more computer literate, respondents would score. In addition, the sample was a convenience sample. The respondents were not randomly selected from the target population at large and were not normally distributed. The years of experience varied widely and, given an insufficient number of respondents within age and experience groups, outliers possibly skewed the results.

Implications of Findings
This study contributes to literature with regard to the usability and adoption decisions of a tool embedded in an EHR system. Furthermore, it provides information in reference to the necessity of evaluating usability of the fit of a new technology designed to aid users complete their tasks effectively and efficiently. Usability results, whether high (found useful) or poor (did not find useful) are an expected part of the evaluation process, and such information needs to be collected and used as constructive feedback. Nursing informatics personnel with knowledge of a tool's characteristics can aid in nursing practice by identifying the extent and limits of the tool for feedback revision. Developing a structured planning approach to evaluating current practice processes can contribute to improving EHR systems and processes.
In nursing practice, SmartPhrase tools with good usability features have significant positive implications. For example, the tool was designed to increase job performance, decrease documentation time, and standardize clinical practice. When usability findings do not match the intent, re-design is a necessary step. Therefore, this study evaluated the SmartPhrase tool's usability to assess whether or not the tool was a clinical fit and permitted the telephone triage nurse to accurately reflect the patient's condition and the clinical services offered. These findings raised questions regarding the extent of the SmartPhrase tool's usability in relation to nurses' job performance.
A compromised SmartPhrase tool has significant negative implications in a nursing clinical setting. For instance, usability issues can cause errors that potentially lead to patient harm and negatively impact attenuation of adoption rates. Ensuring usability is essential for nurses on the front line of healthcare delivery. If usability is unfriendly or not helpful, use of an implemented tool, such as this SmartPhrase, will not result in increased end-user satisfaction and may result in adverse events and unintended negative consequences.

Nursing Informatics
Nurse informatics plays a role in assessing the fit of an EHR system and accompanying tools, such as the SmartPhrase. This study implies an imperative for nursing informatics that the technology must fit the user's practice if the intent is to enhance job performance. This study emphasizes the importance of nursing informatics role in healthcare transformation and asserts that informatics nurses should be in a full partnership with other healthcare professionals in developing and evaluating a system's use.
In reality, the informatics nurse serves as a liaison between technical projects and nursing staff, as well as an advocate and coordinator for ensuring that technology fits the clinical practice. These nurses have the clinical background for understanding clinical documentation, which allows for an evaluation process that reflects pertinent use of the proposed software. The informatics nurse is in a unique position to understand both the clinical and technical sides of SmartPhrase evaluation, and this role is critical to the success of tool adoption. Concepts such as usability are of fundamental interest to the informatics nurse, especially in the enhancement of the relationship between the nurse and the EHR interface. It is the informatics nurse who can best identify the gaps between nursing workflow and EHR design. For instance, in training telephone triage nurses, one identifiable gap is a need for training on how to use a new tool. The training needs to be detailed and provide examples and/or case studies. This training needs to be followed up with refresher courses scheduled at predetermined regular intervals. At this time, feedback should be collected so that the nursing informatics personnel and health information technicians can begin redesign efforts based on informed feedback. Nurses need to be involved, in this case, in choosing the smart phrases that best fit the nurse-patient communication process.

Research
The results of this study supported the rationale that the usability evaluation of the SmartPhrase tool had important implications for patient safety and quality of care.
A tool that is too difficult to learn may be conflictual with established practice routines. Navigation issues were the most reported concern about the SmartPhrase tool's use. For example, the process of searching, selecting, and editing using the SmartPhrase tool was not necessarily easier than writing a short note in the free text field which was the usual documentation practice. It is conceivable that the SmartPhrase structure caused the telephone triage nurses to refrain from performing tasks that are routinely supported by the EHR system.
This study focused on nurses' satisfaction and usability issues with the SmartPhrase tool. It confirmed negative findings reported in previous studies and attempted to determine what factors contributed to success or failure in SmartPhrase tool implementation. At this point, access to pre-text phrases were more useful to the telephone triage nurses with less oncology nursing and telephone experience. With appropriate redesign, the use of high usability smart phrases in the EHR could help telephone triage nurses to expedite prioritizing while still providing emotional care and treatment.
Self-administered questionnaires allow an evaluator to collect data from a representative sample of the population. The SUS questionnaire was used as a summative evaluation of the SmartPhrase tool's usability. The findings supported the use of SUS for both practical and research settings in HIT and suggested that the questionnaire would be useful in collecting information in the phases following implementation. In this study, the majority of the telephone triage nurses found the tool either acceptable (but not excellent) or unacceptable. The results clearly indicated that the SmartPhrase tool was not yet ideal in fulfilling its purpose.
The TTF model has a direct and transparent connection to both usability and the Information System Success model (Delone & McLean, 2003). The variables of efficiency, effectiveness, and satisfaction have the requisite elements to form a relationship for academic research in information system security, usability, and healthcare. The relationship can be used to create a diagnostic tool to address breakdowns in processes that affect job performance, usability, and utilization.

Recommendations for Future Research
Expanding this research to examine changes over time will provide a deeper understanding of the evaluation of human factors related to usability concepts.
Establishing a continual evaluation and feedback loop can provide valuable insight into the development of a SmartPhrase tool which will ultimately fit the telephone triage nurses' workflow. This, in turn, may decrease medical errors toward promoting patient safety.
Conducting post-study open-ended interviews with telephone triage nurses from each hospital in the survey would provide information on how the nurses were using, or were not using, pre-texted smart phrases. One possible consideration for a qualitative study would be conduct open-ended interviews of telephone triage nurses from each hospital in the study. Although this study was limited to usability assessment based on the SUS measurement tool, a more comprehensive usability assessment of SmartPhrase tools could be obtained by using an in-depth survey that evaluates individual usability principles, thereby yielding a more insightful analysis of the phenomenon. Finally, it is important to evaluate performance one-year post implementation. Telephone triage nurses may need more time to be comfortable with the tool and may require refresher courses to enhance usage. To ensure the EHR system is designed to optimize usability, a healthcare organization needs to "test, test, and then test some more" the usability of EHR systems (AHRQ, 2013).

Conclusion
Nurses face usability challenges when new technology is introduced. Usability evaluation of any intended implementation, such as a SmartPhrase tool embedded into the EHR system, should not be neglected. If technology is to be useful, the users must be involved so that the use fits the workflow. If a technology is to be implemented and adopted, it is less costly in the long run to do a thorough usability study and use feedback to redesign or tweak the tool's features than it is to implement a tool that nurses ignore or find cumbersome. The results of the present study provided information as to the usefulness of a SmartPhrase tool to telephone triage nurses in oncology centers. In this case, usability test scores reflected that the tool did not increase self-perceived performance or enhance user satisfaction. However, even negative results provide relevant feedback that nursing informatics can utilize in overcoming barriers to effective use. This study is a step forward in designing and developing information processing tools that enhance nurse performance. This, in turn, permits a decrease in costs to the institution.

Project team, tele-triage nurses
Step 1: Problem identification/evidence assessment for potential translation. Focus groups and interviews with patients and telephone-triage staff nurses supported Press-Ganey patient satisfaction and Lifespan quality data revealing a need to improve patient satisfaction with 7 aspects of symptom management (managing chemotherapy side effects, fatigue management education, managing appetite loss, emotional needs addressed, home-based instructions, pain well controlled, and perceived safety and security, as per Lifespan's Press-Ganey Outpatient Oncology Patient Satisfaction Survey, 2016. Evaluated the effective and efficient telephone-triaging of patients to the most appropriate level of care from self-management to ER admission.
Interviews with telephone-triage nurses and auditing of EHR documentation as part of this quality improvement work also revealed a striking lack of standardized nurses' telephone-triage symptom assessment across the 3 hospital settings and one outpatient clinic. Telephone-triage nurses usually bypassed the EHR assessment tool and free texted their reports on symptom management calls. There was concern that such workarounds could dilute efforts to improve patient safety by making it difficult to capture quality data.
Next, comprehensive CINAHL and Med-line literature searches were conducted. It was found that evidence-based user-friendly approaches, such as use of smartphrases, to improve symptom management through cuing nurses' telephone-triage assessments exist but are not widely used in practice (Teriq et al. 2016). Eight studies were selected for roundtable discussions in step 2.
Step 2: Roundtable discussions to evaluate strength of evidence and generate recommendations for SmartPhrase development. Three roundtables were conducted, one in each of the Lifespan Cancer Institute hospitals. Nine studies were evaluated by leadership members of the quality team, inpatient oncology nurses, telephone-triage nurses, and affiliating senior nursing students and their clinical instructor to determine the strength of evidence using Polit and Beck's criteria (2014). They also evaluated the studies for fit of setting, potential implementation risks, readiness for change among telephone-triage nurses, resources required, and current practice.
Step 3: Design of SmartPhrase by quality improvement team led by the Nursing Safety and Quality Manager of the Lifespan Cancer Center in collaboration with 21 telephone-triage staff nurses. Based on recommendations generated in the roundtable discussions. It is believed that rapid uptake and sustainability may be enhanced by embedded forcing functions, easily-accessed hyperlinks, and pop-ups designed by the triage nurses themselves. Three design options were posed to the telephone triage nurses who then voted on what they believed to be the most useful design, which was then sent to the Lifespan IT team for embedding into the EMR.
With this preliminary quality improvement work having been completed, this dissertation was concerned with evaluating the usability of the SmartPhrase tool (6 months post implementation) and determining non-causal associations between telephone triage nurse self-perceived changes in job performance (from preimplementation to 6-months post) and utilization of the SmartPhrase.
Step 4. Telephone-triage nurses, trained by team educators using "just-in-time" coaching to provide real time feedback and suggestions on using the SmartPhrase, implemented and pilot-tested the SmartPhrase tool.
Step 5: The decision will be made to adopt, alter, or further test the SmartPhrase following data analysis. Following analysis of survey data, a permanent integration into the EHR will be completed.
Step 6: Dissemination of findings and further testing. Further translational projects may provide compelling evidence that such tools as SmartPhrases may significantly increase clinician uptake, sustainability, and impact nurse-sensitive outcomes in other centers. Dissemination through publication in Worldview in Evidence-based Nursing, the Clinical Journal of Oncology Nursing, and presentations at ANCC Magnet, Oncology Nursing conferences are planned.
15. Can you think of anything else that would make it easier, quicker or improve the quality of your assessment of symptoms over the phone?
Thank you for participating in our goal towards excellence in our oncology nursing practice.