Assessment and Revision of Clinical Pharmacy Practice Internet Web Sites

Background: Health care professionals, trainees, and patients use the Internet extensively. Editable Web sites may contain inaccurate, incomplete, and/or outdated information that may mislead the public’s perception of the topic. Objective: To evaluate the editable, online descriptions of clinical pharmacy and pharmacist and attempt to improve their accuracy. Methods: The authors identified key areas within clinical pharmacy to evaluate for accuracy and appropriateness on the Internet. Current descriptions that were reviewed on public domain Web sites included: (1) clinical pharmacy and the clinical pharmacist, (2) pharmacy education, (3) clinical pharmacy and development and provision for reimbursement, (4) clinical pharmacists and advanced specialty certifications/training opportunities, (5) pharmacists and advocacy, and (6) clinical pharmacists and interdisciplinary/interprofessional content. The authors assessed each content area to determine accuracy and prioritized the need for updating, when applicable, to achieve consistency in descriptions and relevancy. The authors found that Wikipedia, a public domain that allows users to update, was consistently the most common Web site produced in search results. Results: The authors’ evaluation resulted in the creation or revision of 14 Wikipedia Web pages. However, rejection of 3 proposed newly created Web pages affected the authors’ ability to address identified content areas with deficiencies and/or inaccuracies. Conclusions: Through assessing and updating editable Web sites, the authors strengthened the online representation of clinical pharmacy in a clear, cohesive, and accurate manner. However, ongoing assessments of the Internet are continually needed to ensure accuracy and appropriateness.

Background: Healthcare professionals, trainees, and patients use the Internet extensively.
Editable websites may contain inaccurate, incomplete, and/or outdated information that may mislead the public's perception of the topic.
Objective: To evaluate the editable, online descriptions of clinical pharmacy and pharmacist and attempt to improve their accuracy.
Methods: The authors identified key areas within clinical pharmacy to evaluate for accuracy and appropriateness on the Internet.Current descriptions that were reviewed on public domain websites included: 1) clinical pharmacy and the clinical pharmacist, 2) pharmacy education, 3) clinical pharmacy and development and provision for reimbursement, 4) clinical pharmacists and advanced specialty certifications/training opportunities, 5) pharmacists and advocacy, and 6) clinical pharmacists and interdisciplinary/interprofessional content.The authors assessed each content area to determine accuracy and prioritized the need for updating, when applicable, to achieve consistency in descriptions and relevancy.The authors found that Wikipedia, a public domain that allows users to update, was consistently the most common website produced in search results.
Results: The authors' evaluation resulted in the creation or revision of 14 Wikipedia webpages.However, rejection of three proposed newly created webpages, impacted the authors' ability to address identified content areas with deficiencies and/or inaccuracies.
Conclusions: Through assessing and updating editable websites, the authors strengthened the online representation of clinical pharmacy in a clear, cohesive, and accurate manner.However, ongoing assessments of the Internet are continually needed to ensure accuracy and appropriateness.

INTRODUCTION
Consumers extensively use publicly available electronic media (i.e., the Internet) for a variety of purposes.Four out of five Internet users, representing 59% of all adults in the United States, search for information about health online. 1 In 2010, 8 of the top 10 searches for medical treatments and procedures were medication-related, with the top 4 being medications, including pain relievers, antidepressants, high blood pressure medications, and corticosteroids. 1 Web-based resources provide instant information access to the general public, including patients, caregivers, and healthcare professionals and trainees.The quality of information related to the profession of pharmacy, both accurate and current, is critical to balance the naïveté of the prospective searcher and the potential for biased information.3][4][5][6][7][8][9] A systematic review of 79 studies that evaluated the quality of health-focused websites determined content quality to be problematic in 55 (70%) of the studies; specific problems mentioned included technical criteria, design, readability, accuracy, and completeness. 2Only seven of the studies concluded that quality was not a problem on healthrelated websites.
Pharmacy as a profession is represented widely in electronic media, including the Internet, through a variety of perspectives.1][12] Internet sources may include community pharmacies, drug companies, pharmacy organizations, health-systems, and insurance companies.Each has their own perspective of pharmacy and the message they want to convey, which may or may not include a clinical pharmacy perspective.Clinical pharmacy, as a part of pharmacy, carries specific core competencies for those who practice in it and educate and train students and residents.As defined by the American College of Clinical Pharmacy (ACCP), "Clinical pharmacy is that area of pharmacy concerned with the science and practice of rational medication use." 13 The definition is further organized into the discipline of clinical pharmacy, the clinical pharmacist, and the role of the pharmacist in the health care system. 13ee content websites are far-reaching across the Internet.Although many professional pharmacy organizations and colleges of pharmacy maintain websites internally to ensure quality information, additional sites such as Wikipedia, a public domain, are highly trafficked and rely on the contributions of readers and posters to maintain a high-quality level of editable content. 14,15spite documentation of the reliability of Wikipedia information as compared to other reputable and peer-reviewed publications, the accuracy and up-to-dateness of the information correlates with the contributor and may be subject to misinformation. 16Many additional websites related to clinical pharmacy and the profession are privately maintained and not generally editable by the public, but the authors and webmasters often welcome comments and suggestions to continually update the material.
It has become increasingly clear that the discrepancies in many areas of pharmacy practice, not just clinical pharmacy, on the Internet have impacted the profession (current and future), its cohesiveness, and the ability to positively impact the pharmacy society.Because of the known shortcomings in public electronic media, the ACCP Board of Regents tasked its 2011-2012 Publications Committee with researching the accuracy and appropriateness of clinical pharmacy information on the Internet, with the overarching purpose of strengthening the online representation of pharmacy in a clear, cohesive, and accurate manner.The Publications Committee, which included members appointed by the President of ACCP, represented a diverse group of practitioners at different stages in their careers and who practiced in a variety of clinical pharmacy settings.The committee was asked to use ACCP commentaries, position statements, and white papers as well as other scholarly publications to develop proposed revisions to or creation of new websites where content deficiency, improvement, and/or revision was deemed necessary.

METHODOLOGY
The charge of the ACCP Publications committee was to review current descriptions of clinical pharmacy and pharmacy related subject areas in public electronic media as well as to propose revisions and updates to current websites.Based on recommendations from the ACCP Board of Regents and committee discussions, it was determined that the committee's research focus would be on public domain references including, but not limited to Internet search engines.
Figure 1 outlines the overall process of the committee's actions.
Committee members agreed that it would be important to focus primarily on large, major headings and avoid getting into extensive review of subareas with many definitions.Given the large topic area, the committee decided to focus on the following six content areas: 1) clinical pharmacy and the clinical pharmacist, 2) pharmacy education, 3) clinical pharmacy and the development and provision for reimbursement, 4) clinical pharmacists and advanced specialty certifications/advanced training opportunities, 5) pharmacists and advocacy, and 6) clinical pharmacists and interdisciplinary/interprofessional content.The 6 content areas were deemed most relevant by the committee due to their impact on the clinical pharmacy profession and representation on the Internet with respect to the public's understanding of the profession of pharmacy.8][19][20] Because the goal was to review the selected 6 content areas from the public domain perspective, search methods used by committee members mimicked the methods of a layperson searching for information through use of multiple Internet search engines.The searches, completed independently by subcommittees, each assigned one of the 6 content areas, utilized the search terms developed and agreed upon by the larger committee (Table 1).
Following the initial key term search, the subcommittee reviewed the resulting public and private websites for content area relevancy.The websites' content was assessed for consistency and accuracy along with inclusion of appropriate referencing based on a comparison to reputable sources, such as peer-reviewed publications and/or position papers from national pharmacy organizations, to limit potential reviewer bias.Additionally, some evaluated websites were not consistently updated and many included outdated material.Among all content areas, Wikipedia was the most prominent search result, being one of the first, if not the first, result on the search engines' results pages.Furthermore, Wikipedia is an editable website; therefore, the committee decided to focus their evaluation and efforts towards this Internet source.The subcommittees revised websites as needed and when able, created new websites to address identified gaps.

Internet Searches
For the terminology "clinical pharmacy" and "clinical pharmacist," several websites were found to require changes to descriptions and definitions per the delineation created by the ACCP statement in 2008.There were some websites that included the ACCP definition, such as Wikipedia, which were thereby linked appropriately to several other sites.However, other Internet websites that included definitions were broad and ranged from inappropriate (eg, http://www.UrbanDictionary.com) to an appropriate definition without a reference (eg, http://www.PharmacyJobsOnly.com).The subcommittee prioritized the websites that needed to be updated.
The "pharmacy education" terms (Table 1) were searched broadly and included the overall pharmacy degree along with licensing and accreditation.There were 2 components, "pharmacy education" and "pharmacy student," that were not found on Wikipedia which the subcommittee deemed important for inclusion and creation.Wikipedia also contained several other pharmacy education "definition" pages such as "PharmD Degree," "Pharmacy Licensing," and "Pharmacy Schools."Other websites searched for "pharmacy education" were very general and/or did not reference or link to major accrediting and licensing boards/organizations for pharmacy.The subcommittee deemed this a priority for updating.
The broad search in the content area of development and provision for reimbursement for pharmacy services yielded a multitude of results.Some key terms (Table 1) found results that represented non-pharmacy related terminology.For the results related to pharmacy, some were appropriately referenced (Table 2); however, they were not always the first links.The subcommittee suggested moving these up in the search result priority if possible.On the other hand, other websites were inappropriately linked or linked to an opinion paper. 21For the former, the subcommittee recommended linking or referencing websites such as the American Pharmacists Association (APhA), American Society of Health-System Pharmacists (ASHP), and ACCP which include clinically-relevant terminology (Table 2).For the latter, the subcommittee was unsure of how to remove links on non-public websites, so no action was taken.
The search for the content areas of "advanced specialty certifications" and "advanced training opportunities" were found to be very extensive in the results.Many websites describing certifications were found and deemed to be appropriate (Table 2); however, for advanced training, an immense discrepancy was found in the definitions utilized for postgraduate year 1 (PGY1) and postgraduate year 2 (PGY2) residencies/residents and fellowships.The subcommittee suggested that the ASHP definitions for pharmacy residency terminology should be referenced and utilized universally, as this is the accrediting organization for such programs.
For the "pharmacy advocacy" content area, most websites related to pharmacy/pharmacists and advocacy were developed and maintained by national and state pharmacy organizations.Many advocacy informational links and toolkits were available on these sites.The major theme focused on pharmacist involvement in advocacy and areas such as immunizations, patient advocacy, and drug shortages; all of which can make a difference in patient care.Since a common definition with appropriate references was not found on the Internet, the subcommittee suggested creation of a Wikipedia page that included external links to valuable and reliable websites (Table 2) for the public to easily access.
The final content area evaluated included interdisciplinary services involving clinical pharmacists.Within the search, Wikipedia was the most predominant and recognizable public domain found.The search terms utilized (Table 1) routinely returned information from various universities and institutions.These were not analyzed because they are not publicly available for editing; however, excerpts on clinical pharmacists and their role in interdisciplinary care were overall satisfactory.The subcommittee determined that the Wikipedia page needed to be updated for consistency with ACCP, Accreditation Council for Pharmacy Education (ACPE), and National Association of Boards of Pharmacy's (NABP) clinical pharmacy definitions (Table 2).

Website Development and Updates
Several major omissions in content were found, particularly within the terms of "pharmacy education," "pharmacy students," and "pharmacy advocacy."The committee divided into 3 working groups and created new Wikipedia sites.The individual working groups developed the website language, identified the references, selected based upon their relevancy and reliability, and created the webpage.
The "pharmacy education" webpage was divided into six sections titled: 1) pharmacy education in the United States, 2) admissions, 3) professional coursework, 4) pharmacy  3).These references were chosen because the organizations accredit, test, or support pharmacy education.
The "pharmacy student" webpage was divided into 2 sections titled: 1) pharmacy students and 2) differentiation of the PharmD from other degrees in pharmaceutical sciences.The references used were hyperlinked into the webpage.The references consisted of AACP, ACCP, ASHP, APhA, National Community Pharmacists Association (NCPA), ACPE, and NABP websites (Table 4).These references were chosen because they either explain the role of pharmacy students within the profession or explain standards in student education.
The "pharmacy advocacy" webpage was divided into 3 sections titled: 1) definition, 2) organizations/resources/legislation, and 3) the Surgeon General's report. 22The references used were hyperlinked into the webpage.References consisted of APhA, ACCP, ASHP, and AACP websites targeting pharmacy advocacy (Table 5).Another reference used was the U.S. Surgeon General 2011 report entitled, "Improving Patient and Health System Outcomes through Advanced Pharmacy Practice," which endorses pharmacist-provided patient care.References were chosen because they accurately represent pharmacy practice and provide evidence for the value of clinical services provided by pharmacists.
Following the initial webpage proposal, all committee members reviewed each new webpage.Two members took the initiative to create the new Wikipedia pages on "pharmacy education," "pharmacy student," and "pharmacy advocacy."The 2 members reviewed the directions provided online by Wikipedia, formatting the text appropriately to meet html standards and submitting. 23ring the Wikipedia webpage creation process, the committee encountered some challenges.The proposed page describing pharmacy education in the United States was denied because it read more like a "how to guide" than an encyclopedic article.Additionally, the page describing the pharmacy student was said to include "not reliable" references.To address these issues, the committee integrated the proposed "Pharmacy Education in the United States" and "Pharmacy Student in the United States" webpages into the already existing "Pharmacy School" Wikipedia page, which contained similar content.Also incorporated into this page was an updated and correct AACP link to the List of Pharmacy Schools' page.The committee was also not able to create the "Pharmacy Advocacy in the United States" Wikipedia page as intended.The page's creation was denied on the basis that it seemed more like an advertisement than an educational piece.
Despite the setbacks, the committee successfully updated 11 Wikipedia webpages (Table 6).
Revisions to the webpages included updating definitions, including and citing references, providing reliable website links for further information on the given topic, adding hyperlinks to other Wikipedia pages, and/or modifying the description of clinical pharmacists and their role in patient care.In making the updates, the committee acted on its recommendations

CONCLUSION
Through the committee's website review process, deficiencies in content and topics requiring revisions were identified.The committee successfully updated several Wikipedia websites; however, the creation of three websites was denied following review by Wikipedia editors.Despite, the committee's best efforts to improve the online representation of pharmacy in a clear, cohesive, and accurate manner, limitations of the review exist.

Limitations
There are multiple ways to approach such an endeavor; our process is one such approach.
The ability for another to replicate this process and come up with the exact results is unlikely as Internet search engine results change frequently.The order in which website results appear following a search using Internet search engines often depend on several factors, including location and frequency of use of key words, volume of traffic to the website, recent content updates, and payment for search engine advertisement. 24 the planning process, the committee did not consider its ability to and importance of sharing its findings.Therefore, the subcommittees did not record the order in which reliable websites ended and inaccurate websites appeared following use of Internet search engines.While the committee acknowledges that it would be favorable to move the identified reputable websites up in priority, they have no control over the populated result listing.
The committee did not evaluate in detail specific types of found misinformation, as there was no intent to disseminate this type of finding; therefore, they are unable to quantify the identified misinformation on the Internet.Additionally, the committee is unable to provide information on the probability that a layperson would use misinformation rather than accurate information in the Internet search process.It is anticipated that the more familiar one is with a topic the more he/she would be able to detect misinformation or deem a website "unreliable." The committee's review, assessment, and identification of inaccurate references and/or incorrect links, led to limited change.Some websites were private or blogs and the committee was unsure of the ability to make edits.Therefore, committee focused on editable websites, as this was an area that could be updated and/or modified as deemed necessary.

Discussion
With an array of information easily accessible and ever changing on the Internet, the quality of information varies based on the frequency of its maintenance, diligence of the webmaster/editor and/or overseeing organization for accuracy, and the overall need for content updates.Editable websites allow Internet users to update pages with little or no oversight.Thus, the potential for pages to contain inaccurate, incomplete, and/or outdated information depending on the oversight and maintenance of the website exists.
While the committee's charge focused on reviewing current descriptions of clinical pharmacy and pharmacy related content areas in public electronic media and proposing content revisions, the longevity of keeping content current should be considered.At the heart of public electronic media lies the idea of ongoing collaboration among Internet users; therefore, the accuracy of information needs to be continually evaluated and revised as necessary.The ACCP committee was formed for the 2011-2012 year to complete the set charge; therefore, there is no ongoing review of updated content.
Following completion of the committee's review, the committee members chose to disseminate its details so that others' are aware of the presence of misinformation and the need to continually assess and update, when possible.Pharmacists are responsible for maintaining a highly regarded professional reputation.It is too often that pharmacists are overlooked as key players in patient care.With a rapidly changing health system, pharmacists need to be distinguished as vital members of the patient's medical team.
Thus, as Internet users, pharmacists need to take the responsibility to correct any identifiable misinformation by providing accurate statements and/or reliable references.In doing so, the profession will be accurately represented in electronic public media, as the public relies heavily on these resources for information.Based on the committee's experience, the process of updating a website is rather feasible and requires a few minutes to update information pertinent to the pharmacy profession to maintain its integrity.
While ongoing monitoring of public electronic media content can be a time-consuming task, perhaps, each pharmacist could take a few minutes every week to review editable websites.
Additionally, pharmacists could advise national pharmacy organizations to dedicate resources to reviewing and editing websites with focused content on clinical pharmacy and related subject areas.In doing so, as a profession, pharmacists can achieve the goal of continuously maintaining the highly respected professional standing. 25r similar endeavors, the committee acknowledges the need for inclusion of specific results, particularly from a data standpoint.This can support the need for continual monitoring and updating to ensure accuracy and a favorable perception of the profession and its role in patient care.

Conclusion
The Internet contains a vast array of information that may or may not be appropriately representative and/or accurate of clinical pharmacy.Following the completion and evaluation of a thorough search, the authors strengthened the online representation of clinical pharmacy and pharmacy related subject areas through updating editable websites.Despite this feat, ongoing assessments of the Internet are continually needed to ensure accuracy and appropriateness for pharmacy-related topics.
licensure, 5) post-graduate training, and 6) external links suggested.The references used were hyperlinked into the webpage.The references consisted of ACPE, NABP, North American Pharmacist Licensure Examination (NAPLEX), Multi-state Pharmacy Jurisprudence Exam (MPJE), American Association of Colleges of Pharmacy (AACP), ACCP, ASHP, and APhA websites (Table

Table 1 .
Key Search Terms used within each Content Area

Table 2 .
Reliable Websites and References

Table 6 .
Updated Wikipedia Webpages