Advertisement

Googling endometriosis: a systematic review of information available on the Internet

Published:November 10, 2016DOI:https://doi.org/10.1016/j.ajog.2016.11.1007

      Background

      The demand for health information online is increasing rapidly without clear governance.

      Objective

      We aim to evaluate the credibility, quality, readability, and accuracy of online patient information concerning endometriosis.

      Study Design

      We searched 5 popular Internet search engines: aol.com, ask.com, bing.com, google.com, and yahoo.com. We developed a search strategy in consultation with patients with endometriosis, to identify relevant World Wide Web pages. Pages containing information related to endometriosis for women with endometriosis or the public were eligible. Two independent authors screened the search results. World Wide Web pages were evaluated using validated instruments across 3 of the 4 following domains: (1) credibility (White Paper instrument; range 0-10); (2) quality (DISCERN instrument; range 0-85); and (3) readability (Flesch-Kincaid instrument; range 0-100); and (4) accuracy (assessed by a prioritized criteria developed in consultation with health care professionals, researchers, and women with endometriosis based on the European Society of Human Reproduction and Embryology guidelines [range 0-30]). We summarized these data in diagrams, tables, and narratively.

      Results

      We identified 750 World Wide Web pages, of which 54 were included. Over a third of Web pages did not attribute authorship and almost half the included pages did not report the sources of information or academic references. No World Wide Web page provided information assessed as being written in plain English. A minority of web pages were assessed as high quality. A single World Wide Web page provided accurate information: evidentlycochrane.net. Available information was, in general, skewed toward the diagnosis of endometriosis. There were 16 credible World Wide Web pages, however the content limitations were infrequently discussed. No World Wide Web page scored highly across all 4 domains.

      Conclusion

      In the unlikely event that a World Wide Web page reports high-quality, accurate, and credible health information it is typically challenging for a lay audience to comprehend. Health care professionals, and the wider community, should inform women with endometriosis of the risk of outdated, inaccurate, or even dangerous information online. The implementation of an information standard will incentivize providers of online information to establish and adhere to codes of conduct.

      Key words

      Introduction

      Endometriosis is benign gynecological disease that affects 1 in 10 women of reproductive age. It is characterized by pain and subfertility with associated reduced quality of life.
      • Duffy J.M.N.
      • Arambage K.
      • Correa F.J.S.
      • Olive D.
      • Farquhar C.
      • Garry R.
      Laparoscopic surgery for endometriosis.
      The economic burden of endometriosis is of a similar magnitude to other chronic diseases such as diabetes.
      • Simoens S.
      • Dunselman G.
      • Dirksen C.
      • et al.
      The burden of endometriosis: costs and quality of life of women with endometriosis and treated in referral centers.
      There is a paucity of high-quality research to guide clinical practice; this leads to unwarranted and unjustified variations in patient care.
      • Hirsch M.
      • Duffy J.M.
      • Kusznir J.O.
      • Davis C.J.
      • Plana M.N.
      • Khan K.S.
      Variation in outcome reporting in endometriosis trials: a systematic review.
      The Internet is the source of health information, as patients can access health information quickly, conveniently, and privately. There are an estimated 6.75 million health searches daily in Google representing 4.5% of all searches performed.
      • Eysenbach G.
      • Kohler C.
      What is the prevalence of health-related searches on the World Wide Web? Qualitative and quantitative analysis of search engine queries on the Internet.
      There has been rapid growth in the number of World Wide Web pages providing health information with little or no governance.
      • Mackey T.K.
      • Liang B.A.
      • Kohler J.C.
      • Attaran A.
      Health domains for sale: the need for global health Internet governance.
      Seven in 10 adults regularly search for an explanation and information on a new diagnosis or treatment.
      • Handfield B.
      • Turnbull S.
      • Bell R.J.
      What do obstetricians think about media influences on their patients?.
      • Kaicker J.
      • Debono V.B.
      • Dang W.
      • Buckley N.
      • Thabane L.
      Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument.
      • Hesse B.W.
      • Nelson D.E.
      • Kreps G.L.
      • et al.
      Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey.
      Information provided is commonly written at a high literacy level, compounding the difficulties for patients untrained in establishing whether the information is accurate. Exposure to complex, ungoverned, unfounded health information that lacks expert editorial supervision could negatively affect patient understanding, compliance, and decision making. This could lead to poorer health outcomes, including harm.
      • Cline R.J.
      • Haynes K.M.
      Consumer health information seeking on the Internet: the state of the art.
      • Berkman N.D.
      • Sheridan S.L.
      • Donahue K.E.
      • Halpern D.J.
      • Crotty K.
      Low health literacy and health outcomes: an updated systematic review.
      • Sacchetti P.
      • Zvara P.
      • Plante M.K.
      The Internet and patient education–resources and their reliability: focus on a select urologic topic.
      • McMullan M.
      Patients using the Internet to obtain health information: how this affects the patient-health professional relationship.
      • Benigeri M.
      • Pluye P.
      Shortcomings of health information on the Internet.
      There are no systematic reviews assessing the quality of online patient information pertaining to endometriosis.
      We systematically assessed the accuracy, quality, readability, and credibility of World Wide Web pages providing women with endometriosis and the public information regarding the diagnosis and management of endometriosis.

      Materials and Methods

      Sources

      A protocol with explicitly defined objectives, criteria for World Wide Web page selection, and approaches assessing outcome selection was developed and registered with the International Prospective Register of Systematic Reviews, identification number: CRD42016036134. This review was performed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses statement.
      • Moher D.
      • Liberati A.
      • Tetzlaff J.
      • Altman D.G.
      PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.

      World Wide Web page selection

      We developed a comprehensive search strategy in consultation with health care professionals, researchers, and women with endometriosis. We used a key word analytic instrument (www.semrush.com) to inform our selection of search terms, which provides analytical information related to search terms. We are confident we identified and selected search terms commonly used by women with endometriosis. We used the following search terms: (1) “endometriosis,” 4,560,000 searches per annum; (2) “endometriosis symptoms,” 325,200 searches per annum; (3) “endometriosis treatment,” 64,800 searches per annum; (4) “endometriosis pain,” 19,200 searches per annum; and (5) “endometriosis diagnosis,” 15,600 searches per annum. During March 2016, we searched five popular search engines: aol.com, ask.com, bing.com, google.com, and yahoo.com.
      Individuals rarely examine more than the first 3 pages of a search.
      • Sacchetti P.
      • Zvara P.
      • Plante M.K.
      The Internet and patient education–resources and their reliability: focus on a select urologic topic.
      We therefore extracted the World Wide Web pages from the first 3 pages for each search term within each search engine. Location services were disabled to eliminate geographical bias.
      We organized the extracted World Wide Web pages and removed duplicates. Two reviewers (M.H. and S.A.) independently screened the full content of World Wide Web pages to assess eligibility. All data extraction was performed using piloted data extraction instruments. We pilot tested each instrument using a representative sample of the World Wide Web pages to be reviewed. This testing helped identify data missing from the form, or likely to be superfluous. This allows authors trialing the form to provide feedback that certain coding instructions are confusing or incomplete (eg, a list of options may not cover all situations). Any discrepancies between the reviewers were resolved by discussion with a consensus required before the form is modified to avoid any misunderstandings or later disagreements. We repeated the pilot testing on a new set of World Wide Web pages where no major changes were needed.

      Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. Cochrane Collaboration. Available at: http://www.cochrane-handbook.org. Accessed 02/02/2016.

      We included World Wide Web pages providing health information about endometriosis >300 words in length on the initial page following click through from the search engine. We excluded World Wide Web pages for the following reasons; (1) non-English language; (2) inaccessible, for example password restricted; (3) aimed at a professional audience, for example scientific publication; (4) excessive commercial advertising (≥2 commercial advertisements); and (5) content related solely to the lived experience of endometriosis, for example a patient’s diary or blog.
      Those World Wide Web pages that met the criteria for inclusion were saved as a portable document format for evaluation and data extraction by 2 independent authors (M.H. and S.A.). M.H. and J.M.D. did not assess any World Wide Web pages to which they had previously contributed.

      World Wide Web pages characteristics

      Two reviewers (M.H. and S.A.) extracted the World Wide Web page characteristics independently using a piloted data extraction sheet. Information extracted from each World Wide Web page included country of origin, disease-specific information, listed authors, and privacy statements. Two reviewers (M.H. and S.A.) independently assessed each World Wide Web page using validated instruments including assessments of: (1) credibility assessed using the White instrument

      Ambre J, Guard R, Perveiler FM, Renner J, Rippen H. Criteria for assessing the quality of health information on the Internet. 1997. Available at: http://hitiWeb.mitretek.org/docs/criteria.html. Accessed February 2016.

      anchored between 0 (poor) and 10 (excellent); (2) quality assessed using the DISCERN
      • Charnock D.
      • Shepperd S.
      • Needham G.
      • Gann R.
      DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.
      instrument anchored between 0 (poor) and 85 (excellent); and (3) readability assessed using the Flesch-Kincaid
      • Kincaid J.P.
      • Fishburne Jr., R.P.
      • Rogers R.L.
      • Chissom B.S.
      Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel.
      instrument anchored between 0 (poor) and 100 (excellent). Discrepancies were resolved by discussion.

      Quality assessment

      Two reviewers (M.H. and S.A.) underwent training in the use of the quality assessment instruments. We assessed accuracy using a prioritized list of recommendations included within the European Society of Human Reproduction and Embryology (ESHRE) endometriosis guidelines.
      • Dunselman G.A.
      • Vermeulen N.
      • Becker C.
      • et al.
      ESHRE guideline: management of women with endometriosis. European Society of Human Reproduction and Embryology.
      The ESHRE guideline was selected for comparison as this was objectively assessed to represent the highest quality endometriosis guideline.

      Hirsch M, Begum MR, Paniz É, Barker C, Davis CJ, Duffy JMN. Diagnosis and management of endometriosis: a systematic review of international and national guidelines. Submitted for publication to Am J Obstet Gynecol 2016.

      All recommendations were extracted by 2 authors independently. Discrepancies were resolved by discussion. In consultation with health care professionals, researchers, and women with endometriosis, the recommendations were scored as: (1) critical for decision making, (2) important but not critical for decision making, and (3) not critical and not important for decision making. Fifteen guideline recommendations were selected as statements critical for decision making (appendix). The assessment of accuracy was standardized against selected guideline recommendations. This approach has been utilized in similar research studies.
      • Fisher J.H.
      • O'Connor D.
      • Flexman A.M.
      • Shapera S.
      • Ryerson C.J.
      Accuracy and reliability of Internet resources for information on idiopathic pulmonary fibrosis.
      Two reviewers (M.H. and S.A.) independently reviewed each World Wide Web page assessing the accuracy of information. Each recommendation was scored: 0 (if absent or incorrectly described), 1 (present and incompletely described), or 2 (present and completely described). Accuracy assessment was anchored between 0 and 30. Discrepancies were resolved by discussion. We classified World Wide Web pages with a score ≥20 as accurate.
      The World Wide Web page's credibility was assessed by 2 reviewers independently using the validated White instrument.

      Ambre J, Guard R, Perveiler FM, Renner J, Rippen H. Criteria for assessing the quality of health information on the Internet. 1997. Available at: http://hitiWeb.mitretek.org/docs/criteria.html. Accessed February 2016.

      This instrument, designed for consumers of health information, provides a set of criteria that can be used to accurately and reliably assess the quality of health information on the Internet. Credibility was assessed using 10-point criteria: (1) source; (2) context; (3) currency; (4) utility; (5) editorial review process; (6) hierarchy of evidence; (7) statement of original source; (8) disclaimer, which included ownership, sponsorship, funding, and advertising; (9) omissions; and (10) feedback. Each criterion was scored 0 (absent) or 1 (present) giving a score anchored between 0-10.
      • Tirlapur S.A.
      • Leiu C.
      • Khan K.S.
      Quality of information on the Internet related to bladder pain syndrome: a systematic review of the evidence.
      Discrepancies were resolved by discussion. We classified those World Wide Web pages with a score ≥7 as credible.
      The World Wide Web page's quality was assessed by 2 reviewers independently using a validated instrument, DISCERN,
      • Charnock D.
      • Shepperd S.
      • Needham G.
      • Gann R.
      DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.
      designed to assess the quality of written information on treatment choices that can be applied to any disease,
      • Kaicker J.
      • Debono V.B.
      • Dang W.
      • Buckley N.
      • Thabane L.
      Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument.
      • Charnock D.
      • Shepperd S.
      • Needham G.
      • Gann R.
      DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.
      The DISCERN instrument offers a framework for the production, evaluation, and screening of written consumer health information. This includes 16 questions assessed using a Likert scale anchored between 1 (do not agree) and 5 (agree).
      • Charnock D.
      • Shepperd S.
      • Needham G.
      • Gann R.
      DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.
      Discrepancies were resolved by discussion. We classified those World Wide Web pages as high (>53), moderate (27-52), and low (<27) quality.
      The World Wide Web page's readability was assessed using the Flesch-Kincaid reading-ease test.
      • Kincaid J.P.
      • Fishburne Jr., R.P.
      • Rogers R.L.
      • Chissom B.S.
      Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel.
      This formula presents a score as a US grade level, making it easier for teachers, parents, librarians, and consumers of health information to judge the readability level of various texts. The Flesch-Kincaid score is generated from the following equation: 206.835 – 1.015 (total words/total sentences) – 84.6 (total syllables/total words) (www.readability-score.com).
      • Kincaid J.P.
      • Fishburne Jr., R.P.
      • Rogers R.L.
      • Chissom B.S.
      Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel.
      The scores were anchored between 0 (complex language) and 100 (simple language) and could be categorized by reading age or educational status: 90-100 (5th grade); 80-90 (6th grade); 70-80 (7th grade); 60-70 (8th and 9th grade); 50-60 (10th, 11th, and 12th grade); 30-50 (college); or 0-30 (college graduate). Discrepancies were resolved by discussion.
      A large-scale national assessment of the average US reading level performed by the National Center for Education Statistics found that the typical US citizen reads between a 7th and 8th grade level.

      National Center for Education Statistics. National Assessment of Adult Literacy (NAAL). Available at: http://nces.ed.gov/naal. Accessed Aug. 6, 2008.

      It is recommended that online health information should not exceed the level of US 7th grade writing and reading.
      • Bernstam E.V.
      • Shelton D.M.
      • Walji M.
      • Meric-Bernstam F.
      Instruments to assess the quality of health information on the world wide web: what can our patients actually use?.
      We therefore expected World Wide Web pages to have a readability score level of US education ≤7th grade (>70) to be deemed appropriate for a patient and public audience.

      Analysis

      The World Wide Web page characteristics and assessments were summarized in tabular form and presented with descriptive statistics within summary tables and diagrams.

      Results

      The search strategy identified 750 World Wide Web pages assessed for eligibility. We screened 211 World Wide Web pages following the exclusion of 539 duplicate sites. Two authors independently applied an inclusion and exclusion criteria when screening the pages. We included 54 World Wide Web pages in our final assessment (Figure and Table).
      Figure thumbnail gr1
      FigureFlow of included Web pages
      Flow of included World Wide Web pages.
      Hirsch. Online endometriosis information. Am J Obstet Gynecol 2017.
      Table 1World Wide Web page characteristics and summary of quality, accuracy, credibility, and readability assessment
      ID Web domainCountryListed authorsPrivacy statementQuality
      DISCERN tool to assess quality of information (range 16–80)
      Accuracy
      Accuracy assessed using selected criteria from 2013 European Society of Human Reproduction and Embryology guidelines (range 0–30)
      Credibility
      Credibility based on 10 criteria (range 0–10)
      Readability
      Readability assessed using Flesch-Kincaid reading-ease tool (range 0–100).
      1endocenter.orgUnited StatesNoYes466726.8
      2endometriosis.orgGlobalNoYes6210830.7
      3endometriosis.orgGlobalNoYes5012639
      4endometriosis.orgGlobalNoYes5010838.3
      5endometriosis.orgGlobalYesYes371447.6
      6endometriosis.orgGlobalNoYes427538.5
      7home.bt.comUnited KingdomYesYes465538.2
      8lifestyle.oneUnited KingdomYesNo484552.3
      9medical-dictionary.thefreedictionary.comUnited StatesNoYes6210824.3
      10metro.co.ukUnited KingdomYesNo372361
      11pain.about.comUnited StatesYesYes6113645.9
      12patient.infoUnited KingdomYesNo6910948.1
      13shetrust.org.ukUnited KingdomNoNo352423
      14sogc.orgCanadaNoYes429433.7
      15womenshealth.about.comUnited StatesYesYes422632.6
      16activebeat.comCanadaNoYes281334.8
      17babycentre.co.ukGlobalNoYes4010855.4
      18channel4embarrassingillnesses.comUnited KingdomNoYes322549.8
      19cwhn.ca/node/40781CanadaNoNo434338.5
      20endo-resolved.comUnited KingdomNoNo353438.3
      21endo-resolved.comUnited KingdomNoNo372432.2
      22endo-resolved.comUnited KingdomNoNo545447.3
      23endometriosis.ieIrelandNoYes3910423.5
      24endometriosisaustralia.orgAustraliaNoNo5810549.1
      25endometriosisinstitute.comUnited StatesNoNo508423
      26endometriosisinstitute.comUnited StatesNoNo518421.3
      27evidentlycochrane.netUnited KingdomYesNo454729.6
      28evidentlycochrane.netUnited KingdomYesNo562740.4
      29healthline.comUnited StatesYesYes627840.6
      30hellomagazine.comUnited KingdomNoYes383451.7
      31independent.co.ukUnited KingdomYesYes324546.3
      32livescience.comGlobalYesYes476334.9
      33medicalnewstoday.comUnited KingdomYesYes452824.8
      34netmums.comUnited KingdomNoNo455428.1
      35nytimes.comUnited StatesYesYes515857
      36nzendo.org.nzNew ZealandNoNo406434.2
      37pelvicpain.org.ukUnited KingdomNoYes5711821.5
      38pelvicpain.org.ukUnited KingdomNoYes386533.6
      39prevention.comUnited StatesYesYes353432.8
      40students4bestevidence.netUnited KingdomYesYes472875
      41theguardian.comUnited KingdomYesYes406356.8
      42theguardian.comUnited KingdomYesYes315453.4
      43uptodate.comUnited KingdomYesYes6413933.8
      44womens-health.co.ukNew ZealandNoYes223338.1
      45womens-health.co.ukNew ZealandNoYes355249.3
      46youngwomenshealth.orgUnited StatesYesNo614455.1
      47en.wikipedia.orgGlobalNoYes5011823.9
      48health.facty.comCanadaYesYes321544.9
      49betterhealth.vic.gov.auAustraliaNoYes618730.8
      50endometriosis-uk.orgUnited KingdomNoYes422531
      51endometriosis-uk.orgUnited KingdomNoYes401524.8
      52endometriosis-uk.orgUnited KingdomNoYes412532
      53endometriosis-uk.orgUnited KingdomNoYes530551.6
      54endometriosis-uk.orgUnited KingdomNoYes332548.3
      Median IQR44 (37–51)5 (4–7)5 (2–9)38.2 (30–48)
      IQR, interquartile range.
      Hirsch. Online endometriosis information. Am J Obstet Gynecol 2017.
      a DISCERN tool to assess quality of information (range 16–80)
      b Accuracy assessed using selected criteria from 2013 European Society of Human Reproduction and Embryology guidelines (range 0–30)
      c Credibility based on 10 criteria (range 0–10)
      d Readability assessed using Flesch-Kincaid reading-ease tool (range 0–100).

      World Wide Web page characteristics

      In all, 21 (39%) World Wide Web pages did not report authors and 25 (46%) did not report sources of information or academic references. The majority of included World Wide Web pages (25; 46%) were published in the United Kingdom. All World Wide Web pages presented structured content. Almost two thirds of the World Wide Web pages (38; 70%) reported a privacy statement (Table 1).

      Accuracy

      A single World Wide Web page provided accurate information: evidentlycochrane.net. The median accuracy of included World Wide Web pages was 5 (interquartile range [IQR] 4-7). Included World Wide Web pages contained limited information (Table 1), skewed toward the diagnosis of endometriosis. Information pertaining to the medical or surgical management of pain or infertility associated with endometriosis was poorly represented. The most commonly reported recommendation, that clinicians should consider the diagnosis of endometriosis in the presence of gynecological symptoms such as dysmenorrhea, noncyclical pelvic pain, deep dyspareunia, infertility, or fatigue in the presence of any of the above, was described by four fifths of included World Wide Web pages (43, 80%). The least frequently described recommendations, described by a small minority of included World Wide Web pages (3; 6%) were: (1) in infertile women with endometriosis, clinicians may offer treatment with assisted reproductive technologies after surgery, since cumulative endometriosis recurrence rates are not increased after controlled ovarian stimulation for in vitro fertilization/intracytoplasmic sperm injection; (2) clinicians should inform women with endometriosis requesting information on their risk of developing cancer that (a) there is no evidence that endometriosis causes cancer, and (b) there is no increase in overall incidence of cancer in women with endometriosis; and (3) some cancers (ovarian cancer and non-Hodgkin lymphoma) are slightly more common in women with endometriosis. The delivery of inaccurate, outdated, or dangerous information remains prevalent on World Wide Web pages. Inaccuracies included, first, “Your specialist may also suggest flushing out your blocked fallopian tubes. This procedure is an alternative to surgery and is usually successful” (World Wide Web page ID 17, Table 1). Routine tubal flushing is used in diagnostic evaluation of tubal patency and is not a recommended therapeutic approach.
      • Duffy J.M.N.
      • Johnson N.
      • Ahmad G.
      • Watson A.
      Postoperative procedures for improving fertility following pelvic reproductive surgery.
      A second inaccuracy we found was, “The only reliable way to confirm the presence of the disease is by visually inspecting the abdominal organs by a procedure called a laparoscopy” (World Wide Web page ID 20, Table). There are many difficulties associated with visually confirming endometriosis. The most reliable ways to diagnose endometriosis are laparoscopy, biopsy, and histopathological examination. Visual diagnosis is no longer recommended.
      • Dunselman G.A.
      • Vermeulen N.
      • Becker C.
      • et al.
      ESHRE guideline: management of women with endometriosis. European Society of Human Reproduction and Embryology.
      The third inaccuracy we found was, “It is suspected that between 10-20% of reproductive aged women have the disease” (World Wide Web page ID 20, Table 1). The estimated prevalence within the general population is up to 10%.
      • Dunselman G.A.
      • Vermeulen N.
      • Becker C.
      • et al.
      ESHRE guideline: management of women with endometriosis. European Society of Human Reproduction and Embryology.

      Credibility

      Credibility was defined as a score ≥7. Sixteen World Wide Web pages (29%) were assessed as credible. The median credibility of included World Wide Web pages was 5 (IQR 2-8.8). The highest scoring criteria included context relevant to the disease and originality with all World Wide Web pages fulfilling these criteria. The least frequently described area of credibility was the discussion of content limitations, which was reported by 1 World Wide Web page (Table).

      Quality assessment

      Thirteen World Wide Web pages (24%) were assessed to be high quality, 40 (74%) were assessed to be of moderate quality, and 1 (2%) was assessed as low quality. The highest scoring criteria included describing aims (median 5; IQR 3-4) and being unbiased (median 5; IQR 4-5). World Wide Web pages typically did not describe the consequences of no treatment (median 1; IQR 1-1).

      Readability

      All included World Wide Web pages were assessed as fairly difficult to read (10th, 11th, and 12th grade), difficult to read (college), or very difficult to read (college graduate). The median readability score was 38.2 (IQR 30.7-48.0), indicating an average educational status of a college student would be required to understand the written content (Tables 1 and 2). In all, 45 World Wide Web pages (83%) presented written information at a level at or above college standard.
      Table 2Readability presented by US reading age
      Ease of readingUS educational levelWorld Wide Web pages, n
      Very easy (score 90–100)5th Grade0
      Easy (score 80–90)6th Grade0
      Fairly easy (score 70–80)7th Grade0
      Plain English (score 60–70)8th–9th Grade1
      Fairly difficult (score 50–60)10th–12th Grade8
      Difficult (score 30–50)College32
      Very difficult (score 0–30)College graduate13
      Hirsch. Online endometriosis information. Am J Obstet Gynecol 2017.
      There were no substantial discrepancies between authors in the data extraction of quantitative parameters and we observed very high interrater agreement.

      Comment

      Summary

      There are no World Wide Web pages that provide high-quality, accurate, and credible health information pertaining to endometriosis. Currently, World Wide Web pages contain limited amounts of information that are skewed toward the diagnosis of endometriosis. In the unlikely event that a World Wide Web page reports high-quality, accurate, and credible health information, it is typically written in language that is challenging for a lay audience to comprehend.

      Strengths and weaknesses

      To our knowledge, this is the first study to examine the quality, credibility, accuracy, and readability of patient-focused online information pertaining to the diagnosis and management of endometriosis. We followed a robust, prospective systematic review method with validated instruments to assess the information presented. We evaluated individual World Wide Web pages using four validated instruments in a systematic process, independently performing all assessments in duplicate. We involved women with endometriosis, to inform the research question, design, and delivery of the research study, and its dissemination. All reviewers underwent recommended training prior to commencing the study.
      This study is not without limitations. Limiting the search to the first 3 pages may have resulted in the exclusion of potentially eligible World Wide Web pages, however only 2.6% of people search past Googles’ third page (www.protofuse.com). Included World Wide Web pages were only written in the English language, limiting the generalizability of our findings. The search was conducted while computer location services were disabled, however there may have been regional differences in search results, out of the authors control, which account for the predominance of British World Wide Web pages. We designed and registered this systematic review prospectively with a predefined inclusion criteria and analysis plan. There are few scientific publications that evaluate online information for patients allowing limited precedent to guide our methods. We observed diminishing returns, however this was not quantified. All World Wide Web pages were designed and managed within high-resource countries. This limits the applicability of this research to inform low-resource settings. We did not calculate weighted kappa to explore agreement between authors as the statistical level of agreement required in health research is unclear.
      • McHugh M.L.
      Interrater reliability: the kappa statistic.
      This evaluation is not currently recommended by the Cochrane Collaboration.

      Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. Cochrane Collaboration. Available at: http://www.cochrane-handbook.org. Accessed 02/02/2016.

      We could have conducted in-depth qualitative interviews of women with endometriosis to explore their satisfaction with reading individual World Wide Web pages and evaluate the correlation with accuracy, credibility, quality, and readability.

      Interpretation

      As clinicians we must be aware that patients are increasingly seeking unregulated health information online that shapes opinions and treatment choices. The essence of modern clinical consultations is changing from a reliance on face-to-face interaction to information gathering online prior to seeking professional opinion. In the United States, there are >400,000 endometriosis searches per month in Google alone. We have demonstrated that individual World Wide Web pages are frequently incomplete, inaccurate, and poorly written. This is a barrier to patient education and results in those vulnerable patients who seek reliable information being misinformed. This is of greater importance to nonexpert patients (majority) who may be less able to evaluate the reliability of online information and be susceptible to the bias and inaccuracies contained within. These forays into online information gathering can lead to a breakdown in doctor-patient relationships. Inaccurate online health information can lead to clinicians advocating guideline-supported recommendations different from those read on reputable online sources. This mismatch of information can lead to a breakdown in trust in the clinician-patient relationship.
      A review conducted by the US Office of Disease Prevention and Health Promotion (ODPHP) concluded that the potential for harm from inaccurate online information is significant.
      US Department of Health and Human Services
      Healthy people 2010: understanding and improving health.
      Harm can be: (1) physical, from inappropriate treatments, adverse effects, or untreated disease; (2) emotional, from anxiety or false hope arising from inaccurate diagnostic, prognostic, or therapeutic information; and (3) financial, from costs incurred from unnecessary purchase of ineffective health services or products.
      US Department of Health and Human Services
      Healthy people 2010: understanding and improving health.
      The ODPHP concluded that the Internet is critical to disease prevention, health promotion, and health care because of the increasing amount of information and services available via the Internet. This included a key objective to increase the quality of online health information.
      • Crocco A.G.
      • Villasis-Keever M.
      • Jadad A.R.
      Analysis of cases of harm associated with use of health information on the Internet.
      The readability of a World Wide Web page is an essential facet of online information. Information presented at a standard above patients’ comprehension will limit its ability to inform the patient. Health care professionals should be aware that there is very limited information available to women with endometriosis with basic levels of literacy (indicates skills necessary to perform simple and everyday literacy activities), and therefore directing them to online information is of limited value in informing decision making.
      Many online information rating systems use proxy markers for quality that do not consider the needs and opinions of patients and the public. Meric and colleagues
      • Meric F.
      • Bernstam E.V.
      • Mirza N.Q.
      • et al.
      Breast cancer on the world wide web: cross sectional survey of quality of information and popularity of websites.
      determined World Wide Web page popularity did not correlate well with traditional standards of World Wide Web page quality. Quality of online information is crucial as patients want to know about the risks, benefits, and uncertainty associated with diagnostic and therapeutic options. This information must be accurate to ensure that patients seeking information are gaining correct and complete information about the disease from up-to-date scientific evidence. Without access to good-quality information, patients are unable to make informed choices about their treatment.

      Recommendations

      Health care professionals and the wider medical community are increasingly quizzed by patients regarding health information found online. It is essential that health care professionals acknowledge their position of responsibility and proactively inform women with endometriosis about the risk of outdated, inaccurate, or even dangerous information online. Interactive consultations using online clinical practice guidelines such as those produced by the American Congress of Obstetricians and Gynecologists
      Management of endometriosis. Practice bulletin no. 114.
      or the Society of Obstetricians and Gynecologists of Canada
      • Leyland N.
      • Casper R.
      • Laberge P.
      • Singh S.S.
      SOGC
      Endometriosis: diagnosis and management.
      can provide the basis for clear, concise, evidence-based management discussions. Following consultations, patients should be advised of higher quality and more reliable sources of online information to answer questions they may have forgotten to ask during their limited consultation time.
      While it may sound unrealistic to regulate health information on the Internet, codes of conduct have been developed and implemented. Health on the Net Foundation, based in the United States, provides accreditation to World Wide Web pages, which meet predefined standards related to readability, accessibility, and accuracy.
      • Kincaid J.P.
      • Fishburne Jr., R.P.
      • Rogers R.L.
      • Chissom B.S.
      Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel.
      Information Standard, based in the United Kingdom, assesses online health information to ensure the information is clear, accurate, balanced, evidence based, and up to date. Information produced by the Royal College of Obstetricians and Gynecologists is accredited by this Information Standard (https://www.england.nhs.uk/tis/).
      We acknowledge that regulating health information on the Internet has inherent difficulties as online authors are not bound by the same codes of practice as licensed health care professionals. The implementation of a robust Information Standard internationally will incentivize providers of online information to establish and adhere to codes of conduct ensuring an improvement in the quality of online information. Health care professionals and professional bodies should direct women with endometriosis toward higher quality, more reliable sources of online information. In general, World Wide Web pages that comply with the Information Standard should be prioritized.
      The Internet will continue to increase its role as a provider of online health information. The media by which health information is transferred from source to patient should not compromise the fundamental features of accuracy, credibility, quality, and readability. It would not be tolerated if a health care professional were delivering substandard information in a face-to-face consultation. A strategy is required to improve the standard of online information for women with endometriosis with evident need for the development of patient-focused online information with a robust evidence base. The translation of research from trials or systematic reviews into online sources has a direct pathway currently being delivered by Cochrane in the form of Evidently Cochrane summaries. These World Wide Web pages summarize Cochrane systematic reviews into patient-focused bite-size pieces of information.

      Hirsch M. Hormone therapy for the menopause after endometriosis surgery: friend or foe? Available at: http://www.evidentlycochrane.net/hormonetherapy-menopause-endometriosis-surgeryfriend-foe/. Accessed February 2, 2016.

      Conclusion

      In the unlikely event that a World Wide Web page reports high-quality, accurate, and credible health information, it is typically challenging for a lay audience to comprehend. Health care professionals, and the wider community, should inform women with endometriosis of the risk of outdated, inaccurate, or even dangerous information online. Providers of online information should engage with established codes of conduct, such as the Information Standard.

      Acknowledgment

      We thank Prof Khalid Khan (Queen Mary University of London, United Kingdom) for his educational supervision and Mr David J. Mills for administrative and logistical support.

      Appendix

      Summary of European Society of Human Reproduction and Embryology guidelines for accuracy assessment

      • 1.
        The guideline development group (GDG) recommends that clinicians should consider the diagnosis of endometriosis in the presence of gynecological symptoms such as: dysmenorrhea, noncyclical pelvic pain, deep dyspareunia, infertility, or fatigue in the presence of any of the above.
      • 2.
        The GDG recommends that clinicians confirm positive laparoscopy by histology, since positive histology confirms the diagnosis of endometriosis, even though negative histology does not exclude it.
      • 3.
        Clinicians are recommended to perform transvaginal sonography to diagnose or to exclude an ovarian endometrioma.
      • 4.
        Clinicians are recommended not to use immunological biomarkers, including CA-125, in plasma, urine, or serum to diagnose endometriosis.
      • 5.
        The GDG recommends clinicians to counsel women with symptoms presumed to be due to endometriosis thoroughly, and to empirically treat them with adequate analgesia, combined hormonal contraceptives, or progestagens.
      • 6.
        Clinicians are recommended to prescribe hormonal treatment [hormonal contraceptives (level B), progestagens (level A), antiprogestagens (level A), or gonadotropin-releasing hormone agonists (level A)] as one of the options, as it reduces endometriosis-associated pain.
      • 7.
        When endometriosis is identified at laparoscopy, clinicians are recommended to surgically treat endometriosis, as this is effective for reducing endometriosis-associated pain, ie, “see and treat.”
      • 8.
        When performing surgery in women with ovarian endometrioma, clinicians should perform cystectomy instead of drainage and coagulation, as cystectomy reduces endometriosis-associated pain.
      • 9.
        The GDG recommends that clinicians refer women with suspected or diagnosed deep endometriosis to a center of expertise that offers all available treatments in a multidisciplinary context.
      • 10.
        In infertile women with American Fertility Society/American Society for Reproductive Medicine stage I/II endometriosis, clinicians should perform operative laparoscopy (excision or ablation of the endometriosis lesions) including adhesiolysis, rather than performing diagnostic laparoscopy only, to increase ongoing pregnancy rates.
      • 11.
        In infertile women with ovarian endometrioma undergoing surgery, clinicians should perform excision of the endometrioma capsule, instead of drainage and electrocoagulation of the endometrioma wall, to increase spontaneous pregnancy rates.
      • 12.
        The GDG recommends that clinicians counsel women with endometrioma regarding the risks of reduced ovarian function after surgery and the possible loss of the ovary. The decision to proceed with surgery should be considered carefully if the woman has had previous ovarian surgery.
      • 13.
        Clinicians can prescribe gonadotropin-releasing hormone agonists for a period of 3-6 months prior to treatment with assisted reproductive technologies to improve clinical pregnancy rates in infertile women with endometriosis.
      • 14.
        In infertile women with endometriosis, clinicians may offer treatment with assisted reproductive technologies after surgery, since cumulative endometriosis recurrence rates are not increased after controlled ovarian stimulation for in vitro fertilization/intracytoplasmic sperm injection.
      • 15.
        The GDG recommends that clinicians inform women with endometriosis requesting information on their risk of developing cancer that: (1) there is no evidence that endometriosis causes cancer, (2) there is no increase in overall incidence of cancer in women with endometriosis, and 3) some cancers (ovarian cancer and non-Hodgkin lymphoma) are slightly more common in women with endometriosis.

      References

        • Duffy J.M.N.
        • Arambage K.
        • Correa F.J.S.
        • Olive D.
        • Farquhar C.
        • Garry R.
        Laparoscopic surgery for endometriosis.
        Cochrane Database Syst Rev. 2014; 4: CD011031
        • Simoens S.
        • Dunselman G.
        • Dirksen C.
        • et al.
        The burden of endometriosis: costs and quality of life of women with endometriosis and treated in referral centers.
        Hum Reprod. 2012; 27: 1292-1299
        • Hirsch M.
        • Duffy J.M.
        • Kusznir J.O.
        • Davis C.J.
        • Plana M.N.
        • Khan K.S.
        Variation in outcome reporting in endometriosis trials: a systematic review.
        Am J Obstet Gynecol. 2016; 214: 452-464
        • Eysenbach G.
        • Kohler C.
        What is the prevalence of health-related searches on the World Wide Web? Qualitative and quantitative analysis of search engine queries on the Internet.
        AMIA Annu Symp Proc. 2003; : 225-229
        • Mackey T.K.
        • Liang B.A.
        • Kohler J.C.
        • Attaran A.
        Health domains for sale: the need for global health Internet governance.
        J Med Internet Res. 2014; 16: e62
        • Handfield B.
        • Turnbull S.
        • Bell R.J.
        What do obstetricians think about media influences on their patients?.
        Aust N Z J Obstet Gynaecol. 2006; 46: 379-383
        • Kaicker J.
        • Debono V.B.
        • Dang W.
        • Buckley N.
        • Thabane L.
        Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument.
        BMC Med. 2010; 8: 59
        • Hesse B.W.
        • Nelson D.E.
        • Kreps G.L.
        • et al.
        Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey.
        Arch Intern Med. 2005; 165: 2618-2624
        • Cline R.J.
        • Haynes K.M.
        Consumer health information seeking on the Internet: the state of the art.
        Health Educ Res. 2001; 16: 671-692
        • Berkman N.D.
        • Sheridan S.L.
        • Donahue K.E.
        • Halpern D.J.
        • Crotty K.
        Low health literacy and health outcomes: an updated systematic review.
        Ann Intern Med. 2011; 155: 97-107
        • Sacchetti P.
        • Zvara P.
        • Plante M.K.
        The Internet and patient education–resources and their reliability: focus on a select urologic topic.
        Urology. 1999; 53: 1117-1120
        • McMullan M.
        Patients using the Internet to obtain health information: how this affects the patient-health professional relationship.
        Patient Educ Couns. 2006; 63: 24-28
        • Benigeri M.
        • Pluye P.
        Shortcomings of health information on the Internet.
        Health Promot Int. 2003; 18: 381-386
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        BMJ. 2009; 339: b2535
      1. Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. Cochrane Collaboration. Available at: http://www.cochrane-handbook.org. Accessed 02/02/2016.

      2. Ambre J, Guard R, Perveiler FM, Renner J, Rippen H. Criteria for assessing the quality of health information on the Internet. 1997. Available at: http://hitiWeb.mitretek.org/docs/criteria.html. Accessed February 2016.

        • Charnock D.
        • Shepperd S.
        • Needham G.
        • Gann R.
        DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.
        J Epidemiol Community Health. 1999; 53: 105-111
        • Kincaid J.P.
        • Fishburne Jr., R.P.
        • Rogers R.L.
        • Chissom B.S.
        Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel.
        Naval Air Station, Memphis (TN)1975
        • Dunselman G.A.
        • Vermeulen N.
        • Becker C.
        • et al.
        ESHRE guideline: management of women with endometriosis. European Society of Human Reproduction and Embryology.
        Hum Reprod. 2014; 29: 400-412
      3. Hirsch M, Begum MR, Paniz É, Barker C, Davis CJ, Duffy JMN. Diagnosis and management of endometriosis: a systematic review of international and national guidelines. Submitted for publication to Am J Obstet Gynecol 2016.

        • Fisher J.H.
        • O'Connor D.
        • Flexman A.M.
        • Shapera S.
        • Ryerson C.J.
        Accuracy and reliability of Internet resources for information on idiopathic pulmonary fibrosis.
        Am J Respir Crit Care Med. 2016; 194: 218-225
        • Tirlapur S.A.
        • Leiu C.
        • Khan K.S.
        Quality of information on the Internet related to bladder pain syndrome: a systematic review of the evidence.
        Int Urogynecol J. 2013; 24: 1257-1262
      4. National Center for Education Statistics. National Assessment of Adult Literacy (NAAL). Available at: http://nces.ed.gov/naal. Accessed Aug. 6, 2008.

        • Bernstam E.V.
        • Shelton D.M.
        • Walji M.
        • Meric-Bernstam F.
        Instruments to assess the quality of health information on the world wide web: what can our patients actually use?.
        Int J Med Inform. 2005; 74: 13-19
        • Duffy J.M.N.
        • Johnson N.
        • Ahmad G.
        • Watson A.
        Postoperative procedures for improving fertility following pelvic reproductive surgery.
        Cochrane Database Syst Rev. 2009; 2: CD001897
        • McHugh M.L.
        Interrater reliability: the kappa statistic.
        Biochem Med (Zagreb). 2012; 22: 276-282
        • US Department of Health and Human Services
        Healthy people 2010: understanding and improving health.
        2nd ed. US Government Printing Office, Washington (DC)2000
        • Crocco A.G.
        • Villasis-Keever M.
        • Jadad A.R.
        Analysis of cases of harm associated with use of health information on the Internet.
        JAMA. 2002; 287: 2869-2871
        • Meric F.
        • Bernstam E.V.
        • Mirza N.Q.
        • et al.
        Breast cancer on the world wide web: cross sectional survey of quality of information and popularity of websites.
        BMJ. 2002; 324: 577-581
      5. Management of endometriosis. Practice bulletin no. 114.
        Obstet Gynecol. 2010; 116: 223-236
        • Leyland N.
        • Casper R.
        • Laberge P.
        • Singh S.S.
        • SOGC
        Endometriosis: diagnosis and management.
        J Obstet Gynaecol Can. 2010 Jul; 32: S1-S32
      6. Hirsch M. Hormone therapy for the menopause after endometriosis surgery: friend or foe? Available at: http://www.evidentlycochrane.net/hormonetherapy-menopause-endometriosis-surgeryfriend-foe/. Accessed February 2, 2016.

      Linked Article

      • Assessing standards used to review online health information
        American Journal of Obstetrics & GynecologyVol. 217Issue 6
        • Preview
          Our team read the article1 reviewing web information about endometriosis with interest because our site, youngwomenshealth.org , was reviewed as one of those selected. We applaud the authors in their effort to quantitatively analyze widely disparate information, but we are concerned that some of the methods and standards used were not best suited for evaluating patient-targeted information on endometriosis.
        • Full-Text
        • PDF
      • Reply
        American Journal of Obstetrics & GynecologyVol. 217Issue 6
        • Preview
          We thank Kavanaugh and colleagues for their interest in our systematic review of information available on the Internet.1
        • Full-Text
        • PDF