Systematic Reviews: Written by reviewers who use explicit and rigorous methods to identify, critically appraise, and synthesize relevant studies from the published medical research. They use the process of systematically locating, appraising and synthesizing evidence from scientific studies in order to obtain a reliable overview. The function of a systematic review is: 1) to summarize the literature and 2) to provide new information that may not be readily apparent from individual studies where the effects are small, but become apparent in when the data from many studies are pooled together. Example: Cochrane Database of Systematic Reviews.

Meta-analysis: A type of systematic review that is an overview and also uses quantitative methods to summarize the results. A quantitative method of combining the results of independent studies (usually drawn from the published literature) and synthesizing summaries and conclusions which may be used to evaluate therapeutic effectiveness, plan new studies, etc., with application chiefly in the areas of research and medicine. Any study with the Level 1 ranking in ODG must have been accepted for publication in a peer reviewed journal, and that journal must be one of the journals accepted for inclusion in MEDLINE^® by the National Library of Medicine. For this Journal Selection Criteria, see www.nlm.nih.gov/pubs/factsheets/jsel.html. Unpublished studies, or studies in magazines that do not publish original research, would not receive this ranking.

2. Controlled Trial – Randomized (RCT) or Controlled

These are analytical experimental studies, where variables can be better controlled on a prospective basis. In a RCT (Randomized Controlled Clinical Trial), a group of patients is randomized into an experimental group and a control group. These groups are followed up for the variables/outcomes of interest. Advantages: Unbiased distribution of confounders; Blinding more likely; Randomization facilitates statistical analysis. Disadvantages: Expensive: time and money; Volunteer selection bias; Ethically problematic at times. Any study with the Level 2 ranking in ODG must have been accepted for publication in a peer reviewed journal, and that journal must be one of the journals accepted for inclusion in MEDLINE^® by the National Library of Medicine. Unpublished studies, or studies in magazines that do not publish original research, would not receive this ranking.

3. Cohort Study - Prospective or Retrospective

Analytical observational studies involving identification of two groups (cohorts) of patients, one which did receive the exposure of interest, and one which did not, and following these cohorts forward for the outcome of interest. Advantages: Ethically safe; Subjects can be matched; Can establish timing and direction of events; Eligibility criteria and outcome assessments can be standardized; Administratively easier and cheaper than RCT. Disadvantages: Controls may be difficult to identify; Exposure may be linked to a hidden confounder; Blinding is difficult; Randomization not present; For rare disease, large sample sizes or long follow-up necessary. Any study with the Level 3 ranking in ODG must have been accepted for publication in a peer reviewed journal, and that journal must be one of the journals accepted for inclusion in MEDLINE^® by the National Library of Medicine.

4. Case Series

Analytical observational studies involving identifying groups of patients who have the outcome or treatment of interest (cases) and quantifying the results. Ideally, control patients without the same outcome are also tracked, looking back to see if they had the exposure of interest. (The use of controls would influence the quality rating of a Case Series.) Generally, since the minimum ODG quality rating for studies (“c”) requires at least 10 cases, there must be 10 or more cases for a study to be classified as a Case Series, and otherwise the article would be classified in ODG as Case Reports and Descriptions. Advantages of Case Series: Quick and cheap; Only feasible method for very rare disorders or those with long lag between exposure and outcome; Fewer subjects needed than cross-sectional studies. Disadvantages: Reliance on recall or records to determine exposure status; Confounders; Selection of control groups is difficult; Potential bias: recall, selection. Any study with the Level 4 ranking in ODG must have been accepted for publication in a peer reviewed journal, and that journal must be one of the journals accepted for inclusion in MEDLINE^® by the National Library of Medicine.

5. Unstructured Review

Descriptive (versus analytical) and observational (versus experimental) studies, written by reviewers who describe current practice as well as relevant studies from the published medical research, with no attempt to pool the results analytically. Compared to Systematic Reviews, an Unstructured Review makes little attempt to quantify outcomes based on the body of evidence described. Any study with the Level 5 ranking in ODG must have been accepted for publication in a peer reviewed journal, and that journal must be one of the journals accepted for inclusion in MEDLINE^® by the National Library of Medicine.

6. Nationally Recognized Treatment Guideline (from guidelines.gov)

Accepted for inclusion in the National Guideline Clearinghouse by the Federal Agency for Healthcare Research & Quality (AHRQ), which requires that the guideline recommendations be based on a systematic literature search and review of scientific studies published in peer reviewed journals, and revised on a regular basis to maintain currency with new studies.

7. State Treatment Guideline

Treatment guidelines created for use in a specific state in the U.S., or for use in a province in Canada, or for use by another governmental entity, and they have the backing of the respective jurisdictional or governmental authority.

8. Other Treatment Guideline

Other treatment guidelines. These are typically national treatment guidelines not accepted in the National Guideline Clearinghouse, in many cases because the guideline publishers have chosen not to apply for inclusion (for example, commercial guidelines such as Milliman, McKesson, InterQual, etc.), or because they are private guidelines created for use under the terms of a specific health insurance policy (for example, Blue Cross, Medicare, Aetna, Cigna, United Healthcare, etc.). Since studies by healthcare insurers are generally given a rating of Level 8, they are not characterized in ODG as among the highest quality references when there are numerous other studies available. However, when there are limited studies available with the high quality ratings, it may be necessary to identify other studies that could provide guidance on a subject. In fact, many of the healthcare insurance provider structured reviews are very high quality, they represent a thorough analysis and quantitative weighting of all available evidence on a subject, including unpublished studies that the insurer may have conducted, and these healthcare insurance reviews might even rank as Level 1 if they were published in the peer-reviewed literature and available in MEDLINE^®. Furthermore, the fact that a particular treatment is either covered or not covered by healthcare insurance should be relevant to coverage decisions in workers’ compensation.

9. Textbook

Medical reference texts, which may represent standards of practice, but which in and of themselves, are not necessarily evidence based versus consensus based or based primarily on the personal experiences of the authors.

10. Conference Proceedings/Presentation Slides

These are studies that have not been published in peer reviewed journals.

11. Case Reports and Descriptions

Descriptive articles published in the peer reviewed journals covering individual cases, and lacking any comparisons to controls. Generally, since the minimum ODG quality rating for studies (“c”) requires at least 10 cases, there must be 10 or more cases for a study to be classified as a Case Series, and otherwise the article would be classified in ODG as Case Reports and Descriptions. These articles were not included in the evidence base for any treatment guidelines except for the Council on Chiropractic Guidelines for Practice Parameters (CCGPP) chiropractic practice guidelines.

Ranking by Quality within Type of Evidence (treatment procedures):

In evaluating clinical trials ODG has adopted the standards from the "Cochrane Handbook for Systematic Reviews of Interventions," as updated in September 2006. (Higgins, 2006) Specific additional criteria used by ODG include the following:

a. High Quality

Sample size: Generally over 300, but at least 100, depending on other factors below.

Conflict of interest: Authors and researchers had no financial interest in the product or service being studied.

Study design: Ideally, blinded. No identifiable bias, including recall bias, confounding factors, selection bias, compliance bias, non-response bias, or measurement bias. If a case series, should be a case control series.

Statistical significance: 99% Confidence level that the outcomes likelihood ratio will not cross 1.0 (i.e., the p value is .01).

b. Medium Quality

Sample size: From 20-50 up to 100-300, depending on other factors below.

Conflict of interest: Authors and researchers had no financial interest in the product or service being studied.

Study design: No significant bias, including recall bias, confounding factors, selection bias, compliance bias, non-response bias, or measurement bias. If a case series, should be a case control series.

Statistical significance: 95% Confidence level that the likelihood ratio will not cross 1.0 (i.e., the p value is .05).

c. Low Quality

Sample size: Generally under 20-50, depending on other factors below, but no less than 10.

Conflict of interest: Authors and researchers may have had some financial interest in the product or service being studied, even if the sample size was large.

Study design: Some obvious bias, including recall bias, confounding factors, selection bias, compliance bias, non-response bias, or measurement bias.

Statistical significance: Does not meet the 95% Confidence level that the likelihood ratio will not cross 1.0 (i.e., the p value is .05).

E valuating the Body of Evidence:

From the ODG-TWC Background & Description: While ODG has a 30-step alphanumeric rating system for each individual referenced study, ODG describes and summarizes the entire body of medical evidence within the Procedure Summary topic, as support for the overall ODG recommendation on a topic, rather than using a simplistic alphanumeric rating system for the body of evidence. This is important for utilization review and in states that have mandated ODG, where a clear unambiguous ODG recommendation is required, but providers still have an opportunity to fully understand the complete body of evidence along with the relative quality of studies in support of that. Furthermore, summarizing the body of evidence in this fashion allows ODG to take into consideration other factors in addition to study quality, such as: (1) trade-off between risks versus benefits; (2) magnitude of effect of an intervention; (3) availability of dependable sources of the treatment; (4) education and experience of providers; (5) consistency of study outcomes; & (6) variability of treatment parameters being studied. [This paragraph is copied from the original ODG-TWC Background & Description Chapter, in the Summaries of Medical Studies section.]

In evaluating the entire body of evidence, the ODG ranking for a specific study, as explained in the Ranking by Type of Evidence section, would determine its importance in formulating the treatment guideline summaries and recommendations, as well as whether the study is even mentioned in the ODG Procedure Summaries. In other words, there will be studies that would qualify to be included in ODG, but which may not be specifically described or cited. This could happen for lower quality studies when high quality studies were available, especially if the higher quality studies are more current. Unfortunately, for many treatments high quality studies are not available, and then ODG will need to consider whatever evidence there is, even if the quality is not what would be preferred. But if there are recent, high quality studies, such as prospective randomized controlled trials (rated 2 in ODG), there is no reason for ODG to discuss additional low quality evidence, such as retrospective case reports (rated 11 in ODG), which may be very numerous, but which would be trumped by the higher quality evidence. Besides being unnecessary to support the guidelines, such a discussion might be confusing to the ODG user. There is also another common situation where a relevant study may not be discussed individually in ODG. The highest quality evidence of all, based on the ODG ranking system, is a systemic review or meta-analysis (rated 1 in ODG), such as a Cochrane Systematic Review, that combines the results of multiple clinical trials. When these systemic reviews are discussed in ODG, their conclusions already reflect the results of every study in the review, so it may be unnecessary for ODG to discuss each study separately. This is an important consideration since a systematic review may cover dozens, or even hundreds, of separate published clinical trials. However, ODG still might discuss a specific clinical trial already covered in a structured review if there was additional information relevant to the ODG recommendation.

It should be noted that there are many different methods to rate the overall body of evidence. In addition, the types of studies available may be different depending on the study objectives. The ODG methodology for rating studies is focused on medical treatments because these are medical treatment guidelines. Other types of studies, for example Prognostic, Diagnostic, or Economic studies, may lend themselves to somewhat different rankings, as well as different methodologies for evaluating the overall body of evidence. The other categories (Prognostic/Diagnostic/Economic) generally do not have an equivalent to the Randomized Controlled Trial (since they are not a treatment trial). Prognostic studies provide valuable evidence on predictors, risk factors and other epidemiological information, but RCTs are not generally used due to ethical considerations, i.e., it would be unethical to deliberately subject a control group to a potentially harmful exposure to definitively resolve causality issues. When evaluating Prognostic, Diagnostic, or Economic recommendations, ODG users should be aware of alternative ways to consider the body of evidence behind these recommendations. One good example of a rating methodology covering all types of studies, borrowed from the JBJS system, is shown below. (JBJS, 2003)

Levels of Evidence for Primary Research Question (from JBJS)
	Types of Studies
Level	Therapeutic Studies	Prognostic Studies	Diagnostic Studies	Economic Analyses
I	- High-quality randomized controlled trial with statistically significant difference or no statistically significant difference but narrow confidence intervals - Systematic review of Level-I randomized controlled trials (and study results were homogeneous)	- High-quality prospective study (all patients were enrolled at the same point in their disease with ≥80% follow-up of enrolled patients) - Systematic review of Level-I studies	- Testing of previously developed diagnostic criteria in series of consecutive patients (with universally applied reference "gold" standard) - Systematic review of Level-I studies	- Sensible costs and alternatives; values obtained from many studies; multiway sensitivity analyses - Systematic review of Level-I studies
II	- Lesser-quality randomized controlled trial (e.g., <80% follow-up, no blinding, or improper randomization) - Prospective comparative study - Systematic review of Level-II studies or Level-I studies with inconsistent results	- Retrospective study - Untreated controls from a randomized controlled trial - Lesser-quality prospective study (e.g., patients enrolled at different points in their disease or <80% follow-up) - Systematic review of Level-II studies	- Development of diagnostic criteria on basis of consecutive patients (with universally applied reference "gold" standard) - Systematic review of Level-II studies	- Sensible costs and alternatives; values obtained from limited studies; multiway sensitivity analyses - Systematic review of Level-II studies
III	- Case-control study - Retrospective comparative study - Systematic review of Level-III studies	- Case-control study	- Study of nonconsecutive patients (without consistently applied reference "gold" standard) - Systematic review of Level-III studies	- Analyses based on limited alternatives and costs; poor estimates - Systematic review of Level-III studies
IV	Case series	Case series	- Case-control study - Poor reference standard	- No sensitivity analyses
V	Expert opinion	Expert opinion	Expert opinion	Expert opinion

This table above is an example of a numerical rating system for the body of evidence, but it does not apply directly to ODG since ODG does not use an alphanumeric system (such as Level I through V above) to rate the overall body of evidence, as explained above in the original ODG statement of purpose. However, ODG users should be aware that the individual studies that are rated in ODG may be of necessity lower for Prognostic, Diagnostic, or Economic recommendations. Randomized Controlled Trials (rated 2 in ODG) would be much more likely to be found in studies of medical treatments, which is the primary focus of ODG. While there may be major differences in the overall body of evidence supporting a recommendation in a treatment guideline, and users can glean this from the Procedure Summary discussions in ODG, a numerical rating system would be counter-productive. Where ODG is adopted by rule or law, or where it is the evidence-based standard required by a payer, both medical providers and utilization reviewers will rely on the final ODG recommendation in determining whether a treatment will be approved or whether pre-authorization is recommended, regardless of any possible rating system applied to the body of evidence. For more information on this process, see the section, “Documenting Exceptions to the Guidelines.” Because of this, the burden is on ODG to arrive at the best possible recommendation, whatever the quality of evidence available to support this recommendation. Along the same lines, an absence of evidence is not necessarily evidence of absence proving that a treatment should never be used.

L ink between evidence and recommendations:

ODG Treatment is being updated at least quarterly on the Web. The Contents page indicates the last date updated for each chapter. The hard copy version is published once a year, but this is not recommended since it does not link into the actual studies, it does not contain the ODG UR Advisor® or other tools using complete reimbursement codes, and it is not as current as the Web version. The heart of each chapter in ODG Treatment is the "Procedure Summary", which provides a concise synopsis of effectiveness, defined clearly as “Recommended,” “Not Recommended,” or “Under Study,” based on existing medical evidence, hyper-linked directly into brief, evidence-based summaries of the studies on which the conclusions are based, which have been ranked, highlighted and indexed. The "Treatment Planning" section identifies the ideal treatment plans that may be followed after illness or injury, based on the "Procedure Summary". "Codes for Automated-Approval" maps procedure codes to ICD-9 diagnosis codes based on the ideal treatment protocol, with a field for “maximum occurrences”, for auto-approval of charges that meet the guideline. For example, in the Low Back chapter, under Fusion, it says, "Not recommended in the absence of fracture, dislocation, or instability", so the Treatment Protocol does not include fusion. Same for IDET, facet injections, etc. Under Epidural injections, it says, "Recommended as an option prior to surgery when there are radicular signs… and the number of injections should be limited to two...", so the Treatment Protocol for "With Radiculopathy" includes 2 ESI's, and the Codes for Auto Approval includes CPT code 62311 (Epidural steroid injection) 2 times for ICD9 722.x (Intervertebral disc disorders). [Note: These examples above are illustrative of the process, but they are not meant to replace a current version of the guidelines, which may not agree with these illustrative examples if updates have been made to those sections.] This effort to translate the evidence into specific auto-authorization protocols is unique, for pre-approval of treatment plans and triage of claims management. Of course, most cases will not meet this ideal protocol, and that is where the many other listings in the Procedure Summary come into play. In a recent pilot use of these Codes for Auto Approval reduced medical costs by 64%, cut lost days by 69%, minimized treatment delays for injured workers, and drew considerable praise from providers. (Ohio ODG Pilot, Comp Management, 2005)

P rocess for suggesting ODG updates: The ODG process for incorporating suggestions from the public is both inclusive and transparent. [Note: The ODG ongoing regular internal process for updating the guidelines is described in detail in Appendix B, Methodology Description using the AGREE Instrument, http://www.odg-twc.com/odgtwc/ODG_AGREE.htm. Here only the public input process is described.] The public updating suggestion process is document-based i.e., driven by high quality published studies as described above in the Explanation of Medical Literature Ratings. In-person meetings, telephone conferences, or other verbal presentations will not be accepted.

Suggestion submission process outline:

Outside parties with suggestions for change are asked to copy the current procedure summary entry in ODG. ODG requests that the submitting party use Track Changes to highlight their suggestions.
Submit any high quality scientific studies supporting their suggestion:

· The submitter should determine that a submitted study is not already referenced in ODG either as a stand-alone reference or as part of the references included in a Systematic Review or Meta Analysis.

· If a study is not found in ODG and meets ODG’s criteria for inclusion, i.e., the study has been accepted for publication in a peer reviewed journal, and that journal is one of the journals accepted for inclusion in MEDLINE^® by the National Library of Medicine, then WLDI will review and rank the study/ies and circulate them, together with the suggested revision, to topic-specific subject matter experts before considering any updates. (For complete Journal Selection Criteria, see www.nlm.nih.gov/pubs/factsheets/jsel.html.)

· Send suggestions for change(s) and any high quality scientific studies supporting their suggestion:

o Via email to the ODG Helpdesk at odg@worklossdata.com

o Via fax to (760) 753-9995 or

o Via US Mail to: Managing Editor, Work Loss Data Institute, 169 Saxony Road, Suite 101, Encinitas, CA 92024.

· All suggestions will be acknowledged upon receipt via email, fax or US Mail in accordance with the method used for the suggestion submission.

The ODG subject matter experts on any topic would comprise a subset of the ODG Editorial Advisory Board, consisting of those members who have contributed to, or expressed interest in, that topic in the past. For some topics, such as spinal fusion, there may be as many as 30 subject matter experts, whereas for others there would be a smaller group. The list of subject matter experts could also include others who are not even listed on the ODG Advisory Board, if these people are world-renowned experts in a field and have made comments to ODG in the past, but they do not want to assume the broader responsibilities of being a member of the ODG Board.

When circulating suggestions to the ODG contributors who are subject matter experts, ODG medical editors will include relevant study information including ODG’s proprietary ranking system for those studies as described above, if that information is not already available This is a modified Delphi process, which means that the positions taken by individual contributors are not made publicly available. This policy is also important to protect individual ODG contributors from personal repercussions, including personal legal liability, undesired solicitations, and personal attacks. What becomes public is the final ODG determination that becomes part of the guidelines.

When universal agreement among the ODG subject matter experts is obtained to make a change, based on new high quality scientific evidence, that change will be incorporated into ODG. If there is disagreement among these subject matter experts, then the change will need to be reviewed by the entire board, and a change will require support from at least 80% of the members. If there were universal agreement not to make the change, then it would not be accepted.

· The time required for ODG’s external suggestion review process varies considerably. Minor wording improvements for usability and clarification, or adding a new reference which further supports the existing ODG conclusion, can take as little as a few days, whereas a change in overall recommendation for a major treatment could take up to a year, depending on the degree of medical controversy.

· Submitters interested in obtaining information on the status of their submission should contact the ODG Helpdesk via email at ODG@worklossdata.com or call the ODG Helpdesk at 800 488-5548. Inquiries may be given a status of a) in queue for review; b) in internal ranking & review process; c) in circulation among subject matter experts d) in final update/review process.

· After updates have been made to ODG and noted in the update log file, ODG will send the individual/s suggesting revision a final advice as to the outcome of their submission, including whether the referenced study has been accepted or rejected for inclusion and what if any change or update has been made as a result of his/her submission.

· A formal advice or multiple progress reports may be sent from ODG to the commenter in situations which may require a more lengthy review because:

o a significant or highly technical change to a specific topic area may be under consideration;

o there is controversy among the subject matter experts as to whether or not a change should be made, warranting an in-depth advisory board review; or

o newly emerging technology or body of evidence is forthcoming that could have a significant impact within a procedure summary.

· In these instances, the letter is sent from the entire ODG Board with no individual author singled out. This public suggestion process is an important updating tool for ODG. Because ODG Treatment on the Web gets millions of hits per year, the shear volume of ODG users has resulted in a potent force for suggestions that continues to make ODG better, when these users email the ODG Helpdesk with suggestions or requests for clarification or help when they cannot find a topic. Included within these suggestions would be feedback from stakeholders in states that have proposed adopting or already adopted ODG.

· A submitter may track ODG updates by viewing the ODG update log located at http://www.odg-twc.com/odgtwc/ODG-TWC_updates.xls. For complete details on tracking updates, see full explanation below including some state-specific log file addresses:

T racking ODG updates: The ODG update and change process is as transparent as possible. In addition to reviewing the Contents page, which indicates the last date updated for each chapter, there are a variety of methods to obtain more detail on these updates. A log file is maintained on the ODG Site using Microsoft Excel worksheets, which includes four columns: (1) the Date the change or update was posted (month, day and year), (2) the Chapter containing the change (e.g., Ankle, Pain, etc.), (3) the Section within that chapter (usually a Topic listing in the Procedure Summary, e.g., Physical therapy, Fusion, etc.), and (4) what the Change was (e.g., identifying if it is a new topic, or a new topic subheading, or a wording clarification to avoid ambiguity, or identifying new studies added to the ODG evidence base; if the change was adding new studies, then the additional text added to ODG from that study can be found immediately preceding that reference in ODG). This update log file is located at http://www.odg-twc.com/odgtwc/ODG-TWC_updates.xls. It is updated and posted whenever an update is made to any chapter in ODG Treatment. In addition, TDI (the Texas Department of Insurance), Division of Workers Compensation, produces Monthly Updates to Official Disability Guidelines - Treatment in Workers Comp that they post on their Site at http://www.tdi.state.tx.us/wc/dm/documents/odgupdates.pdf. The TDI update list is more robust than the list on the ODG Site, but it is only updated at the end of each month. According to DWC, “these monthly updates are for informational purposes only and are not a substitute for the ODG.” When the TDI change log only identifies adding a reference or references, e.g., (Weinstein, 2008), the new ODG text can be located immediately preceding the reference that is Cited in the text, so there should not be need for further information to locate this text. Of course, if someone does not have access to ODG, they could not get the complete information, but the DWC posting is clear that their posting should not be a substitute for using ODG. Another state, Kansas, also provides Monthly Updates to Official Disability Guidelines - Treatment in Workers Comp that they post on their Site at http://www.dol.ks.gov/wc/html/wc_odg.asp. In addition, ODG offers complete details on changes to users on a complimentary basis, using Microsoft® Track Changes, from any one specified point in time to any other (or to the current version), when they make a request to the Helpdesk. These may be helpful when there are disputes about what ODG has recommended at any point in time. After receiving one of these requests, the Helpdesk will determine whether a requester has a valid ODG paid license, and may suggest that they either subscribe to ODG, or pay for these special requests separately. There is also a limitation on the number of complimentary requests. These are the policies: (1) The requester must have current ODG license; (2) Requests from each authorized single-user sub are limited to one per month, multiple users would be one per user per month (so, for example, a single user could do this 12 times per year for free, whereas an Enterprise licensee with 1,000 users could do this 12,000 times per year); (3) Additional requests are $125 each compared to current date, or $175 each compared to any two dates, paid in advance; (4) Requests must have proper specificity, including a specific topic in a Procedure Summary using the correct ODG wording, and the dates to be compared. In producing the most comprehensive, up-to-date, unambiguous treatment guideline possible, ODG is committed to following a process consistent with the highest standards of evidence-based medicine. After updates have been made to ODG and noted in the update log file, ODG will notify individuals suggesting an update.

########

Higgins JPT, Green S, editors . Cochrane Handbook for Systematic Reviews of Interventions 4.2.5. In: The Cochrane Library, Issue 3, 2005. Chichester, UK: John Wiley & Sons, Ltd. September 2006.

6. ASSESSMENT OF STUDY QUALITY

6.0 Quality assessment of studies

Quality assessment of individual studies that are summarized in systematic reviews is necessary to limit bias in conducting the systematic review, gain insight into potential comparisons, and guide interpretation of findings. Factors that warrant assessment are those related to applicability of findings, validity of individual studies, and certain design characteristics that affect interpretation of results. Applicability, which is also called external validity or generalize-ability by some, is related to the definition of the key components of well-formulated questions outlined in section 4. Specifically, whether a review's findings are applicable to a particular population, intervention strategy or outcome is dependent upon the studies selected for review, and on how the people, interventions and outcomes of interest were defined by these studies and the authors (reviewers).

6.1 Validity

In the context of a systematic review, the validity of a study is the extent to which its design and conduct are likely to prevent systematic errors, or bias. An important issue that should not be confused with validity is precision. Precision is a measure of the likelihood of chance effects leading to random errors. It is reflected in the confidence interval around the estimate of effect from each study and the weight given to the results of each study when an overall estimate of effect or weighted average is derived. More precise results are given more weight.

6.2 Sources of bias in trials of healthcare interventions

There are four sources of systematic bias in trials of the effects of healthcare: selection bias, performance bias, attrition bias and detection bias.

6.3 Selection bias

Participants and those who recruit should remain unaware of next assignment in sequence. Empirical research has shown that lack of allocation concealment is associated with bias. For that reason trials should use approaches such as allocation by a central office unaware of subject characteristics, pre-numbered or coded identical containers which are administered serially to participants, or an on-site computer system combined with allocations kept in an unreadable file that can be accessed only after the characteristics of enrolled participants have been entered.

6.4 Performance bias

This refers to systematic differences in the care provided to the participants in the comparison groups other than the intervention under investigation. To protect against unintended differences in care and placebo effects, those providing and receiving care can be "blinded" so that they did not know the group to which the recipients of care have been allocated.

6.5 Attrition bias

This refers to systematic differences between comparison groups in the loss of participants from the study. The study should consider how losses of participants (withdrawals, dropouts and protocol deviations) are handled.

6.6 Detection bias

This refers to systematic differences between the comparison groups in outcome assessment.

Rating: 1a

########

Explanation of Medical Literature Ratings

Individual Study Ratings (medical treatment studies)

Evaluating the Body of Evidence (and prognostic/diagnostic/economic studies)

Ranking by Type of Evidence (treatment procedures):

Prognostic Studies

I

Link between evidence and recommendations:

L ink between evidence and recommendations: