Evidence-based Clinical Practice – Lecture 4 (UNIVERSITY OF AMSTERDAM)

Logistic regression regards regressing a dichotomous dependent variable on the basis of one or more independent variables. In the case of exploratory logistic regression, there are no a priori hypotheses. It makes use of maximum-likelihood estimation which selects coefficients that make the observed values most likely to have occurred.

Logistic regression cannot make use of linear regression because it violates the assumption of linearity (1), homoscedasticity (2) and normally distributed residuals (3). This is because it makes use of a dichotomous dependent variable.

Logistic regression makes use of a different formula than linear regression

This can be interpreted as the probability that y(i) = 1 equals some function of the intercept plus the independent variable weighed by some variable. The scale is between 0 and 1.  This allows researchers to approach the data in an approach similar to the general linear model.

The logistic regression makes use of several assumptions:

1. Linearity
   It assumes that the relationship between any continuous predictors and the logit of the outcome variable is linear. This can be tested by checking whether the interaction term between the predictor and the log transformation is significant.
2. Independence of error
   It assumes that the errors are independent because otherwise there is overdispersion.

Overdispersion refers to when the observed variance is bigger than expected from the logistic regression model. This can occur because of correlated observations and variability in success probabilities. It limits standard errors which leads to tests becoming significant and the confidence intervals becoming too narrow. It biases the conclusions about the significance of the parameters.

Complete separation refers to the situation where the outcome variable can be perfectly predicted by one variable or a combination of variables. This leads to enormously large standard errors and often occurs when too many variables are fitted to too few cases. The solution is to either collect more data or use a simpler model.

The regression line represents the probability of the dependent variable (e.g. getting in touch with the law). Model selection refers to the selection of independent variables. It is possible to make use of multiple independent variables. The enter method keeps all independent variables in the model. The stepwise method lets SPSS find out which variables are important. The forward stepwise model is typically used. The stepwise method is also used when there is no explicit hypothesis.

The fit of the model is fitting the model with only the intercept and then adding the independent variable and see if the fit is significantly improved. If this is the case, the independent variable should remain in the model. This is what happens in general regression.

The change in loglikelihood is a measure of goodness of fit in logistic regression. Interpreting the b1 value is not straightforward. The direction of change in the model after adding an independent variable can be done by interpreting the odds ratio. There is an improvement in the prediction after adding the independent variable if a ratio is higher than one. There are several rules for this:

1. If exp(b1) = 1, there is no effect
2. If exp(b1) = > 1 odds increase with independent variable
3. If exp(b1) = < 1 odds decrease with independent variable

Logistic regression can determine which factors influence any particular variable that is dichotomous.

Evidence-based clinical practice refers to clinical practice based on the best available evidence combined with the practitioner’s expertise and experience, adapted to the patient’s needs and preferences.

There are interactions between evidence-based guidelines and evidence-based clinical practice. The evidence-based guidelines are systematically developed documents based on scientific insights and accrued clinical experience. It can help practitioners to make decisions about adequate care for a specific health problem. A guideline that describes the ‘what, when and why’ is a directive document. The guidelines form the input for protocol.

The guidelines are often based on expert panels that assess available evidence and integrate them into tools for everyday practice. The guidelines are not universal and may change because of several reasons:

1. New developments in diagnostics or treatment methods.
2. New evidence.
3. New developments on societal or political area.
4. Cultural or historical differences.

The evidence-based guidelines have a major impact on healthcare because they are also used by insurance companies (1), policymakers (2) and industry (e.g. pharmacy (3). The evidence-based guidelines represent consensus.

However, these guidelines are not always applied in daily practice because there is insufficient knowledge about guidelines (1), there is no practical or financial feasibility for application of the guidelines (2) and there are different patient preferences (3). These guidelines require a practice culture which relies on evidence.

The scientist-practitioners need to know how to find, evaluate, assess and appraise evidence-based guidelines (1), scientific findings (2), practical observations (3), mechanisms of change (4) and mediators (5). Furthermore, they need to be able to adapt the interventions to the situation and they need to document, test and disseminate practical experience and expertise. 

The advantages of RCT are that they provide strong evidence (1) and that they often report effects on multiple outcome measures (2).  However, the concerns are that multiple comparisons could occur (1) and there could be questionable research practices (2). It is important to critically assess, evaluate and appraise scientific findings, their reliability and their clinical relevance.

Treatment effectiveness refers to whether a treatment works in clinical practice. Treatment efficacy refers to whether a treatment works in controlled settings.

A treatment which has been found efficacious in at least two studies by independent research teams is an efficacious treatment. A treatment which has only one study supporting the efficacy or if all the research has been conducted by one team is possibly efficacious treatment. Efficacious and specific treatments are treatments which control for other factors that could explain the treatment effect (e.g. therapeutic alliance).

There are different group designs for efficacy research:

1. Comparison with no treatment
   This comparison demonstrates whether a treatment actually works.
2. Comparisons with other treatments or with placebo
   This comparison demonstrates whether a treatment effect is due to the treatment or due to another effect (e.g. placebo).
3. Comparisons with other rival interventions
   This comparison demonstrates whether a treatment effect is due to competing mechanisms (e.g. receiving attention) or due to a unique mechanism. This can also compare the relative benefits of competing interventions.

Evidence from efficacy trials may be difficult to generalize because it only uses a specific group (e.g. one ethnicity) or because of the expertise of the therapists (i.e. the researchers who came up with the treatment).

In order to demonstrate that a treatment is equivalent to a well-established treatment, the research needs to have at least 25-30 participants per condition (1), the unproven test must not be significantly inferior to the established efficacious treatment on tests of significant (2) and the pattern of the data must not indicate trends of the established treatment to be superior (3). Any inferences regarding treatment efficacy can only be framed in the context of how the therapy was delivered and by whom.

It is important to know whether a treatment has an enduring effect and resuming treatment in the absence of symptoms is not necessarily an index of underlying risk as there are plenty of reasons to resume treatment. Differential attrition needs to be taken into account when doing an analysis.

The intention to treat analysis is based on the original treatment assignment (i.e. all participants who were supposed to receive treatment) and not on the treatment which is eventually received or completed. This is intended to avoid potentially misleading artefacts that can arise in intervention research (e.g. non-random attrition of participants). This is standard when aiming to establish if intervention A is better than intervention B. However, it may underestimate potential treatment effects for situations where treatment compliance is more optimal.

Per protocol analysis refers to a comparison of treatment groups that includes only those patients who completed the allocated treatment. However, this may lead to bias as the randomization may not be appropriate if there is non-random attrition. This is standard to assess if intervention A is not worse than intervention B. This is a more optimal estimate of potential treatment effects for situations where treatment compliance is more optimal.

When doing a single-case experiment, potential issues are establishing a stable baseline (1), using typically accepted designs (2), defining efficacy (3) and interpreting the results (4). The ABAB design is a design in which there is a baseline period, a treatment period, a period of non-treatment and treatment again.

A treatment needs to be feasible. This means that the patients need to be able to adhere to the treatment (1), there need to be enough therapists to be able to provide the treatment to patients (2), the treatment needs to be cost-effective (3) and the costs and benefits of the treatment need to be evaluated for both the short- and long-term (4).

There are several characteristics of therapy with children and adolescents:

1. Dysfunction is difficult to assess
   The problematic behaviour may represent short-lived problems or perturbations in development rather than signs of lasting clinical impairment.
2. Identifying cases is problematic
   Youth rarely refer themselves to treatment which makes that externalizing problems are overrepresented in treatment.
3. Dependence on adults
   The dependence of children on adults makes them vulnerable to multiple influences over which they have little control (e.g. living circumstances; parental mental health). Therefore, the family context needs to be addressed in treatment as well.
4. Social environment and treatment
   The social environment plays a very important role for children which makes that taking treatment alone (i.e. without a peer, parent or sibling) is often not possible.
5. Methodological challenges
   It is not clear whether self-report is an appropriate measure for young children and other methods may be flawed when used with youth (e.g. standardized assessment methods).
6. Heterogeneity of samples
   The studies that are conducted typically have very heterogeneous samples which makes interpretation of the results difficult.

Most therapy studies focus on non-referred cases (1), provide relatively brief treatments conducted in group format (2), evaluate the treatment in relation to symptom reduction and neglect impairments or adaptive functioning (3), do not evaluate clinical significance of symptom change (4) and do not conduct a follow-up (5).

Diverse differences among age-groups (e.g. language skills) indicate that treatment with similar general features must differ in numerous specific details when applied in different developmental periods. This leads to a classification dilemma (i.e. what cut-off to use).

Issues for treatment include the magnitude of therapeutic change (1), maintenance of change (2), identifying individual circumstances on which effective treatment depends (3), comorbidity (4), mechanisms of change (5), effectiveness of treatments (6) and the fit of therapy in clinical settings.

Researchers might not have difficulty finding statistical results because of researcher degrees of freedom. This contributes to the replication crisis. The type M error refers to overestimating the effect size as a result of a statistically significant result.

Ideas to resolve the replication crisis include better science communication (1), improved design and data collection (2) and better data analysis (3). The replication crisis appears to be the result of a flawed scientific paradigm rather than the result of a set of individual errors.

There are five major challenges in assessing experimental evidence within clinical neuropsychology:

1. Clinical tests can include a wide range of outcomes
   A test with a large number of dependent variables might have several significant results because of chance.
2. Replicating findings in specific and sufficiently large patient groups
   It is difficult to replicate findings in rare diseases and other specific groups. However, this could be aided by replicating a small part of a study in another study, so that researchers can help each other.
3. Determining whether a finding is robust and should be used in clinical practice
   It is difficult to determine whether a finding is robust and should consequently be used in clinical practice.
4. Determining whether an earlier finding can be disregarded
   It is difficult to determine whether there is enough evidence to disregard earlier conclusions that are widespread in clinical practice.
5. Translate group findings to individual patients
   It is difficult to generalize the findings of a study to an individual patient.