There are three important questions when assessing the relationship between variables:Statistical significanceEffect sizePractical significanceThe practical significance (i.e. clinical significance) can be assessed by considering factors such as clinical benefit (1), cost (2) and side effects (3). In order to assess the practical significance, the strength of the association (i.e. ‘r’) (1), magnitude of the difference between the difference between treatment and comparison (i.e. ‘d’) (2) and measures of risk potency (3) can be used. Risk potency can be assessed using the odds ratio (1), risk ratio (2), relative risk reduction (3), risk difference (4) and number needed to treat (5). The p-value refers to the probability that the found outcome or more extreme is found, given that the null hypothesis is true. It is possible to have statistical significance by chance and outcomes with lower p-values are sometimes interpreted as having stronger effect sizes. A non-significant result also does not tell us anything about the truth of the null hypothesis. There are several different effect size measures:The ‘r’ familyThis is expressing effect size in terms of strength of association (e.g. correlation). The ‘d’ familyThis is an effect size that can be used when the independent variable is binary and the dependent variable is ordered. It can be computed by subtracting the mean of the second group from the mean of the first group and divide it by the pooled standard deviation of both groups. The effect size ranges from minus to plus infinity. Measures of risk potencyThis is an effect size that can be used when both the dependent and independent variables are binary. The odds ratio and risk ratio vary from 0 to infinity and 1 indicates no effect. Risk relative reduction and risk difference vary between -1 and 1 with 0 indicating no effect. Number needed to treat (NNT) ranges from 1 to plus infinity with very large values indicating no...
1 - Go to JoHo.org/en/joinjoho, and join JoHo WorldSupporter by choosing a membership with online access
2 - Return to WorldSupporter.org and create an account with the same email address
3 - State your JoHo WorldSupporter Membership during the creation of your account, and you can start using the services
JoHo can really use your help! Check out the various student jobs here that match your studies, improve your competencies, strengthen your CV and contribute to a more tolerant world
Dennis et al. (2009). Why IQ is not a covariate in cognitive studies of neurodevelopmental disorders.” – Article summary
Neurodevelopmental disorders are different from adult acquired disorders involving traumatic brain injury because they involve no period of normal development. Any IQ score in a neurodevelopmental disorder is confounded with the condition. The IQ score can never be separated from the effects of the condition.
Early on in intelligence testing, IQ was seen as a latent variable. Intelligence, as measured by ‘g’ was believed to have a causal power. It was believed to be independent of test conditions and other methodological issues. However, ‘g’ or intelligence varies among time and place.
ANCOVA was devised to minimize pre-existing group differences (e.g. differences in SES). However, it is not possible to treat IQ as a covariate in neurocognitive research since it is an attribute of the disorders (e.g. ADHD). A covariate should be used when the assignment to the independent variable is done randomly (1), the covariate is related to the outcome measure (2) and the covariate is unrelated to the independent variable (3). It can be used if the researcher is trying to find out the direct effect of the independent variable on the outcome variable and the covariate is spuriously related to either variable or when it mediates the relationship between the independent and the outcome variable.
The ANCOVA has several assumptions:
The presence of a relationship between the covariate and the dependent variable does not imply that the covariate mediates or moderates the relationship between the group measure and the dependent variable.
There is no consensus on the definition of IQ. IQ can be used as a covariate for acquired brain damage if the preinjury IQ scores are available or when preinjury IQ proxies are available.
Kazdin (2008). New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care.” - Article summaryIt is unclear which treatment is the most effective in particular cases and it is not clear which treatments have sufficient evidence to be used, as there are no guidelines for this. Evidence-based treatment (i.e. EBT) refers to interventions or techniques that have therapeutic changes in controlled trials. Evidence-based practice (EBP) refers to clinical practice informed by evidence about interventions, clinical expertise and patient needs, values and preferences. Evidence-based practice is not what researchers have studied and might lack evidence.
One critique of evidence-based treatment is that key conditions and characteristics of treatment research (e.g. context) is different from clinical practice. This makes the results difficult to generalize. Another critique is that the research tends to focus on the symptoms, rather than the patients as a whole. There are also three points of criticism regarding the methods of treatment research:
There are also several concerns about evidence-based clinical practice. This includes clinical judgement as a way of integrating information because of concerns of reliability and validity. Many critical clinical issues and concerns are not heavily researched. The clinical decision making is criticized because there is no replicable method of doing it. Furthermore, it is not clear how the results in clinical practice can be generalized. This is difficult to do because it is not always possible to generalize clinical results from one patient to the next. This makes it difficult to make clinical judgement. Another concern is determining which variables make a difference in treatment, as this is always probabilistic (i.e. clinical decision making is analogous to multiple regression). A final concern is the way in which clinical progress is evaluated. This is often done based on clinician impressions rather than systematic observations. There are a plethora of treatments available for clinicians and not all have evidence to support them.
The goals of research are optimally developing the knowledge base (1), provide the best information to improve patient care (2) and materially reduce the divide between research and practice (3). This can be better achieved by shifting the emphasis to give a greater priority to the study of mechanisms of therapy (1), study the moderators of change (2) and by conducting more qualitative research (3).
It is imperative to determine the mechanisms of change in order to assess treatments
.....read more Kraemer et al. (2003). Measures of clinical significance.” - Article summaryThere are three important questions when assessing the relationship between variables:
The practical significance (i.e. clinical significance) can be assessed by considering factors such as clinical benefit (1), cost (2) and side effects (3). In order to assess the practical significance, the strength of the association (i.e. ‘r’) (1), magnitude of the difference between the difference between treatment and comparison (i.e. ‘d’) (2) and measures of risk potency (3) can be used. Risk potency can be assessed using the odds ratio (1), risk ratio (2), relative risk reduction (3), risk difference (4) and number needed to treat (5).
The p-value refers to the probability that the found outcome or more extreme is found, given that the null hypothesis is true. It is possible to have statistical significance by chance and outcomes with lower p-values are sometimes interpreted as having stronger effect sizes. A non-significant result also does not tell us anything about the truth of the null hypothesis.
There are several different effect size measures:
This standard is relative and it should be noted that ‘larger than typical’ should be used rather than ‘large’. However, it might be best to find information about typical effect sizes in the context of a particular research field.
The disadvantages to the ‘d’ and the ‘r’ as measures of clinical significance are that they are relatively abstract (1), they were not intended as measures of clinical significance (2) and they are not readily interpretable in terms of how much individuals are affected by treatment (3).
There is no consensus on the externally provided standards for the clinical significance of treatments. Clinical significance could be defined as a change to normal
.....read more Funder et al. (2014). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice.” - Article summary There appears to be a high rate of false positives in science. The p-value refers to the conditional probability that the present data or more extreme will be observed in a random sample given that the null hypothesis is true. Critique of using the p-value includes not interpreting the p-value properly (1) and the fact that the p-value varies with a varying sample size (2).
The effect size refers to the magnitude of the found statistical relationship. Unstandardized effect sizes are preferred when there is a clear consensus regarding that the measurement unit is in the interval level (e.g. seconds; blood pressure).When this is not the case (e.g. in psychology), standardized effect sizes should be used. The effect size can be easily interpreted if the sample sizes are equal and the total sample size is moderate to large. However, this becomes more complex if the sample sizes differ greatly.
Power refers to the probability that a true effect of a precisely specified size in the population will be detected using significance testing. It is the probability of finding an effect given that an effect of the specified size (i.e. the power) exists. The statistical power is one minus the type II error rate. The type II error rate refers to the probability that a true effect will not be detected using significance testing. It is the probability that the alternative hypothesis is wrongfully rejected.
Power should be maximized in a study. The power, however, is affected by the sample size (1), the measurement error (2) and the homogeneity of the participants (3). According to Cohen, research should use power of at least 0.8.
It is problematic to solely focus on the observed p-value because findings with equivalent p-values can have very different implications. In cases with large sample sizes, very small effect sizes can be significant and in cases with small sample sizes, a significant effect may produce an implausible large effect size. Therefore, there is a big difference between practical and statistical significance. A lot of research with small sample sizes is underpowered.
There are several recommendations to improve research:
Halpern, Karlawish, & Berlin (2002). The continuing unethical conduct of underpowered clinical trials.” - Article summary Randomized controlled trials without sufficient statistical power may not adequately test the underlying hypotheses. This makes it useless and, thus, unethical. However, underpowered research continues to be conducted.
Arguments in favour of using underpowered research are that meta-analyses may save the small studies by providing a means of combining the results with those of similar studies (1) and that confidence intervals could be used to estimate treatment effects (2).
However, underpowered trials can only be properly used in interventions of rare diseases (1) and in the early-phase of the development of drugs and devices (2). The interventions for rare diseases need to specify that they are planning to use their results in a meta-analysis of similar interventions. However, in both cases, the participants must be informed that their participation may only indirectly contribute to future health care benefits.
The number of participants (1), the expected variability (2) and the chosen probability of a type-I error (3) are used to calculate statistical power. There is no consensus regarding of how small an effect size must be to be clinically significant and thus must be recognized and in cases like this, the expected effect size can be used to calculate the number of participants for any given study. Empirical definitions of clinically meaningful effects (1) and data from earlier trials (2) must be used if it exists or if it doesn’t, the moderate effect sizes described by Cohen (3) should be used in the sample size calculation.
Studies that contain too few participants to detect a positive effect via hypothesis testing will also include unacceptably wide confidence intervals, which cannot properly be used to estimate the effect size of the treatment. Furthermore, problems with synthesizing the results may prevent the calculation of valid treatment effects in a meta-analysis. The ideal conditions for meta-analysis (e.g. comparable research methods among the primary trials) are not often met, which leads different meta-analyses to have different results. Lastly, underpowered trials are more likely to produce negative results and thus less likely to be published, this reduces the probability of it being used in a meta-analysis.
It is necessary to inform the participants of an underpowered study of their limited value to the participants because of ethical concerns. However, this information is often not given to participants because investigators do not do an a priori power analysis (1), investigators do not enrol enough participants in a timely fashion (2) or investigators fear it will reduce enrolment (3).
Only prospectively designed meta-analyses can justify the risks to participants in individually underpowered trials because they provide sufficient assurance that a study’s results will eventually contribute to valuable or important knowledge. Plans for large, comparative trials of experimental intervention can justify the conduct of small studies in earlier phases of drug or device development.
Moore (2016). Pre-register if you want to.” - Article summarySelective reporting refers to not publishing and reporting all research results and this is a problem for psychological research. Many scientists do not preregister their studies because they fear it would constrain their creativity (1), it brings added scrutiny to their research reporting (2) and it is another requirement in a field which already has a lot of requirements (3).
Preregistering makes testing a theory more difficult because sometimes scientists think of the best way to test a theory after data collection. However, exploratory studies can be used for this. Preregistering is not as laborious as researchers might think.
Preregistering clarifies the distinction between confirmatory and exploratory tests. Conducting more tests and reporting fewer inflates the type-I error rate. Preregistration helps to obtain the truth. Preregistration contributes to the confidence in a published result. It makes p-values useful for their intended purposes. The risk of a false positive is higher than the reported p-value without preregistration but it is difficult to tell how much higher.
Gelman & Geurts (2017). The statistical crisis in science: how is it relevant to clinical neuropsychology?” - Article summaryThe statistical crisis (i.e. replication crisis) refers to the fact that statistical significance does not necessarily provide a strong signal in favour of scientific claims.
One major challenge for researchers is accepting that one’s claims can be spurious. Mistakes in statistical analyses and interpretation should be accepted and learned from rather than resisted. Criticism and replication are essential steps in the scientific process and one should not accept scientific claims at face value nor should they believe they are 100% true. Once an idea is integrated in the literature, it is very difficult to disprove it even though evidence supports the rebuttal attempts. Researchers should remain very critical of their own work and if possible, replicate their own studies.
When data analysis is selected after the data have been collected, p-values cannot be taken at face value. Published results should be examined in the context of their data, methods and theoretical support. Assessing the strength of evidence remains difficult for most researchers. Researchers might not have difficulty with finding statistically significant results that can be construed as being part of the general constellation of findings that are consistent with the theory because of the researcher degrees of freedom (e.g. choosing statistical analyses).
Statistical significant is less meaningful than originally thought because of the researcher degrees of freedom (1) and because statistically significant comparisons systematically overestimate effect sizes (2). The type M error refers to overestimating the effect size as a result of a statistically significant result.
There have been several ideas to resolve the replication crisis:
The replication crisis appears to be the result of a flawed scientific paradigm rather than the result of a set of individual errors.
The problem of multiple comparisons should be taken into account and an a priori power analysis should be conducted in science. However, when the power analysis is based on previous effect size estimates, there should be caution as these effect sizes are thus often overestimated.
The issue of knowledge translation is a general issue in science, as it is not clear whether case studies can be generalized to the population and whether the observed effect in group studies is sufficiently large to have clinical implications for each individual of a specific group.
There are five major challenges in assessing experimental evidence within clinical neuropsychology:
Kazdin & Weisz (1998). Identifying and developing empirically supported child and adolescent treatments.” - Article summaryThere are several characteristics of therapy with children and adolescents:
Many emotional and behavioural problems that are treated in therapy are often evident in less extreme forms in early development. Treatments have shown beneficial effects in the treatment of children.
Most therapy studies focus on non-referred cases (1), provide relatively brief treatments conducted in group format (2), evaluate treatment in relation to symptom reduction and neglects impairment or adaptive functioning (3), do not evaluate clinical significance of symptom changes (4) and do not conduct a follow-up (5).
Diverse differences among different age groups (e.g. language skills) indicate that treatment with similar general features must differ in numerous specific details when applied in different developmental periods. This leads to a classification dilemma (i.e. what cut-off to use).
A study needs to meet the following criteria to be a good study:
It is likely that dysfunctional anxiety becomes a self-perpetuating cycle of elevated biological response to stress, debilitating cognitions and avoidance of stressful circumstances. CBT appears to be effective for child anxiety.
Depressed children are seen as subject to schemas and cognitive distortions that cast everyday experience in an unduly negative light and as lacking important skills needed to generate supportive social relationships and regulate emotion through daily activity.
Coping skills training (CST) appears to be effective in the treatment of depression for children. It includes structured homework assignments as well as peer or therapist modelling. The mediators and differential effectiveness relative to alternative, simpler treatments still need to be tested.
Cognitive processes refer to a broad class of constructs that
.....read more Kahn (2011). Multilevel modelling: Overview and applications to research in counselling psychology.” - Article summaryMultilevel modelling (MLM) is becoming the standard for analysing nested data.
The unit of analysis problem refers to when the moderator variable exists at a different level than the independent and dependent variable (e.g. moderator variable is on the level of the course itself and the independent and dependent variables are on the individual level). Multilevel modelling (MLM) or hierarchical linear modelling is designed to analyse data that exists at different levels.
When individuals exist in natural groups such as schools, there is a hierarchical or nested structure. Ignoring the nested structure of the data could have adverse consequences. Person-analyses with nested data ignore the fact that individuals sharing a common environment and are more similar to each other than they are to individuals from another environment. Nested data may thus violate the assumption of independence of observations. This leads to an increase of type-I errors due to the small standard errors.
In a typical regression analysis, the slope and the intercept are fixed as the same parameters apply to each case in the sample. However, these parameters may vary as a function of group membership in the case of nested data, thus needing the need for a different analysis.
The formula for the level one, multilevel analysis model is the following:
Yij=β0j+β1jXij-Xj+rij
The letter i refers to the level 1 unit (e.g. individuals). The letter j refers to the level 2 unit (e.g. group). Yij
is the level 1 dependent variable (e.g. individual score in a given group). Xij
is the score on the level 1 independent variable. This formula makes use of a form of centring, as the score on the level 1 independent variable is subtracted from the mean of the group. β0j
is the intercept for a given group (e.g. predicted level of a score in a particular group). This can be interpreted as the mean level of a dependent variable for all people in a particular group. β1j
is the slope for a given group. It is the predicted increase in the dependent variable per 1-unit increase in the independent variable within a given group. rij
is the residual at the individual level.
The level two model addresses what the average intercept and slope across groups is (i.e. fixed effects) (1), how much intercepts and slopes vary across groups (i.e. random effects) (2) and how useful group-level variables for predicting group intercepts and slopes are (3). The level two model uses the following formula:
The level two model describes the difference between group level and not between individuals within the groups. The first formula represents the level two model for
.....read more Robinaugh, Hoekstra, Toner, & Borsboom (2020). The network approach to psychopathology: A review of the literature 2008-2018 and an agenda for future research.” - Article summary The network model to psychopathology states that mental disorders can be conceptualized and studied as causal systems of mutually reinforcing symptoms. This holds that symptoms are not passive indicators of a latent, common cause, of the disorder but agents in a causal system.
The causality hypothesis states that when causal relations among symptoms are strong, the onset of one symptom will lead to the onset of others. The connectivity hypothesis states that a strongly interconnected symptom network is vulnerable to a contagion effect of spreading activation through the network. Widespread symptom activation as a result of an external stressor can lead to symptoms persisting when the initiating stressor is removed. The centrality hypothesis states that highly central symptoms have greater potential to spread symptom activation throughout the network than do symptoms on the periphery. The comorbidity hypothesis states that symptoms can occur in multiple disorders and that some symptoms can thus bridge different disorders.
A mental disorder is characterized by both the state and the structure of the network (i.e. a mental disorder is characterized by a state of harmful equilibrium). The boundary between health and disorder will vary as a function of network structure. In weakly connected networks, activation varies dimensionally. However, in strongly connected networks, activation within the system rapidly leads to a state of psychopathology (i.e. more discrete rather than continuous).
The momentary perspective states that symptoms are aggregates of moment-to-moment experiences. According to this perspective, these experiences constitute the true building blocks of psychopathology. This highlights the importance of understanding the chronometry of experiences, symptoms and disorders.
The assumptions of the network model currently do not always align with how disorders are believed to operate.
There is a conditional positive manifold for most disorders. This states that symptoms of a positive disorder tend to be positively interconnected, even after controlling for shared variance among symptoms. This suggests meaningful clustering of symptoms in the syndromes we call mental disorders. Connectivity tends to be consistent across time and demographic groups. However, differences have been observed across countries.
Greater connectivity (i.e. network density) may confer risk for psychopathology. This is based on the fact that there appears to be greater connectivity between symptoms in people with more severe mental disorders. It is also possible that greater connectivity leads to disorder persistence. However, there is no consensus regarding these topics. There is some evidence that connectivity of negative mood state networks is associated with psychopathology but minimal evidence that broader networks of momentary experience exhibit such associations.
Node strength refers to the summed absolute strength of a node’s direct link. Non-DSM symptoms often exhibit elevated centrality (e.g. feeling disliked in depression) and some DSM-nodes are weakly connected to the network. It is not clear whether the symptoms which the DSM identifies as especially important are more central to less important DSM-symptoms. The DSM most likely has not captured all symptoms of a disorder and has not
.....read more De Vent et al. (2016). Advanced neuropsychological diagnostics infrastructure (ANDI): A normative database created from control datasets.” - Article summaryIn the advanced neuropsychological diagnostics infrastructure (ANDI), datasets of several research groups are combined into a single database. It contains scores of neuropsychological tests from healthy participants. This allows for accurate neuropsychological assessment as the quantity and the range of the data surpasses most traditional normative data. It facilitates normative comparison methods (e.g. those in which entire profiles of scores are evaluated).
An important element of neuropsychological practice is to determine whether a patient who presents with cognitive complaints has abnormal scores on neuropsychological tests. In the diagnostic process, a number of neuropsychological tests are administered and the test results of a patient are compared to a normative sample.
Scores of patients are typically compared to normative data published in the manuals of an instrument. However, this data may be outdated (1), it may lack norms for very old populations (2), some tests do not have any norms (3), normative scores are often only corrected for age but not for other demographic variables (4) and normative data are often collected for one test at a time (5). This results in univariate but not multivariate data being available while multivariate normative comparison methods are more sensitive to deviating profiles of test scores.
There are several benefits of the ANDI database:
Limitations of the ANDI database are that it is not necessarily based on a random sample (1) and some participants had lenient inclusion criteria and, thus, did not necessarily have no pathology (2).
Maric et al. (2015). Evaluating statistical and clinical significance of intervention effects in single-case experimental designs: An SPSS method to analyze univariate data.” - Article summary Single-case experimental designs (SCED) are useful methods in clinical research to investigate individual client progress. It could also be used to determine whether an intervention works. In single-case experimental designs, a single participant is repeatedly assessed on one or multiple symptoms during various phases (e.g. baseline). The statistical analysis of this data is difficult.
Advantages of the single-case experimental designs are that it can be used to test novel interventions before RCTs are conducted (1), it may be the only way to investigate treatment outcomes in heterogeneous groups (2) and it offers the possibility to systematically document the knowledge of researchers and clinicians, preventing loss of information (3).
The most common single-case experimental design is the AB design. It consists of two phases (i.e. baseline and treatment). It is similar to an interrupted time series. In order to obtain an adequate analysis of the differences, the overall pattern in the time series should be modelled adequately (1) and there should be adequate modelling of potential correlations between residuals (2).
A common assumption of adequate models is that there is a linear function for both the baseline and the treatment phase. Both of these linear functions should be described by an intercept and a slope. Modelling of potential correlations between residuals means that there should be adequate modelling after the overall pattern has been accounted for. The correlation between residuals of the observations is the autocorrelation. It implies that the residuals are not independent. If residuals are correlated, the correlations are likely to decrease with increasing separation between timepoints.
The tests on the intercepts and the slopes of the linear functions will be unreliable if the correlations between the residuals are not modelled adequately (e.g. incorrectly assuming that the residuals are uncorrelated). If positively correlated residuals are assumed to be uncorrelated, the chances of finding significant results will be too high.
Modelling the overall pattern by the intercept and the slope for each phase (i.e. each timepoint) does not yield a direct test of intercept and slope differences. Therefore, it is useful to re-parameterize the model in terms of an intercept and slope for the baseline phase and baseline-treatment differences in the intercepts and slopes. This takes the following formula:
Y(i) denotes the outcome variable score at time point i. Phase(i) denotes the phase in which time point i is contained. Time_in_phase denotes time points within each phase. E(i) denotes the residual at time point i.
The parameter b0 is interpreted as the baseline intercept. B1 is interpreted as the treatment-baseline difference in intercepts. B2 is interpreted as the baseline slope and b3 is interpreted as the treatment-baseline differences in slopes. These parameters can also be interpreted as effect sizes.
The b0 and b1 refer to symptom scores when time_in_phase is zero. Therefore, when coding the variables, time_in_phase zero should denote the start of each phase. However, when time_in_phase zero
.....read more Evidence-based Clinical Practice – Lecture 1 (UNIVERSITY OF AMSTERDAM)Evidence-based treatment refers to interventions or techniques that have therapeutic changes in controlled trials. Disadvantages are that key conditions and characteristics of treatment research are different from clinical practice and that research tends to focus on symptoms, rather than the patient as a whole. Determining whether a treatment is evidence-based or empirically supported has different criteria (1), makes use of arbitrary rating scales (2) and often has to deal with mixed results (3).
Evidence-based practice refers to clinical practice informed by evidence about interventions, clinical expertise, patient needs, values and preferences. Concerns regarding this are that not everything has been researched (e.g. rare disease) (1), clinical decision making is unreliable (2), generalization of results is difficult (3), determining which variables make the difference in a treatment is difficult (4) and clinical progress is often evaluated based on clinical impressions (5).
The goals of research are to optimally develop the knowledge base (1), provide the best information to improve patient care (2) and reduce the divide between research and practice. These goals can be achieved by shifting the emphasis to give a greater priority to the study of mechanisms of therapy (1), study the moderators of change (2) and conducting more qualitative research (3).
Conducting more qualitative research could connect metrics that are not arbitrary (e.g. connecting objective measures to a disorder). Furthermore, it could also generate hypotheses. Clinical practice should use more systematic measures of patient progress because it helps to provide high-quality care (1), it helps make decisions regarding treatment (e.g. continuation or not) (2) and it helps to complete clinical judgement (3). Clinical practice should also add accumulated knowledge to the knowledge-base.
The outcome of treatment can be improved by identifying effective interventions (1), understanding why and how an effective treatment works (2) and identifying moderators of treatment (3).
There are three distinctions for the classification of variables:
There are variables that can be both a manifest or a latent variable, depending on how it’s being used (e.g. intelligence).
The foundation of most statistical tests (i.e. general linear model) are comparing two or more conditions and checking whether the difference between the conditions are significant. This is done by calculating the test statistic (e.g. t-statistic, F-statistic). This is typically calculated by having the
.....read more Evidence-based Clinical Practice – Lecture 2 (UNIVERSITY OF AMSTERDAM)In the case of a large number of tests, the probability of a false positive does not change for individual tests. However, the tests taken together have an inflated probability of false positives (i.e. inflated type-I error rate). This is also called the multiple comparison problem.
A research needs to have enough power. Besides that, statistical significance does not tell us anything about practical significance as any result can be significant when the sample size is sufficiently large. A large sample size will almost always lead to significant results and this does not tell us anything about the practical relevance. This is partially due to the arbitrary cut-off point for the p-value (i.e. p<.05). The cut-off point is not adjusted for the sample size. Therefore, the p-value is not indicative of practical relevance.
Practical significance can be assessed by considering factors such as clinical benefit (1), cost (2) and side-effects (3). This requires the effect size and the risk potency.
The absence of evidence does not imply evidence for absence. This means that not finding evidence does not mean that there is no evidence. It is possible that the study is underpowered. Furthermore, it is possible that there are small, meaningless differences which are significant and large, meaningful differences which are not significant.
The power refers to the probability of finding an effect when there actually is an effect (equal to the size of power). This is mostly considered when failing to reject the null hypothesis but it should also be considered when the null hypothesis is rejected.
The power affects the capacity to interpret the p-value. The p-value exhibits a wide sample-to-sample variability unless the statistical power is very high. This implies that evaluating a study by the p-value alone is spurious. A research with power of 50% (i.e. standard in psychology) leads to a less than 50% odds of replicating the significant result in the study. This contributes to the replication crisis. In short, the p-value does not provide information regarding the strength of evidence against the null hypothesis.
The p-values are typically only meaningful with large samples. Therefore, it is useful to look at the effect size. The effect size refers to how strong the effect of an intervention is. It corresponds to the degree of non-overlap between sample distributions (1) and the probability that one could guess which group a person came from, based only on their test score.
Typical methods of denoting the effect size are Cohen’s d (1), Hedges’ g (2) and Pearson’s r (3).
An intervention that is compared to a placebo requires an effect size of 0.8. However, an intervention compared to another intervention requires an effect size of 0.5. Effect size is heavily inflated in small samples and thus requires large samples.
The less overlap
Evidence-based Clinical Practice – Lecture 3 (UNIVERSITY OF AMSTERDAM)There are several ethical guidelines in science:
A lot of people are exposed to a potentially not-working intervention in an overpowered study because it requires a lot of participants (i.e. large sample size). In an educational setting, there should be a culture of getting it right rather than finding a significant result (1), students should be taught transparency of data reporting (2), the methodological instructions (e.g. confidence interval) should be improved (3) and junior researchers who seek to conduct proper research should be encouraged (4).
A-prior power analysis regards the question of how many participants are needed in the study. If a study has sufficient power, then non-significant results are also informative. The power of the test affects the capacity to interpret the p-value. The p-value exhibits wide sample-to-sample variability and does not indicate the strength of evidence against the null hypothesis unless the statistical power is high.
Arguments in favour of underpowered studies are that meta-analyses will provide useful results by combining the results and confidence intervals could be used to estimate treatment effects. However, underpowered studies are more likely to produce non-significant results and are thus more likely to disappear into the file drawer. Furthermore, the ideal conditions for meta-analyses are not often met, which makes it not possible for a meta-analysis to always save the smaller, underpowered studies.
Underpowered studies can only be used in interventions of rare diseases and the early-phase of the development of drugs and devices. It is important for interventions of rare diseases to specify that they are
.....read more Evidence-based Clinical Practice – Lecture 4 (UNIVERSITY OF AMSTERDAM)Logistic regression regards regressing a dichotomous dependent variable on the basis of one or more independent variables. In the case of exploratory logistic regression, there are no a priori hypotheses. It makes use of maximum-likelihood estimation which selects coefficients that make the observed values most likely to have occurred.
Logistic regression cannot make use of linear regression because it violates the assumption of linearity (1), homoscedasticity (2) and normally distributed residuals (3). This is because it makes use of a dichotomous dependent variable.
Logistic regression makes use of a different formula than linear regression
This can be interpreted as the probability that y(i) = 1 equals some function of the intercept plus the independent variable weighed by some variable. The scale is between 0 and 1. This allows researchers to approach the data in an approach similar to the general linear model.
The logistic regression makes use of several assumptions:
Overdispersion refers to when the observed variance is bigger than expected from the logistic regression model. This can occur because of correlated observations and variability in success probabilities. It limits standard errors which leads to tests becoming significant and the confidence intervals becoming too narrow. It biases the conclusions about the significance of the parameters.
Complete separation refers to the situation where the outcome variable can be perfectly predicted by one variable or a combination of variables. This leads to enormously large standard errors and often occurs when too many variables are fitted to too few cases. The solution is to either collect more data or use a simpler model.
The regression line represents the probability of the dependent variable (e.g. getting in touch with the law). Model selection refers to the selection of independent variables. It is possible to make use of multiple independent variables. The enter method keeps all independent variables in the model. The stepwise method lets SPSS find out which variables are important. The forward stepwise model is typically used. The stepwise method is also used when there is no explicit hypothesis.
The fit of the model is fitting the model with only the intercept and then adding the independent variable and see if the fit is significantly improved. If this is the case, the independent variable should remain in the model. This is what happens in general regression.
The change in loglikelihood is a measure of goodness of fit in logistic regression. Interpreting the b1 value is not straightforward. The direction of change in the model after adding an independent variable can be
.....read more Evidence-based Clinical Practice – Lecture 5 (UNIVERSITY OF AMSTERDAM)Nested data (i.e. multilevel data) refers to data gathered in multiple groups (e.g. intervention study in multiple classrooms). Nested data that is analysed in the usual way can have strong consequences for the significance of the results (i.e. results being significant while they are actually not). Nested data being analysed in the wrong way leads to drawing false conclusions that something is effective while it is not.
The unit of analysis problem refers to the situation where the moderator variable exists at a different level than the independent and the dependent variable. Using the regular statistical analyses with nested data violates the assumption of independence of observations because it ignores the fact that individuals share a common environment. This may lead to an increase in type-I errors due to the small standard errors.
The multilevel analysis makes use of a general linear model. In the multilevel analysis, data are measured at different levels. For example, multiple children (i.e. level 1) are within multiple classrooms (i.e. level 2). Classrooms are level two because multiple children can be in a classroom but a child cannot be in multiple classrooms. In the multilevel analysis, the dependent variable is continuous and the independent variable is nominal or continuous (i.e. the same as the GLM).
The gamma00 (i.e. 
) refers to a general change. There is a general change in the population but the change differs per level 2 variable (e.g. therapist). However, there is also variation within level 1 variables (e.g. individuals). This can be seen as the error.
In nested data analysed in the same way as non-nested data leads to the assumption of uncorrelated errors being violated. This means that the tests are too liberal and the results cannot be trusted. The assumption is violated because each level 1 variable (e.g. individuals) in a certain level 2 variable (e.g. classroom) has the same error, meaning that the errors become correlated (e.g. each individual in a classroom has the same error). A multilevel analysis accounts for these correlated errors.
The formula for a multilevel analysis without independent variables is the following:
The level 2 independent variables are often grand mean centred such that 0 reflects the mean of all groups. In this case, a score of 0 on a predictor would mean that can be interpreted as the mean level of the variable, adjusted for individuals in that group. Y
refers to the level two slope and can be interpreted as the change in that group’s mean for the dependent variable per 1-unit increase in the independent variable.
The intraclass correlation refers to the percentage of the total variance that is between group. This reflects the degree to which scores on the dependent variable are due to a nested factor.
The one-way ANOVA with random
.....read more Evidence-based Clinical Practice – Lecture 6 (UNIVERSITY OF AMSTERDAM)There are three types of theories:
The DSM-5 makes use of theories based on latent variables.
A theory focusing on manifest variables has the problem of whether there is a cut-off or not (e.g. a person is distracted or not). Besides that, if there is a cut-off, the question is what the cut-off is. Lastly, there is a problem of whether the treatment should be per manifest variable or not.
A theory focusing on latent variables has the problem of whether the latent variable is nominal, continuous or both (e.g. a person has ADHD or does not have ADHD). Besides that, it is the question of whether a latent variable should be based on consensus or on empirical evidence. Lastly, it is the question whether the treatment should focus on the latent variable or on the manifest variables (i.e. the symptoms). This regards to whether symptoms are treated or whether the presumed underlying cause is treated.
An empirical latent variable model requires a sample of the entire population including an enormous variety of symptoms.
Disadvantages of nominal latent variables based on consensus are that there is high comorbidity between disorders (1), there are arbitrary limits (i.e. why the cut-off for ADHD at 6/9 symptoms and not 5/9) (2) and there is a lot of heterogeneity within groups (3).
Basing a latent variable on empirical evidence can make use of a mixture analysis (1), factor analysis (2) or a factor mixture analysis (3). Typically, these three types of models are being fitted to the data and then it is determined which model fits the data best. This is then used in the explanation of the latent variable. The factor loading refers to how close the line is to the original variable.
A nominal model that fits best according to empirical data has the advantages that there is no comorbidity because there are only subgroups of the latent variable (e.g. ADHD subgroup with anxiety symptoms and ADHD subgroup that is distracted). There are no arbitrary limits as the limits are obtained from the data. In addition to that, there is reduced heterogeneity within groups because the subgroups of the latent variable are by definition homogenous.
The network model of psychopathology states that mental disorders can be conceptualized and studied as causal systems of mutually reinforcing symptoms. The causality hypothesis holds that when
.....read more Evidence-based Clinical Practice – Lecture 7 (UNIVERSITY OF AMSTERDAM)The advantages of analysing multiple individuals are that it allows for generalization of findings to others. However, it does not tell us anything about the individual. The analysis of an individual does not allow for generalization to others but does tell us something about the individual.
In an N=1 analysis, the baseline is compared with a treatment for a single individual. Normative comparisons make use of comparing a single individual to a distribution of the norm (e.g. normal distribution of test scores of a normative sample).
The question of whether there is an improvement requires a comparison while the question of whether there is a significant improvement requires multiple observations (e.g. baseline). There are several ways in which N=1 designs bridge the gap between science and practice:
There are several methodological criteria for an N=1 study:
In an N=1 design, the end of phases is compared and the change of phases is compared. This implies that there should be a higher or lower score for the dependent variable after the treatment phase than after the end of the baseline phase. Furthermore, the change in the treatment phase should be higher than the change in the baseline phase.
An N=1 analysis is a form of regression analysis.
The interpretation of the bs depends on the coding of the independent variables.
A problem of the N=1 analysis using the regression analysis is that the assumption of uncorrelated errors is violated. This leads to results that are too liberal. This can be solved by modelling the dependence in the data. This makes use of autoregressive order 1. It makes sure that the correlation between time points is only dependent on their distance. The gist of this is that timepoints that are close to each other
Dennis et al. (2009). Why IQ is not a covariate in cognitive studies of neurodevelopmental disorders.” – Article summaryNeurodevelopmental disorders are different from adult acquired disorders involving traumatic brain injury because they involve no period of normal development. Any IQ score in a neurodevelopmental disorder is confounded with the condition. The IQ score can never be separated from the effects of the condition.
Early on in intelligence testing, IQ was seen as a latent variable. Intelligence, as measured by ‘g’ was believed to have a causal power. It was believed to be independent of test conditions and other methodological issues. However, ‘g’ or intelligence varies among time and place.
ANCOVA was devised to minimize pre-existing group differences (e.g. differences in SES). However, it is not possible to treat IQ as a covariate in neurocognitive research since it is an attribute of the disorders (e.g. ADHD). A covariate should be used when the assignment to the independent variable is done randomly (1), the covariate is related to the outcome measure (2) and the covariate is unrelated to the independent variable (3). It can be used if the researcher is trying to find out the direct effect of the independent variable on the outcome variable and the covariate is spuriously related to either variable or when it mediates the relationship between the independent and the outcome variable.
The ANCOVA has several assumptions:
The presence of a relationship between the covariate and the dependent variable does not imply that the covariate mediates or moderates the relationship between the group measure and the dependent variable.
There is no consensus on the definition of IQ. IQ can be used as a covariate for acquired brain damage if the preinjury IQ scores are available or when preinjury IQ proxies are available.
Kazdin (2008). New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care.” - Article summaryIt is unclear which treatment is the most effective in particular cases and it is not clear which treatments have sufficient evidence to be used, as there are no guidelines for this. Evidence-based treatment (i.e. EBT) refers to interventions or techniques that have therapeutic changes in controlled trials. Evidence-based practice (EBP) refers to clinical practice informed by evidence about interventions, clinical expertise and patient needs, values and preferences. Evidence-based practice is not what researchers have studied and might lack evidence.
One critique of evidence-based treatment is that key conditions and characteristics of treatment research (e.g. context) is different from clinical practice. This makes the results difficult to generalize. Another critique is that the research tends to focus on the symptoms, rather than the patients as a whole. There are also three points of criticism regarding the methods of treatment research:
There are also several concerns about evidence-based clinical practice. This includes clinical judgement as a way of integrating information because of concerns of reliability and validity. Many critical clinical issues and concerns are not heavily researched. The clinical decision making is criticized because there is no replicable method of doing it. Furthermore, it is not clear how the results in clinical practice can be generalized. This is difficult to do because it is not always possible to generalize clinical results from one patient to the next. This makes it difficult to make clinical judgement. Another concern is determining which variables make a difference in treatment, as this is always probabilistic (i.e. clinical decision making is analogous to multiple regression). A final concern is the way in which clinical progress is evaluated. This is often done based on clinician impressions rather than systematic observations. There are a plethora of treatments available for clinicians and not all have evidence to support them.
The goals of research are optimally developing the knowledge base (1), provide the best information to improve patient care (2) and materially reduce the divide between research and practice (3). This can be better achieved by shifting the emphasis to give a greater priority to the study of mechanisms of therapy (1), study the moderators of change (2) and by conducting more qualitative research (3).
It is imperative to determine the mechanisms of change in order to assess treatments
.....read more Kraemer et al. (2003). Measures of clinical significance.” - Article summaryThere are three important questions when assessing the relationship between variables:
The practical significance (i.e. clinical significance) can be assessed by considering factors such as clinical benefit (1), cost (2) and side effects (3). In order to assess the practical significance, the strength of the association (i.e. ‘r’) (1), magnitude of the difference between the difference between treatment and comparison (i.e. ‘d’) (2) and measures of risk potency (3) can be used. Risk potency can be assessed using the odds ratio (1), risk ratio (2), relative risk reduction (3), risk difference (4) and number needed to treat (5).
The p-value refers to the probability that the found outcome or more extreme is found, given that the null hypothesis is true. It is possible to have statistical significance by chance and outcomes with lower p-values are sometimes interpreted as having stronger effect sizes. A non-significant result also does not tell us anything about the truth of the null hypothesis.
There are several different effect size measures:
This standard is relative and it should be noted that ‘larger than typical’ should be used rather than ‘large’. However, it might be best to find information about typical effect sizes in the context of a particular research field.
The disadvantages to the ‘d’ and the ‘r’ as measures of clinical significance are that they are relatively abstract (1), they were not intended as measures of clinical significance (2) and they are not readily interpretable in terms of how much individuals are affected by treatment (3).
There is no consensus on the externally provided standards for the clinical significance of treatments. Clinical significance could be defined as a change to normal
.....read more Funder et al. (2014). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice.” - Article summary There appears to be a high rate of false positives in science. The p-value refers to the conditional probability that the present data or more extreme will be observed in a random sample given that the null hypothesis is true. Critique of using the p-value includes not interpreting the p-value properly (1) and the fact that the p-value varies with a varying sample size (2).
The effect size refers to the magnitude of the found statistical relationship. Unstandardized effect sizes are preferred when there is a clear consensus regarding that the measurement unit is in the interval level (e.g. seconds; blood pressure).When this is not the case (e.g. in psychology), standardized effect sizes should be used. The effect size can be easily interpreted if the sample sizes are equal and the total sample size is moderate to large. However, this becomes more complex if the sample sizes differ greatly.
Power refers to the probability that a true effect of a precisely specified size in the population will be detected using significance testing. It is the probability of finding an effect given that an effect of the specified size (i.e. the power) exists. The statistical power is one minus the type II error rate. The type II error rate refers to the probability that a true effect will not be detected using significance testing. It is the probability that the alternative hypothesis is wrongfully rejected.
Power should be maximized in a study. The power, however, is affected by the sample size (1), the measurement error (2) and the homogeneity of the participants (3). According to Cohen, research should use power of at least 0.8.
It is problematic to solely focus on the observed p-value because findings with equivalent p-values can have very different implications. In cases with large sample sizes, very small effect sizes can be significant and in cases with small sample sizes, a significant effect may produce an implausible large effect size. Therefore, there is a big difference between practical and statistical significance. A lot of research with small sample sizes is underpowered.
There are several recommendations to improve research:
Halpern, Karlawish, & Berlin (2002). The continuing unethical conduct of underpowered clinical trials.” - Article summary Randomized controlled trials without sufficient statistical power may not adequately test the underlying hypotheses. This makes it useless and, thus, unethical. However, underpowered research continues to be conducted.
Arguments in favour of using underpowered research are that meta-analyses may save the small studies by providing a means of combining the results with those of similar studies (1) and that confidence intervals could be used to estimate treatment effects (2).
However, underpowered trials can only be properly used in interventions of rare diseases (1) and in the early-phase of the development of drugs and devices (2). The interventions for rare diseases need to specify that they are planning to use their results in a meta-analysis of similar interventions. However, in both cases, the participants must be informed that their participation may only indirectly contribute to future health care benefits.
The number of participants (1), the expected variability (2) and the chosen probability of a type-I error (3) are used to calculate statistical power. There is no consensus regarding of how small an effect size must be to be clinically significant and thus must be recognized and in cases like this, the expected effect size can be used to calculate the number of participants for any given study. Empirical definitions of clinically meaningful effects (1) and data from earlier trials (2) must be used if it exists or if it doesn’t, the moderate effect sizes described by Cohen (3) should be used in the sample size calculation.
Studies that contain too few participants to detect a positive effect via hypothesis testing will also include unacceptably wide confidence intervals, which cannot properly be used to estimate the effect size of the treatment. Furthermore, problems with synthesizing the results may prevent the calculation of valid treatment effects in a meta-analysis. The ideal conditions for meta-analysis (e.g. comparable research methods among the primary trials) are not often met, which leads different meta-analyses to have different results. Lastly, underpowered trials are more likely to produce negative results and thus less likely to be published, this reduces the probability of it being used in a meta-analysis.
It is necessary to inform the participants of an underpowered study of their limited value to the participants because of ethical concerns. However, this information is often not given to participants because investigators do not do an a priori power analysis (1), investigators do not enrol enough participants in a timely fashion (2) or investigators fear it will reduce enrolment (3).
Only prospectively designed meta-analyses can justify the risks to participants in individually underpowered trials because they provide sufficient assurance that a study’s results will eventually contribute to valuable or important knowledge. Plans for large, comparative trials of experimental intervention can justify the conduct of small studies in earlier phases of drug or device development.
Moore (2016). Pre-register if you want to.” - Article summarySelective reporting refers to not publishing and reporting all research results and this is a problem for psychological research. Many scientists do not preregister their studies because they fear it would constrain their creativity (1), it brings added scrutiny to their research reporting (2) and it is another requirement in a field which already has a lot of requirements (3).
Preregistering makes testing a theory more difficult because sometimes scientists think of the best way to test a theory after data collection. However, exploratory studies can be used for this. Preregistering is not as laborious as researchers might think.
Preregistering clarifies the distinction between confirmatory and exploratory tests. Conducting more tests and reporting fewer inflates the type-I error rate. Preregistration helps to obtain the truth. Preregistration contributes to the confidence in a published result. It makes p-values useful for their intended purposes. The risk of a false positive is higher than the reported p-value without preregistration but it is difficult to tell how much higher.
Chambless & Hollon (1998). Defining empirically supported therapies.” - Article summary Empirically supported treatments refer to clearly specified psychological treatments shown to be efficacious in controlled research with a delineated population. Treatment efficacy must be demonstrated in controlled research in which it is reasonable to conclude that benefits observed are due to the effects of the treatment and not to chance or confounds. This is best demonstrated in randomized clinical trials. Replication is critical in this case.
A treatment which has been found efficacious in at least two studies by independent research teams is an efficacious treatment. A treatment which only has one study supporting the efficacy or if all the research has been conducted by one team is possibly efficacious treatment.
The efficacy research has to be done with proper methods. There are different types of group design of efficacy research:
The more stringent the comparison is, the more value researchers should attach to the results. Treatments that are efficacious and specific are treatments which control for other factors that could explain the treatment effect (e.g. therapeutic alliance). This increases confidence in the theoretical explanatory model of the treatment.
Problems with comparisons with other rival interventions are interpretation (i.e. interpreting the treatment as being efficacious while it is a downgraded version of a well-established treatment) (1) and low statistical power (2).
In order to demonstrate that a treatment is equivalent to a well-established treatment, it should meet the following criteria:
A psychological treatment plus placebo condition which is better than a placebo condition may not be due to the psychological intervention alone but due to the interaction between the placebo and the psychological treatment.
Efficacy needs to be described for a specific population or problem rather than in general. Factors that may determine the population (e.g. socioeconomic status) need to be described. The tools used in efficacy research need to have properly demonstrated reliability and validity. It is best to use multiple methods of assessment. Measures of life quality and functioning, rather than just symptoms need to be included as well, to properly evaluate a treatment.
It is important to know whether a treatment has an enduring effect and whether treatments
.....read more Gelman & Geurts (2017). The statistical crisis in science: how is it relevant to clinical neuropsychology?” - Article summaryThe statistical crisis (i.e. replication crisis) refers to the fact that statistical significance does not necessarily provide a strong signal in favour of scientific claims.
One major challenge for researchers is accepting that one’s claims can be spurious. Mistakes in statistical analyses and interpretation should be accepted and learned from rather than resisted. Criticism and replication are essential steps in the scientific process and one should not accept scientific claims at face value nor should they believe they are 100% true. Once an idea is integrated in the literature, it is very difficult to disprove it even though evidence supports the rebuttal attempts. Researchers should remain very critical of their own work and if possible, replicate their own studies.
When data analysis is selected after the data have been collected, p-values cannot be taken at face value. Published results should be examined in the context of their data, methods and theoretical support. Assessing the strength of evidence remains difficult for most researchers. Researchers might not have difficulty with finding statistically significant results that can be construed as being part of the general constellation of findings that are consistent with the theory because of the researcher degrees of freedom (e.g. choosing statistical analyses).
Statistical significant is less meaningful than originally thought because of the researcher degrees of freedom (1) and because statistically significant comparisons systematically overestimate effect sizes (2). The type M error refers to overestimating the effect size as a result of a statistically significant result.
There have been several ideas to resolve the replication crisis:
The replication crisis appears to be the result of a flawed scientific paradigm rather than the result of a set of individual errors.
The problem of multiple comparisons should be taken into account and an a priori power analysis should be conducted in science. However, when the power analysis is based on previous effect size estimates, there should be caution as these effect sizes are thus often overestimated.
The issue of knowledge translation is a general issue in science, as it is not clear whether case studies can be generalized to the population and whether the observed effect in group studies is sufficiently large to have clinical implications for each individual of a specific group.
There are five major challenges in assessing experimental evidence within clinical neuropsychology:
Kazdin & Weisz (1998). Identifying and developing empirically supported child and adolescent treatments.” - Article summaryThere are several characteristics of therapy with children and adolescents:
Many emotional and behavioural problems that are treated in therapy are often evident in less extreme forms in early development. Treatments have shown beneficial effects in the treatment of children.
Most therapy studies focus on non-referred cases (1), provide relatively brief treatments conducted in group format (2), evaluate treatment in relation to symptom reduction and neglects impairment or adaptive functioning (3), do not evaluate clinical significance of symptom changes (4) and do not conduct a follow-up (5).
Diverse differences among different age groups (e.g. language skills) indicate that treatment with similar general features must differ in numerous specific details when applied in different developmental periods. This leads to a classification dilemma (i.e. what cut-off to use).
A study needs to meet the following criteria to be a good study:
It is likely that dysfunctional anxiety becomes a self-perpetuating cycle of elevated biological response to stress, debilitating cognitions and avoidance of stressful circumstances. CBT appears to be effective for child anxiety.
Depressed children are seen as subject to schemas and cognitive distortions that cast everyday experience in an unduly negative light and as lacking important skills needed to generate supportive social relationships and regulate emotion through daily activity.
Coping skills training (CST) appears to be effective in the treatment of depression for children. It includes structured homework assignments as well as peer or therapist modelling. The mediators and differential effectiveness relative to alternative, simpler treatments still need to be tested.
Cognitive processes refer to a broad class of constructs that
.....read more Kahn (2011). Multilevel modelling: Overview and applications to research in counselling psychology.” - Article summaryMultilevel modelling (MLM) is becoming the standard for analysing nested data.
The unit of analysis problem refers to when the moderator variable exists at a different level than the independent and dependent variable (e.g. moderator variable is on the level of the course itself and the independent and dependent variables are on the individual level). Multilevel modelling (MLM) or hierarchical linear modelling is designed to analyse data that exists at different levels.
When individuals exist in natural groups such as schools, there is a hierarchical or nested structure. Ignoring the nested structure of the data could have adverse consequences. Person-analyses with nested data ignore the fact that individuals sharing a common environment and are more similar to each other than they are to individuals from another environment. Nested data may thus violate the assumption of independence of observations. This leads to an increase of type-I errors due to the small standard errors.
In a typical regression analysis, the slope and the intercept are fixed as the same parameters apply to each case in the sample. However, these parameters may vary as a function of group membership in the case of nested data, thus needing the need for a different analysis.
The formula for the level one, multilevel analysis model is the following:
Yij=β0j+β1jXij-Xj+rij
The letter i refers to the level 1 unit (e.g. individuals). The letter j refers to the level 2 unit (e.g. group). Yij is the level 1 dependent variable (e.g. individual score in a given group). Xij is the score on the level 1 independent variable. This formula makes use of a form of centring, as the score on the level 1 independent variable is subtracted from the mean of the group. β0j is the intercept for a given group (e.g. predicted level of a score in a particular group). This can be interpreted as the mean level of a dependent variable for all people in a particular group. β1j is the slope for a given group. It is the predicted increase in the dependent variable per 1-unit increase in the independent variable within a given group. rij is the residual at the individual level.
The level two model addresses what the average intercept and slope across groups is (i.e. fixed effects) (1), how much intercepts and slopes vary across groups (i.e. random effects) (2) and how useful group-level variables for predicting group intercepts and slopes are (3). The level two model uses the following formula:
The level two model describes the difference between group level and not between individuals within the groups. The first formula represents the level two model for
.....read more Robinaugh, Hoekstra, Toner, & Borsboom (2020). The network approach to psychopathology: A review of the literature 2008-2018 and an agenda for future research.” - Article summary The network model to psychopathology states that mental disorders can be conceptualized and studied as causal systems of mutually reinforcing symptoms. This holds that symptoms are not passive indicators of a latent, common cause, of the disorder but agents in a causal system.
The causality hypothesis states that when causal relations among symptoms are strong, the onset of one symptom will lead to the onset of others. The connectivity hypothesis states that a strongly interconnected symptom network is vulnerable to a contagion effect of spreading activation through the network. Widespread symptom activation as a result of an external stressor can lead to symptoms persisting when the initiating stressor is removed. The centrality hypothesis states that highly central symptoms have greater potential to spread symptom activation throughout the network than do symptoms on the periphery. The comorbidity hypothesis states that symptoms can occur in multiple disorders and that some symptoms can thus bridge different disorders.
A mental disorder is characterized by both the state and the structure of the network (i.e. a mental disorder is characterized by a state of harmful equilibrium). The boundary between health and disorder will vary as a function of network structure. In weakly connected networks, activation varies dimensionally. However, in strongly connected networks, activation within the system rapidly leads to a state of psychopathology (i.e. more discrete rather than continuous).
The momentary perspective states that symptoms are aggregates of moment-to-moment experiences. According to this perspective, these experiences constitute the true building blocks of psychopathology. This highlights the importance of understanding the chronometry of experiences, symptoms and disorders.
The assumptions of the network model currently do not always align with how disorders are believed to operate.
There is a conditional positive manifold for most disorders. This states that symptoms of a positive disorder tend to be positively interconnected, even after controlling for shared variance among symptoms. This suggests meaningful clustering of symptoms in the syndromes we call mental disorders. Connectivity tends to be consistent across time and demographic groups. However, differences have been observed across countries.
Greater connectivity (i.e. network density) may confer risk for psychopathology. This is based on the fact that there appears to be greater connectivity between symptoms in people with more severe mental disorders. It is also possible that greater connectivity leads to disorder persistence. However, there is no consensus regarding these topics. There is some evidence that connectivity of negative mood state networks is associated with psychopathology but minimal evidence that broader networks of momentary experience exhibit such associations.
Node strength refers to the summed absolute strength of a node’s direct link. Non-DSM symptoms often exhibit elevated centrality (e.g. feeling disliked in depression) and some DSM-nodes are weakly connected to the network. It is not clear whether the symptoms which the DSM identifies as especially important are more central to less important DSM-symptoms. The DSM most likely has not captured all symptoms of a disorder and has not
.....read more De Vent et al. (2016). Advanced neuropsychological diagnostics infrastructure (ANDI): A normative database created from control datasets.” - Article summaryIn the advanced neuropsychological diagnostics infrastructure (ANDI), datasets of several research groups are combined into a single database. It contains scores of neuropsychological tests from healthy participants. This allows for accurate neuropsychological assessment as the quantity and the range of the data surpasses most traditional normative data. It facilitates normative comparison methods (e.g. those in which entire profiles of scores are evaluated).
An important element of neuropsychological practice is to determine whether a patient who presents with cognitive complaints has abnormal scores on neuropsychological tests. In the diagnostic process, a number of neuropsychological tests are administered and the test results of a patient are compared to a normative sample.
Scores of patients are typically compared to normative data published in the manuals of an instrument. However, this data may be outdated (1), it may lack norms for very old populations (2), some tests do not have any norms (3), normative scores are often only corrected for age but not for other demographic variables (4) and normative data are often collected for one test at a time (5). This results in univariate but not multivariate data being available while multivariate normative comparison methods are more sensitive to deviating profiles of test scores.
There are several benefits of the ANDI database:
Limitations of the ANDI database are that it is not necessarily based on a random sample (1) and some participants had lenient inclusion criteria and, thus, did not necessarily have no pathology (2).
Maric et al. (2015). Evaluating statistical and clinical significance of intervention effects in single-case experimental designs: An SPSS method to analyze univariate data.” - Article summary Single-case experimental designs (SCED) are useful methods in clinical research to investigate individual client progress. It could also be used to determine whether an intervention works. In single-case experimental designs, a single participant is repeatedly assessed on one or multiple symptoms during various phases (e.g. baseline). The statistical analysis of this data is difficult.
Advantages of the single-case experimental designs are that it can be used to test novel interventions before RCTs are conducted (1), it may be the only way to investigate treatment outcomes in heterogeneous groups (2) and it offers the possibility to systematically document the knowledge of researchers and clinicians, preventing loss of information (3).
The most common single-case experimental design is the AB design. It consists of two phases (i.e. baseline and treatment). It is similar to an interrupted time series. In order to obtain an adequate analysis of the differences, the overall pattern in the time series should be modelled adequately (1) and there should be adequate modelling of potential correlations between residuals (2).
A common assumption of adequate models is that there is a linear function for both the baseline and the treatment phase. Both of these linear functions should be described by an intercept and a slope. Modelling of potential correlations between residuals means that there should be adequate modelling after the overall pattern has been accounted for. The correlation between residuals of the observations is the autocorrelation. It implies that the residuals are not independent. If residuals are correlated, the correlations are likely to decrease with increasing separation between timepoints.
The tests on the intercepts and the slopes of the linear functions will be unreliable if the correlations between the residuals are not modelled adequately (e.g. incorrectly assuming that the residuals are uncorrelated). If positively correlated residuals are assumed to be uncorrelated, the chances of finding significant results will be too high.
Modelling the overall pattern by the intercept and the slope for each phase (i.e. each timepoint) does not yield a direct test of intercept and slope differences. Therefore, it is useful to re-parameterize the model in terms of an intercept and slope for the baseline phase and baseline-treatment differences in the intercepts and slopes. This takes the following formula:
Y(i) denotes the outcome variable score at time point i. Phase(i) denotes the phase in which time point i is contained. Time_in_phase denotes time points within each phase. E(i) denotes the residual at time point i.
The parameter b0 is interpreted as the baseline intercept. B1 is interpreted as the treatment-baseline difference in intercepts. B2 is interpreted as the baseline slope and b3 is interpreted as the treatment-baseline differences in slopes. These parameters can also be interpreted as effect sizes.
The b0 and b1 refer to symptom scores when time_in_phase is zero. Therefore, when coding the variables, time_in_phase zero should denote the start of each phase. However, when time_in_phase zero
.....read more
JoHo kan jouw hulp goed gebruiken! Check hier de diverse studentenbanen die aansluiten bij je studie, je competenties verbeteren, je cv versterken en een bijdrage leveren aan een tolerantere wereld
There are several ways to navigate the large amount of summaries, study notes en practice exams on JoHo WorldSupporter.
Do you want to share your summaries with JoHo WorldSupporter and its visitors?
Field of study
Add new contribution