Saturday, 17 July 2010

How reliable is alpha reliability?

One of the most popular reliability statistics in use today is Cronbach's alpha (Cronbach, 1951). It is widely used as de-facto indicator of reliability and sometimes used blindly by test administrators to determine reliability of the tool. 3 common misconceptions associated with alpha reliability are:
1. High alpha means the test score will be almost the same for the same individual if the test is taken twice.
2. The higher the alpha, the better the test.
3. Alpha reliability is the property of the test.
If you think the three statements above are anywhere close to reality, maybe there is a need to revisit the concepts.

What is reliability?
As per most standard definitions reliability is ‘stability of scores over time as per the concept/construct’, the words as ‘stability’ and ‘as per the concept’ are key to understand reliability. First characteristic of a reliable instrument is that it gives similar results on two subsequent administrations on the same group of people. For example, if a candidate`s IQ is assessed as 120 in the first administration he/she should get scores close to 120 on all subsequent administrations (This is called test-retest reliability in standard terms). If a candidate gets score very different from 120, it raises questions about the reliability of instrument. Second notion attached to this is ‘the variation of scores has to be as per concept’. Let’s take an example to make it clearer. Suppose, there is a tool which measures age and there is a candidate whose age as to today is 18years 6 months and 3 days. If the tool administered today gives an assessment that age is 19 years and after one year, on second administration, it gives an assessment that age is 19 years. The scores are stable but is the tool reliable? Obviously no. why? Because after one year it should show some increase in the score as actual age goes up by 365 days in 1 year.

The illustration above clearly indicates that mere ‘stability’ is not enough to determine reliability. The stability has to be considered as per the concept or construct the tool is measuring. Psychological constructs are not as concrete as age and this makes the issue little more subtle. For psychological constructs we could divide them into two categories less changeable and more changeable over time and a good instrument should demonstrate less changed scores on less changeable constructs overtime and more changed scores on more changeable constructs.
Most test-developers do not report this is their test administration manual because
1. It asks for a lot of money to contact same group of candidates over a period of time and get them the test taken twice.
2. It demands for more time than the other kind of reliability estimation known as alpha reliability.

What is Cronbach's alpha reliability?
Alpha reliability is an indicator to assess the other kind of construct in the realm of reliability and that is ‘consistency of scores on different items of a single scale’ in a dataset. An illustration, let’s assume that there are 5 questions in a scale and a candidate scores low on first two, medium on the third one and gets high scores on the last two items. Is that a reliable set of score? Can you add up these scores to get a total scale score? The answer to these questions are possibly ‘no’, we cannot safely assume that all these 5 questions are measuring the same thing. But if high or low score on one question corresponds well with high or low score on other questions, then we can be rather confident that all questions are measuring almost the same thing. This is what alpha reliability tells, ‘to what extent the scores on the questions in this scale correspond with other questions’. Technically this translates to ‘what is the amount of shared variance among questions of a scale’. If the correspondence (shared variance) is high that results into an alpha greater than .70, if the correspondence is low that results into a lower alpha.
Since alpha is most reported reliability indicator, lets understand a little bit more by examining how it is calculated. Here is the formula to calculate alpha:
Here N is equal to the number of items, c-bar is the average inter-item covariance among the items and v-bar equals the average variance.

One can see from this formula that if you increase the number of items, you increase Cronbach's alpha. Additionally, if the average inter-item correlation is low, alpha will be low. As the average inter-item correlation increases, Cronbach’s alpha increases as well (holding the number of items constant).

Now, let’s go back to common misconceptions noted at the beginning of the article. First, misconception is that high alpha suggests high possibility that scores will be the same. As is obvious from the formula that it is confined to one single dataset only (there is nothing in Cronbach`s alpha formula that compares scores of two subsequent test administrations). The second misconception, ‘the higher the alpha the better the test’ is because most test users are not aware that a test developer can increase the alpha artificially (without adding any value to the tool) just by adding some more questions and can increase it significantly by adding questions which are just the rephrased version of another question (two rephrased version of a single question are likely to have more than .70 correlation, thus increasing the alpha). Adding paraphrased version of a question in the scale does not add any more information about the subject under assessment. It, therefore, does not provide any value to the test administrator but it provides the test developer to present his test in more favorable way. A test administrator should therefore look at two things to determine the practical utility of alpha coefficient. Alpha coefficient and inter-item correlation matrix. If there is any pair of items (questions) having high positive correlation (suggesting presence of rephrased items) the overall alpha reliability coefficient should be taken with a pinch of salt.

The third most common misconception is that alpha reliability is property of the test (tool). It translates into the assumption that a tool with high alpha will have high internal consistency in all situations, which is not true. Alpha depends upon the inter-correlations among the questions (items) and in different samples, these inter-correlations vary depending upon the sample characteristics and the way test was administered. Assuming that alpha obtained in standardization sample will replicate itself in all other samples, and in all other situations, is likely to lead to less useful results. Test administrators, therefore should examine reliability closely by looking at inter-item correlation matrix, sample characteristics and standard administration process and not go by just one reliability coefficient which tells almost nothing (if used in isolation) about the tool. And lastly, a low alpha indicates that the tool is useless but a high alpha does not indicate that the tool is useful.