BASIC QC
#01 CHECK RESPONSE VALUES (IF THEY ARE CODED INTO CORRECT NUMERALS)
Items used for Rasch model analysis are usually ordinal variables based on response values such as “Strongly agree,” Agree,” “Disagree,” and “Strongly Disagree.” Code these so that higher agreement receives higher numbers:
- Strongly agree 4
- Agree 3
- Disagree 2
- Strongly Disagree 1
If by mistake these numbers are flipped, you will have a catastrophic situation where the result is flipped. Do two things to prevent such a catastrophe:
- Confirm this by looking at the actual survey and by looking at the data (Look at it until your eyes bleed).
- People are likely to agree with items as they have social pressure to report good things when taking a survey. Look at the original data and see if you see a lot of positive responses.
#02 CHECK THE N OF SUBJECTS INCLUDED IN THE ANALYSIS
Check the output and confirm that the number of subject used is correct. Checking the number of subject is el numero uno protection against errors.
#03 CHECK THE N OF ITEMS INCLUDED IN THE ANALYSIS
Check the output and confirm that the number of items used is correct. Especially when you are not using all item’s data in your analysis (you might have decided to drop some items), be sure you used the ones you wanted to use. With Winsteps, misspecification of a control file can lead to inclusion of subject IDs as response data by mistake. Avoid this (such a case will produce an extremely low reliability score).
#04 CHECK WHAT VALUE WAS USED FOR MISSING SCORES
When a subject does not provide any response, Winsteps imputes a token number (-2, I think) to indicate that it is a missing value. This value should be treated as a missing value and should NOT be included in the analysis dataset. If you treat a token value (-2 in the case of Winsteps) as a true value, you will have a catastrophic situation where you have an arbitrary value used as a real data point. You should replace such a number with “.” (dot) before analysis as statistical software, such as SAS or SPSS, will treat a dot as a missing value.
Winteps Reference: Definition of status variable in Winsteps output
When a subject lacks data, missing value is indicated by -2.
http://www.winsteps.com/winman/ifile.htm
ADVANCED QC
Basic QC procedures should catch 99% of errors. Advanced ones are more intricate ones.
#05 INVESTIGATE ITEM DIFFICULTY SCORES
If you are using item difficulty parameters provided by the developer, compare them against the ones you derived from the dataset you collected. They must be more or less comparable. If not, investigate whether it is caused by a data error.
#06 HISTORICALLY COMPARE RESULTS
If you are repeating the study, compare your results with historical data (e.g., last year’s result).