When critically reviewing an experiment it is important to consider whether the experiment achieved what it set out to do and the applicability of these findings to the design. This can be assessed by examining the experimental procedure and the results obtained. To gain an understanding of an experimental procedure it is useful to view it from the subject's perspective. This can be a valuable exercise, both in planning experiments and for reviewing reports of other people's experiments.
There are four key issues to consider:
User preparation whether the instructions given to the users were adequate and whether the amount of practice allowed before starting the experiments was sufficient.
Impact of variables what the changes in the independent variables mean to the users as they undertake the experimental tasks.
Structure of the tasks whether the tasks were complex enough to allow the use of the interface facilities (or at least the ones of interest) and whether the users understood the aims of the tasks.
Time taken whether the length of the task sequence produced any fatigue or boredom in the users.
While an experiment can be very well designed at the technical level, these practical issues can have marked effects on the results. For example, an experiment that involves complex tasks and has few practice tasks may be predisposed to large error scores and poor user performance; an experiment that uses very long sequences of tasks may produce fatigue in the users. These kinds of pitfall can be avoided by carrying out small pilot studies before running the experiment on a larger scale. Although this practice might require more preparation time, it can help to avoid cumbersome or unnecessary data collection and analysis, so saving time and money and avoiding frustration in the longer term.
Experimental results need to be critically reviewed in order to establish exactly what has been found out, how useful this is, and whether it is of practical as well as theoretical significance.
There are four main points to consider:
Size of effect the absolute size of the differences found in the dependent variables is important in assessing the results. For example, performance differences of perhaps a few seconds may be statistically significant, but from a practical point of view, they may have little impact when the interface is used in a normal working environment where there are all kinds of distractions and interruptions.
Alternative interpretations experimental results are interpreted as arising from the manipulation of the independent variables. It is useful to consider whether there are any alternative interpretations of the results, perhaps based on other variables which may not have been controlled in the experiment. For example, this could be done by examining the effects of insufficient practice on complex task performance.
Consistency between dependent variables when several dependant variables are used in one experiment the relationship between them should be studied. In some cases, there may be inconsistency across variables. For example, task completion rates and error scores may indicate that one interface is better than another but user preferences and learning scores may show the reverse. Such inconsistencies indicate that the situation is complex and further experiments may be needed.
Generalization of results depending on the nature of an experiments its results may not generalize to other tasks, users, or working environments. For example, the results obtained from an experiment using one multimedia learning system may not be applicable to another. It is dangerous to over-generalize experimental results, particularly when the results are given the status of "guidelines".
Understanding and being able to apply statistical tests to validate experimental findings is important.
Photo by Tobias van Schneider on Unsplash
Comments