Many studies of stroke rehabilitation use the Action Research Arm Test (ARAT) as an outcome, which measures upper limb function by scoring the ability to complete functional tasks. This report describes an issue encountered when analysing the ARAT subscales in a trial of upper limb therapies after stroke. The subscales of the ARAT at three months followed a ‘U-shaped’ distribution, and therefore, comparing means or medians was not appropriate. A simple alternative approach was chosen that dichotomised the subscales. When analysing the ARAT, the shape of the distributions must be checked in order to choose the most appropriate descriptive and inferential statistical techniques. In particular, if the data follows a ‘U-shaped’ distribution, a simple dichotomising or a more sophisticated approach is needed. These should also be considered for heavily skewed distributions, often arising from substantial floor or ceiling effects. Inappropriate analyses can lead to misleading conclusions.