motivnlscale

Smith, R, Cumming, S, and Smoll, F (2008) 'Development and Validation of the Motivational Climate Scale for Youth Sports', in Journal of Applied Sport Psychology, 20: 116 -- 36.

[This is an amazing and detailed account of the development of a particular psychological scale to measure the motivational climate in sport for younger people (9 --14). It offers an excellent example of how research psychologists actually do develop these scales that so many of us just use. There's a great deal of work to make sure that scales are internally consistent and that the results correspond to what else is known about motivation. In what follows, I shall be describing mostly the strategy that the authors pursued to validate their scale. You'll have to read the article itself for details of which measures were actually used, which correlations were associated with which measures, and so on.]

Motivational climate has been researched extensively before, in connection with trying to find factors associated with success. There seemed to be two main orientations -- mastery or task involvement, where participants believe that 'ability is demonstrated through the development and/or mastery of new skills, putting forth maximum effort, and improving one's performance close', while ego involvement involves a 'social comparison conception of success demonstrated when... performance surpasses that of others or when they perform equally well but with less... effort... people are motivated to avoid perceived failure' (116). Apparently, situational factors can affect these orientations, including coaching behaviour such as the distribution of rewards and sanctions, and the nature of interpersonal interactions. Thus ego involvement is encouraged by 'an emphasis on outperforming others, a focus on outcome, preferential attention to top performers, and punishment of mistakes' (117). [Here, as throughout, there are many extensive careful references to these assertions].

Apparently, there is a widely used questionnaire and scale to measure motivational climate -- the PMCSQ [perceived motivational climate in sport question] -- which has since been improved and is now more reliable and valid. However, there are problems using it for children below the age of 11. Such children may not make the same sort of distinctions between effort and ability, for example, and may not be able to classify coach behaviours. They may also experience different sorts of anxiety. There is also the issue of the readability levels of the items in the questionnaire. This research comes from a programme designed to help intervention in youth sport by coaches and parents. The team felt the need to develop more appropriate measures for younger children, especially focusing on readability [they used the Flesch -Kincaid index that used to be found bundled up with copies of Word. Basically, this assesses readability by the number of polysyllabic words].

First a panel of experts generated the initial draft of the scale with 'face - valid items' that corresponded to the adult scale but at a suitable reading level. The reading level was then checked, and any demanding items rejected. The initial test was piloted, with young athletes being asked about whether they understood the items. This left 16 items, and each one was assessed by a five point Likert scale.

Then a large overall sample was split into six independent samples, each one of which would be used to evaluate particular items or stages. Thus sample one looked at the clarity of the language in the questionnaire; sample two was used to undertake some factor analysis [basically to test interrelationships between items in the questionnaire]; sample three was used in 'confirmatory factor analysis' and other measures of validity; sample four was used to test reliability and validity; sample five looked at 'test - re-test reliability data'; sample six explored issues of class and ethnicity (118 - 9). A number of measures and other data were gathered from the samples.

An early stage was to undertake 'exploratory factor analyses'. The idea here is to see if the new scale displays the same kind of correlations between the items as the old one [in particular, a negative correlation between task and ego scales]. [Detailed accounts of the statistical analyses then follow, pages 119 - 120]. The factor analysis also provided a slight regrouping of the items into two modified subscales -- 'Mastery and Ego' (119). Analysis showed these two main factors explained 49.6 per cent of the variance [which apparently is a good result]. The actual items that emerged are summarised in a table on page 120, and include things like 'the coach made players feel good when they improved a skill', down to 'coach told us to try to be better than our team mates'.

Having established that there were two distinct factors underlying the two dimensions of mastery and ego, further analysis was undertaken -- 'confirmatory factor analyses'. Again the details are available pages 120 - 21. The team assessed the goodness of fit of their model with the data using a whole range of indices including chi square. The model showed a strong fit both to the total sample and to sample three, and this led to the final version of the scale [included in the Appendix].

Age and gender differences were examined by comparing statistics for male and female athletes from samples two and three combined. A table on page 121 shows comparisons between mean scores on the test and standard deviations of males and females, split into age groups. For age differences, overall, all participants suggested that their coaching was more mastery than ego, although the older group reported smaller differences 'Such a pattern would be consistent with a greater emphasis on social comparisons and winning at the older age level' (122). For gender differences, we would expect differences arising from different socializing experiences and from previous research (122). Generally, ego-involving coach behaviour was more common with males, following an analysis of variance on the total sample and for each age group.

The scale itself was tested for consistency by combining samples, using a particular statistical test [the Cronbach alpha coefficient]. This scale reached the acceptable level of consistency. Test - retest reliability was also assessed with sample five -- they took the test and were then retested one week later. Again, there was adequate reliability.

There is still a problem with validity, however. Ideally, the scale should also fit with constructs derived from theoretically related studies. The authors even propose a strong test that there should be a relation with constructs in unrelated studies. This was done by combining samples 2, 3 and 4 and then examining the results. First, there should be a negative correlation between the two main approaches as with other tests. Then there should be some correlation with other measures of 'achievement goal orientation, competitive traits anxiety, self-esteem, and intrinsic motivation', which previous research has suggested is relevant (123). They examined results following an experimental intervention designed to influence motivational climate. They explored relations with 'social desirability'. Again, they tested results against predictions.

Because coaches have to choose either mastery or ego styles, and because these are incompatible, we would expect to find an actual negative correlation between them, not just a low one. Other researchers have found such a negative relation. These researchers also found a negative relation, but it was weaker.

Motivational climate is assumed to affect long-term goal orientations of athletes. This was tested by correlating the results gained in this study with other measures of goal orientation. Correlations were positive and consistent when results were compared with the results on two independent scales. The mastery and ego scores gave different results as predicted [this time, there was just a low correlation, not a negative one]. The team were even able to pursue a longitudinal study over the course of the sport season, and found 'significant partial correlations between... [their own scales]... and goal orientation change scores... [on these other independent measures]' (124). The authors accept that these results are only 'correlational in nature', but suggest there is enough evidence to support theoretical expectations about these links.

The next issue was to consider connections with notions of intrinsic and extrinsic motivation. According to one study, intrinsic motivation 'lies on one end of the continuum of self determination' (125), and there are four kinds of extrinsic motivation:

Integrated regulations (in which the activity is congruent with other aspects of the self); identified regulation (participation based on value to the self, such as self-esteem maintenance); introjected regulation (participation because of internalized norms and "shoulds"); and extrinsic regulation (participation for external rewards, such as money or trophies) (125)

On this continuum we also find amotivation, where people are not involved but feel they have no choice. These aspects of motivation are connected with motivational climate, in that mastery climates are supposed to encourage 'autonomy,... relatedness, ...and a sense of competence... [which]... thereby enhance intrinsic motivation, whereas an ego climate... will undermine intrinsic motivation' (125).

In order to test these associations with types of motivation, sample four were also given a standard test asking for reasons for participation based on the notion of self determination. Again, tests of consistency matched those of other studies, and overall, 'mastery scores were positively related to... intrinsic motivation scores, whereas ego scores were negatively related' (125). Incidentally, ego scores were correlated with scores for amotivation.

Other work also predicts that there will be a connection with performance anxiety, with ego climates positively related to anxiety and a feeling of psychological threat, while a mastery climate accepts mistakes as learning opportunities, and results as reflecting collective resources not personal ones. The association was tested by correlations with a suitable sport anxiety scale, and there was evidence of a strong fit and internal consistency.

Another prediction is that mastery involved climates will produce positive relations with self-esteem, by avoiding the personal threats of ego climates. Several studies report this relation. This team administered a scale based on one of these other studies and found consistency and positive correlation with the results.

Sport enjoyment is also related to mastery climates, and to attitudes towards the coach. Such attitudes can be tested by a seven-point scale, and results correlated with results on the climate scale. There were significant and positive relationships between mastery climate, liking for the coach, and desire to play for the coach in future. Variation in evaluative responses to the coach were more affected by these factors of climate than by the actual results! (126). Enjoyment and attraction towards team-mates were also positively correlated with mastery, and negatively correlated with ego.

There was even an attempt at an experimental manipulation. A particular educational programme, designed to help coaches establish a mastery climate, and discourage an ego climate was evaluated. The test was whether athletes' perceptions of the climate would change. One experimental group received the treatment, and there was a control group which did not, and all participants were tested on the motivational scale test. Trained coaches were rated by their athletes as creating 'a more mastery - involving climate than were the coaches in the control group' (127). That group also produced lower scores on the ego scale. There were also differences in achievement goal orientations and a decrease in performance anxiety. Training for mastery was also delivered for coaches on 'an inner-city recreational basketball programme', affecting mostly ethnic minority groups. Results for the groups were similar before the training, and differed by the end of the season in the predicted direction -- trained coaches produced higher mastery and lower ego scores. These experiments indicate both that the motivational scale is 'sensitive to experimental intervention', and also, that it has a practical use with 'lower socio-economic and populations [sic -- ethnic?], which are likely to have lower reading abilities than middle-class populations' (127).

The team are aware of the problem known as 'social desirability', where respondents give socially desirable answers. They assumed mastery behaviours would be seen as more socially desirable, and tested this by correlating their results with a special desirability scale. They did indeed find a positive correlation, but they were 'quite low' in samples two, three and four (127).

Finally, there should be some [real] agreement between athletes in their climate scores. However, such agreement can be 'a subjectively experienced product of individual perceptions' (128). One way of testing this is to calculate 'intra-class correlation, which indexes the amount of test score variance that is attributable to shared perceptions' (128). They did this with the people in sample three, and found only a modest correlation: 'It thus appears that less than 20 per cent of the variance represents common perceptions among team mates/class mates' (128). [This is a problem I think -- see below]

[This is one that baffles me, probably because I do not know the psychological work involved. The issue is further discussed a bit later]. Motivational climate appears to be a 'group level situational variable', which means that reports of coaches behaviour should be consistent within the group being coached. However, there will also be some individualized, personalized and therefore unshared experiences as well. Apparently, intra-class correlations tell us what can be attributed to common perceptions among members of the group, as a test of inter-rater reliability [as in inter- coder reliability?]. Apparently different raters are measured in terms of their agreement when observing particular behavioural segments, and they are not very consistent in this case. This lack of subjective agreement may arise because personal experiences are more vivid, and because athletes may be less able to see how coaches interact with other team members. Coaches may also moderate their motivational behaviour according to the ability of the athlete. [So far, low inter-rater reliability seems to be a problem, then, which is not the impression I gained earlier]. One way round this is to get professional coders to code behaviour, although so far, existing coding systems do not correlate that well with student reports. Perhaps the coders need to code athlete behaviour as well. However, it is probably the individual perception of the climate that has the greatest effect on subsequent behaviour .

Overall, 'much remains to be learned about motivational climate, its measurement, and its effects on sport outcome' (131). [This problem is returned to again in the final section -- and in the last paragraph of this reading guide].

It seems athletes can discriminate between coaching behaviours, even when younger. The results of all the factor analysis and correlations convinces the researchers that they 'are on solid psychometric grounds' in using their scale (128). The particular scale is also brief and easy to administer, making it suitable for longitudinal research. The scale is consistent internally, and appears reliable, although further test - re-test exercises would be ideal. However, there is little additional work on how stable motivational climate perceptions are, and there is always the danger that particularly memorable events will influence the overall judgment. There is evidence that these two coaching behaviours are incompatible, that mastery types seem more common with young people and women: 'It seems likely that with older ages and in more competitive programmes, ego - initiating coaching behaviours would become more common' (129).

However, despite the impressive validity of the scale in terms of its correlations with other relevant variables, it is still the case that the situation might be different in the field. The scale still measures 'dispositional goal orientations', although it should be suitable for field trials as well (129).

More generally, theory development is intrinsically connected with the development of measurement techniques: some general theoretical issues are raised by the work to test the scale [theoretical issues in the limited psychological sense of identifying pyschoimetric categories?] [They include the issues of measurement discussed above under intra-class correlation]. Motivational climate might need even more specific analysis than the basic mastery or ego distinction, and some recent work is cited on page 131. One study turns on spelling out some of the implications of achievement goal theory, and describing 'avoidant climates' (131) in their own right, and not as part of a more general ego involvement. Another new scale has 'mastery approach, mastery avoidance, performance approach, and performance avoidance climates' (131), for example. This scale asks athletes to infer their coach's goals, rather than to list behaviours, and there is an issue concerning whether children are able to do this accurately. They are yet to be tested as rigorously as this scale.

The final problem returns to the issue of individuals and teams. People in teams are already likely to have converged on particular variables, perhaps including the ones tested here. As a result, gaining data from individuals which are then aggregated may not be appropriate, since they are not independent individuals. Multi-level statistical modelling might be the answer, since it apparently helps us analyze 'nested data in... aggregated units' (132). This technique might solve the problem associated with intra-class correlations mentioned above.