Notes on: Quinlan,K., Male, S., Baillie, C., Stamboulis A, Fill J, and Jaffer Z. (2013) 'Methodological challenges in researching threshold concepts: a comparative analysis of three projects'. Higher Education 66: 585--601

Dave Harris

The work by Land and Meyer has been very influential, but it does not seem to have produced a research methodology or a discourse about methodology.  These projects, undertaken by the authors, refer to engineering.  They all used interviews with students and academics, but there were other differences which affected outcomes, including 'differences in project goals, researchers' backgrounds, curricular context, participants' experiences, negotiated or independent knowledge and degree of comprehensiveness sought' (585).

[Their definition of a threshold refers to the metaphor of the portal and transformation].  The notion does engage academics in conversations about teaching within their disciplines.  Lots of work has been done on economics, computer science, and biology [refs on 586], designed to focus teachers time, and help them discuss methods of teaching, especially with difficult concepts.

A variety of methods have been used, including interviews, surveys, observations and analyses of grades and course feedback.  There are no quantitative approaches as yet, and discussions of methods generally are not usually detailed.  The original notion had eight or potential key factors, but two are particularly 'nonnegotiable'[according to a personal communication from Meyer]: epistemological and ontological transformation, enabling students to see the world differently.  However, these features are not always well debated, and most authors assume a need for different selections to be present, although they are rarely saying why they selected particular features: different selections could well lead to different methods.

There is also the question of who defines threshold concepts, especially for constructed lists, while positivists simply 'may assume there is a fundamental truth' about academic disciplines.  They similarly vary in terms of the importance to be placed on social context.  Phenomenographic researchers focus on perceptions of students and teachers, starting with experience not theory, and showing different ways of experiencing features of the threshold concept [there is a reference to Bowden and Marton 1998 The university of learning: Beyond quality and competence in higher education.  London: Kogan Page].  Researchers can influence the research process themselves, by choosing which questions to ask them how to follow up, bringing in their own interests and background knowledge: some will thematize and code, for example.  Some used grounded theory, concept mapping and case studies as well as phenomenography.  More explicit methodological discussion is needed [apparently also addressed by Kinchin 2011 'Visualising knowledge structures in biology: Discipline, curriculum and student understanding'.  Journal of Biological Education 45 (4): 183-189].

Three case studies were chosen, all in engineering and materials, all focusing on the first year of study.  Methodological differences were suspected of making a difference to the outcome, especially choosing the criteria for the focus of research, and who defines what might be a threshold.  The three cases were discussed after writing different drafts of case study methods and discussing them with participants.  The three teams also engaged in two exercises to identify key differences.  One involved concept mapping with a comparison of the concept maps.  The second one involved combining data from the different projects that dealt with a single topic to show what the different teams made of these data [there is another article on the results of the second discussion, online].  Overall, the project started in Australia and then got some comparative work at Birmingham and Oxford.

Each project had different methods.  The Australian one used phenomenography, based on the experience of Baillie.  The approach investigated whether the negotiated knowledge that emerged did indeed change views of the world.  The point was to not define a threshold concept but to study the experience of teachers and students as they meet troublesome concepts or what teachers see as transformative ones.  Trouble can arise from 'the cultural context, the teaching, the teacher, the students background or learning approach' or from developments in the discipline (589).  The team met some resistance from the engineers asking for a clear definition of a threshold concept, however.  Both main researchers were trained in engineering and education, enabling some specialist discussion of the engineering concepts.  The focus was on transformation and troublesomeness.  There was an initial diverging phase, where tutors identified threshold concepts in their own specialist areas of teaching, then an integrating phase where they came together to discuss what they might have in common despite their diverse perspectives.  Particularly valuable were discussions between students from different disciplines and years; between students and academics; between academics from different specialities inside engineering; between academics in different engineering and science disciplines; between academics from different universities.  Interviews or focus groups were held, and workshops for students, and for both students and staff.  There were audio recordings from interviews, focus groups and workshops, notes taken by the researchers, and handouts to be completed by participants.  Data were analysed to identify troublesome and transformative concepts.  Whole transcripts or notes were analysed rather than segments.  The result was 'an inventory of potential thresholds' (590) [reported in another paper -these blokes know how to salami slice!].  Parts of the inventory were re-presented to participants for further negotiation.  [Then there seems to have been some subsequent recoding to identify 'overarching threshold concepts of which previous items were examples, or underlying threshold concepts on which other thresholds were built'- these were based on the results of participant negotiation and also conceptual comparisons].  Teachers 'pooled' examples they had had identified.  Those that remained were experienced as threshold by at least one student, and most of the others had been experienced as such by many students.

The second case study at Oxford interviewed 14 academics who tutor first year students.  Each tutor had to suggest one major threshold, and their responses were then further probed thing for 'transformation, integration and troublesomeness' and how they would know if students had got the idea.  The discussion was about thresholds rather than threshold concepts as such, since there had been a problem identifying a concept in the Australian study.  The curriculum is delivered by a lecture series, with accompanying problem sets for students.  They also have a tutorial usually in pairs.  The idea is to develop a long-term relation with small cohorts of students who will tutor them over several years.  Eight students were also interviewed, and they were asked to identify thresholds which were then probed in the same way.  More attention was given to learning strategies and perceptions of the tutorial process.  Students are asked to reflect back on their experiences.  They were also given a prompt list based on topics tutors had identified.  The interviews were analyzed by reading the transcripts and then identifying segments of text which could then be coded.  This identified topics for further discussion, and included thresholds, transformations, blocks and barriers.  Sub codes were then developed.  All the texts could be used to find examples of these codes enabling the comparison of transcripts [lots of references to grounded theory here].  Concepts identified as difficult in the interviews were then clustered into a one page diagram.  The team further experimented with concept mapping [Novak, J (1990) 'Concept mapping: a useful tool for science education'.  Journal of Research in Science Teaching, 27 (10): 937 – 49] to illustrate vertical connections between ideas.  A workshop was held to review and interpret findings, with some outsiders present, and discussions lead to further data to 'verified, clarify and elaborate the findings'.

The third case study at the University of Birmingham was based on action research approaches, with teachers reflecting on their own students' learning.  There was also an expert on engineering threshold concepts.  Student and lecturer interviews were undertaken, together with some focus groups.  Both first and second year students were interviewed, with the intention of identifying troublesome threshold concepts - students were asked if they struggled with particular topics, how they understood them and their application, how they decided, and what they did to cope.  And lecturers are asked about how they viewed the module, which topics they thought students would struggle with, what their own experience was like, and how they assessed whether learning outcomes had been met.  The result was an initial list of troublesome concepts, although these might not have been 'truly threshold concepts' (592).  Again concept maps were constructed where there was 'considerable agreement that a concept was troublesome', and also drawing on the syllabus and learning outcomes, and textbooks.  Transcripts were reviewed to identify list of topics and explanations for difficulty, and any other information.  This list was then 'refined' involving project advisers and education [and engineering] experts 'familiar with threshold concepts'.  It seems that lecturers were not very helpful because they did not 'reflect deeply on the learning experience of students', so focus shifted to perceptions of thresholds by students and teachers, 'compared with textbook notions of curriculum content'.  The connecting words  associated with topics [provided by interviewees] were added to the concept maps: these connections 'describe the relationships between potential threshold concepts'.  This helped probe presenting difficulties: phase diagrams were seen as troublesome by many students, but there are a number of concepts that are required before they can be understood, and the concept map helped identify them.  The words they actually used to explain their difficulties helped understand the perception of the [expert] relationships and subordinate concepts - they mentioned the need to visualize processes, and to use adequate maths, for example.

The goals of the research exercises were different.  The Australian one was intended to inform curricular reform and build consensus.  The Oxford one was designed to understand the role of tutorials.  At Birmingham, the aim was to improve the particular first year modules.  Researchers also differed in terms of their backgrounds and 'epistemological orientations'.  Australian researchers had backgrounds in engineering and education, and were keen on phenomenography.  At Oxford, the main investigator was an education researcher interested in constructivism and qualitative research.  At Birmingham, the principal investigator was a materials science person interested in practitioner research with no particular expertise in educational research as such.

The context of the exercise was different as well, for example whether it covered large courses or single modules.  Larger courses involve problems of integrating underlying ideas.  The Oxford tutors working across different parts of the programme were much more interested in 'more abstract thresholds', and 'problem solving processes', or connecting maths and the physical world, which are elements underlying all engineering.  The Australian researchers found most interest in the integrating phase, discussing generic engineering and its threshold concept 'system identification' which has different embodiments.  The Birmingham study by focusing on one particular unit did permit more attention to difficulties and relations between concepts in one specific module.

All three studies used interviews and focus groups primarily, but interviewees differed: some were more specialized and isolated and less interested in integration, and their students were also less likely to identify common principles.  It was difficult for senior tutors and lecturers to have remembered their own experiences as students.  Sometimes student meanings did not relate well to 'the underlying concept'.  The views of participants looking back were often more useful.  The Australian study moved on to discuss their findings with people all over the world in different contexts.

Another difference turned on negotiated or independent knowledge: some interviews were 'conducted iteratively', in a process deliberately attempting to build consensus, and this produced negotiated knowledge.  Others were interviewed independently, so that each tutor at Oxford gave their own opinion, and the researchers decided whether common themes emerged unprompted: even here, there may be a particularly strong common ethos at Oxford.  Both approaches can be justified, but researchers need to be clear about their research design instead of just referring to identifying threshold concepts.

Approaches also varied to the extent to which they 'sought comprehensiveness and depth' in technical understanding (595).  At Oxford, more depth was pursued by asking tutors to focus on only one or two perceived thresholds, with the intention of trying to grasp the effectiveness of tutorials.  Curriculum design was not questioned.  As argued above, more abstract and general issues tended to emerge.  At Birmingham and in Australia, there was a more exhaustive list, but more interest in the technical nature of concepts themselves, and the detail of what produced them as troublesome - as in difficulties in grasping visual representations with the phase diagrams.  We still need to know why matters such as graphical representations might lead to difficulties.  Substantial inventories and then technical concept maps were used to pursue these issues.

There are factors affecting data.  Researchers need to compare and contrast there approaches and develop more rigorous protocol's.  There are still problems about the best methods to use and analyze the data, especially as interest in the threshold concept widens to include different schools of thought.  A useful way forward might be to begin with defining criteria for thresholds, then thinking of different sorts of research methods to explore different dimensions: if troublesomeness is the most important, it makes sense to look at student experience [rocket science!] [Apparently, there is some useful American based research on misconceptions in science, 597.  This shows troublesomeness or difficulty associated with some concepts, but not their transformative or integrative qualities].

If we were interested in integration, we might begin looking for key principles that lie beneath lots of examples and applications.  Here, we might offer students a variety of examples and ask them to organize them conceptually, perhaps using card sorts [Kelly grids!]: Again we would not know about transformations [we would if we used Kelly grids to look at laddering].  Each of the other characteristics might have its own special methodology.  This bloke Flanagan who has compiled a bibliography of threshold concept papers has indexed particular words, with the order of frequency of mention being 'threshold concept', 'troublesome', 'transform', 'liminality' and 'integrate'[looks interesting - Flanagan, M.   Smith, J (2006) 'Threshold concepts: Troublesome topographies for the Google generation', a paper given at the annual conference of SRHE, Brighton].  We can see that there are at least eight different and distinct research questions.

Theoretical traditions of researchers will also vary.  In social sciences, we find multiple theories 'to describe largely the same thing'(597), yet researchers tend to live in silos.  Looking at threshold concepts would permit wider discussion and enrich research.

If transformation is the essential feature, as Meyer suggests, this might indicate some future directions.  Transformative learning has already been discussed in adult learning theories [I thought of Perry, who comes up in a minute as an example of a 'student development theory'.  The reference for transformative theory is Mezirow and Taylor (2009), although blow me down, it's not mentioned in the references!].  Student development theory with stages of epistemological development are also relevant, and context free, although this can also be a problem, and it needs to be 'translated into particular disciplines' (598).

Overall, simply seeking to identify threshold concepts 'was ultimately unproductive.  It is a form of disciplinary essentialism and positivism'.  Differences including the focusing on particular dimensions, and attending to methodological issues  are more important.  Better to say, for example, that we are interested in tutors' perceptions of integrative concepts that are transformative for students.  The methodological issues identified are well known, but they need to be contextualised in this sub field.

back to ed studs page