Anticipating changes in the measurement and assessment of teamwork over the next decade

Much is known about the acquisition and persistence of individual skills, but very little in regard to team skills: How do team process skills develop? Are team skills learned in one context transferable to other contexts? How long do unused team skills persist, and what training or interventions can most quickly restore them?

Filling this knowledge gap is important for the development of cost-effective training that produces quantifiable behaviour change in teams. However, this is a challenging task due to:

• a lack of quantitative teamwork measures
• difficulties in taking a scientific approach because of the absence of theory and methods aligned to the dynamics of team performance, and
• questions about how researchers will be able to make sense of the expected coming onslaught of detailed data about teams.

These issues suggest the need for novel methods and ways of thinking about team dynamics and measurement, and this was the focus of a panel presentation¹ to the Human Factors and Ergonomics Society 59th Annual Meeting. The five panel members were asked to speculate where the measurement and assessment of learning and performance of teams and of individuals in teams might go in the next decade and how we might get there.

Toward Quantitative Descriptions of the Neurodynamic Organizations of Teams – Ron Stevens, UCLA School of Medicine, The Learning Chameleon, Inc.

Using symbolic models of teamwork that capture and context the brainwave levels of team members, Stevens and his colleagues have been able to show that:

• neurodynamic rhythms are measurable and entrained by the task
• the structure of these rhythms is multifractal, and
• quantitative differences in team’s neurodynamic rhythms are linked with team expertise, resilience, and communication.

Measured teams were positioned at an organizational neurodynamic point midway between rigid and elastic, suggesting that there are team-related neurodynamical processes that quantitatively track across the novice-expert continuum.

Team Performance at the Level of Emergent, Dynamic Coordinative Relations – Jamie C. Gorman, Georgia Institute of Technology

Recent simple experiments on dyadic, interpersonal coordination have shown that the tendency to spontaneously synchronize one’s movements with those of a teammate can interfere with human performance in team tasks.

The same phenomena is at work in more complex, cognitive settings, such as intelligence analysis and planning and team command-and-control. In those situations, team members self-organize “cognitive-behavioral” (e.g., communication) patterns without being consciously aware of it.

Gorman and his colleagues believe that the key to understanding how emergent coordinative relations shape team performance is in identifying the mathematical and statistical dynamics that occur as team members interact. He states that in the next decade, this approach will be characterized by mathematical models that go beyond the individual mental state or top-down knowledge with the goal of understanding how general coordinative mechanisms structure team performance.

Communication Dynamics for Team Assessment – Nancy J. Cooke, Arizona State University

Teams can be viewed as complex dynamic systems made up of interacting components that are systems themselves, so assessment of team learning can benefit from an understanding of the dynamics of team interactions. Communication is the most straightforward interaction data source to collect, and can be collected unotrusively and in near real time.

Cooke advises that research is needed into the association between communication dynamics and effective team process and performance in various contexts. The discovery of communication dynamics signatures linked to specific process and performance across contexts would allow for interventions that are impactful and timely.

Synchronization of Autonomic Arousal in Dyads and Teams – Stephen J. Guastello, Marquette University

Physiological synchronization of autonomic arousal between people is thought to be an important component of work team coordination and other interpersonal dynamics. In turn, the dynamics of the group contribute to workload and fatigue effects at the group level in addition to the individually-defined work assignments.

The minimum requirements for two living or non-living entities to synchronize are two oscillators, a feedback loop between them, and a control parameter that speeds up the process. When speed reaches a critical level, a phase shift occurs such that the system goes into phase-lock.

Analytic challenges include:

• determining the statistical time series models that capture the dynamics of the teams’ interaction patterns
• finding the optimal lag length between observations for structuring the models, and
• connecting synchronization parameters to performance variables, subjective ratings of workload and other group dynamics, and individual differences of the contributing group members.

In the latest study involving teams of four people playing a game against a single opponent, the participants appear to be in phase lock.

Guastello recommends that future research dissect the experimental tasks to identify the primary oscillators, feedback loops, possible control parameters, and conditions that induce higher levels of synchronization and involve different types of internal group coordination. Analyses of biometrics should focus on continuous streams of data and the analysis of dynamics therein, and the dynamics can then be related to qualitative variables of interest, thereby building a comprehensive biopsychosocial model and generalizable theory.

Collaborative Assessments and Data Analysis – Alina A. von Davier, Educational Testing Service

Collaborative assessments in game-like environments integrate many different assessment approaches and tools. Collaboration is one of the skills identified as the “21st-century skills” and it receives attention among stakeholders in both higher education and the workplace. The OECD included a test of collaborative problem solving skills in its PISA 2015 survey of critical skills, and collaborative assessment is also being promoted by a global initiative called Assessment and Teaching of 21st Century Skills.

Collaborative interactions in computerized educational environments produce data of extraordinarily high dimensionality. Extracting key features from the noise in such data is crucial not only to make analysis computationally tractable, but also to extract relevant features of student performance from the noise surrounding them.

von Davier advises that one way to attempt to find patterns among the different types of data is to make use of data mining techniques. The purpose of data mining techniques is to reduce the dimensionality of the dataset something more manageable by extracting implicit, interesting, and interpretable patterns in order to allow research questions to be addressed that would not otherwise be feasible.

References:

Also published on Medium.