Перевод статьи (МГОУ)

Research Article
Embodiment,Multimodality, and Composition: Convergent Themes across HCI and Education for Mixed-Reality Learning Environments
David Birchfield,1 Harvey Thornburg,1, 2 M. ColleenMegowan-Romanowicz,1, 3 Sarah Hatton,1, 4 BrandonMechtley,1, 5 Igor Dolgov,1, 6 andWinslow Burleson1, 5

1Arts, Media and Engineering, Arizona State University, Tempe, AZ 85281, USA
2Department of Electrical Engineering, Arizona State University, Tempe, AZ 85281, USA
3 School of Educational Innovation and Teacher Preparation, Arizona State University, Tempe,
AZ 85281, USA
4 School of Art, Arizona State University, Tempe, AZ 85281, USA
5 School of Computing and Informatics, Arizona State University, Tempe, AZ 85281, USA
6Department of Psychology, Arizona State University, Tempe, AZ 85281, USA
Correspondence should be addressed to David Birchfield, dbirchfield@asu.edu
Received 6 October 2007; Revised 27 July 2008; Accepted 14 October 2008
Recommended by Adrian Cheok
We present concurrent theoretical work fr om HCI  and Education that reveals a convergence of trends focused on the importance of three themes: embodiment, multimodality, and composition. We argue that there is great potential for truly transformative work that aligns HCI and Education research, and posit that there is an important opportunity to advance this effort through the full integration of the three themes into a theoretical and technological framework for learning. We present our own work in this regard, introducing the Situated Multimedia Arts Learning Lab (SMALLab). SMALLab is a mixed-reality environment where students collaborate and interact with sonic and visual media through full-body, 3D movements in an open physical space. SMALLab emphasizes human-to-human interaction within a multimodal, computational context. We present a recent case study that documents the development of a new SMALLab learning scenario, a collaborative student participation framework, a student-centered curriculum, and a three-day teaching experiment for seventy-two earth science students. Participating students demonstrated significant learning gains as a result of the treatment. We conclude that our theoretical and technological framework can be broadly applied in the realization of mixed reality, student-centered learning environments.
Copyright © 2008 David Birchfield et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
1. Introduction
Emerging research fr om Human Computer ........
2. PriorWork
Recent research spanning Education and HCI has yielded three themes that inform our work across learning and play: embodiment, multimodality, and composition. Here, we define the scope of these terms in our research and discuss their theoretical basis before presenting examples of prior related applications.
2.1. Embodiment
2.1.1. Learning Sciences
By embodiment we mean that SMALLab interactions engage students both in mind and in body, encouraging them to physically explore concepts and systems by moving within and acting upon an environment.
A growing body of evidence supports the theory that cognition is “embodied”-grounded in the sensorimotor system [2–5]. This research reveals that the way we think is a function of our body, its physical and temporal location, and our interactions with the world around us. In particular, the metaphors that shape our thinking arise fr om the body’s experiences in our world and are hence embodied [6].
A recent study of the development of reading comprehension in young children suggests that when children explicitly “index” or map words to the objects or activities that represent them, either physically or imaginatively, their comprehension improves dramatically [7]. This aligns well with the notion, advanced by Fauconnier and Turner [4], that words can be thought of as form-meaning pairs. For example, when a reader encounters the lexical form, “train” in a sentence, he can readily supply the sound form (tr¯an). If he then maps it to the image of a train (a locomotive pulling cars situated on a track), we have a form-meaning pair that activates the student’s mental model of trains, which he can then use to help him understand and interpret the sentence in which the word “train” appears [6].
SMALLab is a learning environment that supports and encourages students in this meaning-making activity by enabling them to make explicit connections between sounds, images, and movement. Abstract concepts can be represented, shared, and collaboratively experienced via physical interaction within a mixed-reality space.
2.1.2. HCI
Many emerging developments in HCI also emphasize the connections between physical activity and cognition [8–14], and the intimately embedded relationship between people and other entities and objects in the physical world [15–17]. The embodied cognition perspective [10, 14] argues based on strong empirical evidence fr om psychology and neurobiology [7, 18] that perception, cognition, and action, rather than being separate and sequential stages in human interaction with the physical world, in fact occur simultaneously and are closely intertwined. Dourish [8, 9] in particular emphasizes the importance of context in embodied interaction, which emerges fr om the interaction rather than being fixed by the system. As such, traditional HCI frameworks such as desktop computing (i.e., mouse/keyboard/screen) environments, which facilitate embodied interaction in a limited sense or not at all, risk binding the user to the system context, restricting many of his/her capacities for creative expression and free thought which have proven so essential in effective learning contexts. Fr om cognitive, ecological, and design psychology, Shepard [17], Gibson [15], Norman [16], and Galperin [19] further emphasize the importance of the embedded relationship between people and things, and the role that manipulating physical objects has in cognition. Papert, Resnick, and Harel (see [20–23]) extend these approaches by explicitly stating their importance in educational settings. Design-based learning methodologies such as Star Logo, Lego Mindstorms, and Scratch [21, 24, 25] emphasize physical-digital simulation and thinking. These have proven quite popular and effective in fostering and orienting students’ innate creativity toward specific learning goals.
In order for these tools to extend further into the physical world and to make use of the important connections provided by embodiment, they must include physical elements that afford embodied interactions. Ishii has championed the field of tangible media [26] and coined the term tangible user interfaces (TUIs versus GUI: graphical user interfaces). His Tangible Media group has developed an extensive array of applications that pertain to enhancing not only productivity (e.g., Urban Simulation, SandScape) but also artistic expression and playful engagement in the context of learning (e.g., I/O Brush, Topobo, and Curlybot) [27].
Some prior examples of HCI systems that facilitate elements of embodiment and interaction with immersive environments include the Cave Automated Visualization Environment (CAVE) [28]. CAVEs typically present an immersive environment through the use of 3D glasses or some other head-mounted display (HMD) that enables a user to engage through a remote control joystick. A related environment, described as a step toward the holodeck, was developed by Johnson at USC to teach topics ranging fr om submarine operation to Arabic language training [29]. In terms of extending physical activity through nontraditional interfaces and applying them to collaboration and social engagement, the Nintendo Wii’s recent impact on entertainment is the most pronounced. The Wii amply demonstrates the power of the body as a computing interface. Some learning environments that have made strides in this area include Musical Play Pen, KidsRoom, and RoBallet [30–32]. These interfaces demonstrate that movement-based HCI can greatly impact instructional design, play, and creativity.
2.1.3. Example
A particularly successful example of a learning environment that leverages embodiment in the context of instructional design is River City [33–36]. River City is a multiuser, online desktop virtual environment that enables middle school children to learn about disease transmission. The virtual world in River City embeds a river in various types of terrain which influence water runoff and other environmental factors that in turn influence the transmission of disease through water, air, and/or insect populations. The factors affecting disease transmission are complex and have many causes, paralleling conditions in the physical world. Student participants are virtually embodied in the world, enabling exploration through avatars that interact with each other, with facilitators’ avatars, and with the auditory and visual stimuli comprising the River City world. Participants can make complex decisions within this world by, for example, using virtual microscopes to examine water samples, and sharing and discussing their proposed solutions. In several pilot studies [33, 34], the level of motivation, the diversity and originality of participants’ solutions, and their overall content knowledge were found to increase with River City as opposed to a similar paper-based environment. Hence, the River City experience provides at least one successful example of how social embodiment through avatars in a multisensory world can result in learning gains.
However, a critical aspect of embodiment not addressed by River City is the bodily-kinesthetic sense of the participant. Physically, participants interact with River City using a mouse and keyboard, and view 2D projections of the 3D world on a screen. The screen physically separates users’ bodies from the environment, which implies that perception and bodily action are not as intimately connected as they are in the physical world, resulting in embodiment in a lesser sense [10]. In SMALLab, multiple participants interact with the system and with each other via expressive, fullbody movement. In SMALLab there is no physical barrier between the participant and the audiovisual environment they manipulate. It has long been hypothesized [37] that bodily kinesthetic modes of representation and expression are an important dimension of learning and severely underutilized in traditional education. Thus, it is plausible that an environment that affords full-body interactions in the physical world can result in even greater learning gains.
2.2.Multimodality
2.2.1. Learning Sciences
By multimodality we mean interactions and knowledge representations that encompass students’ full sensory and expressive capabilities including visual, sonic, haptic, and kinesthetic/proprioceptive.Multimodality includes both student activities in SMALLab and the knowledge representations it enables.
The research of Jackendoff in cognitive linguistics suggests that information that an individual assimilates is encoded either as spatial representations (images) or as conceptual structures (symbols, words or equations) [38]. Traditional didactic approaches to teaching strongly favor the transmission of conceptual structures, and there is evidence that many students struggle with the process of translating these into spatial representations [6]. By contrast, information gleaned fromthe SMALLab environment is both propositional and imagistic as described above.
Working in SMALLab, students create multimodal artifacts such as sound recordings, videos, and digital images. They interact with computation using innovative multimodal interfaces such as 3D physical movements, visual programming interfaces, and audio capture technologies. These interfaces encourage the use of multiple modes of representation, which facilitates learning in general, [39, 40] and are robust to individual differences in students’ optimal learning styles [37, 41], and can serve to motivate learning [1].
2.2.2. HCI
Many recent developments in HCI have emphasized the role of immersive, multisensory interaction through multimodal (auditory, visual, and tactile) interface design. This work can be applied in the design of new mixed-reality spaces. For example, in combining audio and video in perceptive spaces, Wren et al. [42] describe their work in the development of environments utilizing unencumbered sensing technologies in situated environments. The authors present a variety of applications of this technology that span data visualization, interactive performance, and gaming. These technologies suggest powerful opportunities for the design of learning scenarios, but they have not yet been applied for this purpose.
Related work in arts and technology has influenced our approach to the design of mediated learning scenarios. Our work draws from extensive research in the creation of interactive sound environments [43–45]. While much of this work is focused on applications in interactive computer music performance, the core innovations for interactive sound can be directly applied in our work with students. In addition, we are drawing from the 3D visualization community [46] in considering how to best apply visual design elements (e.g., color, lighting, spatial composition) to render content in SMALLab.
There are many examples wh ere HCI researchers are extending the multimodal tool set and applying it to novel technologically mediated experiences for learning and play. Ishii’s Music Bottles offer a multimodal experience through sound, physical interaction, and light color changes as different bottles are uncorked by the user to release sounds. The underlying sensor mechanism is a resonant RF coil that is modulated by an element in the cork. Edmonds has chronicled the significant contribution physiological sensors have made to the interactive computational media arts [47]. RoBallet uses laser beam-break sensors, such as those found in some elevators and garage doors, along with video and sonic feedback to engage students in interactive choreography and composition. Cavallo argues that this system would enable new forms of teaching not only music butmath and programming as well [32]. The work described in this paper builds upon this prior work and is similarly extending the tools and domains for multimodal HCI interfaces as they apply to learning and play.
2.2.3. Example—the MEDIATE Environment
............
2.3. Composition
2.3.1. Learning Sciences
..........
Student engagement in SMALLab experience is motivated both by the novelty of a learning environment that affords them some measure of control [57] and by the opportunity to work collaboratively to achieve a specific goal, wh ere the pathway they take to this goal is not predetermined by the teacher or the curriculum. Hence, SMALLab environment rewards originality and creativity with a unique digital-physical learning experience that affords new ways of exploring a problem space.
2.3.2. HCI
Compositional interfaces have a rich history in HCI, as evidenced by Papert and Minsky’s Turtle Logo which fosters creative exploration and play in the context of a functional, lisp-based programming environment [24]. More recent examples of HCI systems that incorporate compositional interfaces include novice level programming tools such as Star Logo, Scratch, and Lego Mindstorms. Resnick extends these approaches through the Playful Invention and Exploration (PIE) Museum Network and the Intel Computer Club Houses [58], thus providing communities with tools for creative composition in rich, informal sociocultural contexts. Essentially, these interfaces create a “programming culture” at community technology centers, classrooms, and museums. There has been extensive research on the development of programming languages for creative practitioners, including
graphical programming environments for musicians and multimedia artists such as Max/MSP/Jitter, Reaktor, and PD. This research has made significant contributions toward improving the impact and viability of programming tools as compositional interfaces.
Embedding physical interactions into objects for composition is a strategy for advancing embodied multimodal composition. Ryokai’s I/O Brush [59] is an example of a technology that encourages composition, learning, and play. This system enables capture from the physical world through a camera in the end of a paint brush that allows individuals to capture colors and textures from the physical world and compose with them in the digital world. It can even take video sequences such as a blinking eye that can then become part of the user’s digital painting. Composition is a profoundly empowering experience and one that many learning environments are also beginning to emphasize to a greater extent.
2.3.3. Example—Scratch
The Scratch programming environment [60] emphasizes the power of compositional paradigms for learning. Scratch enables students to create games, interactive stories, animations, music and art within a graphical programming environment. The interface extends the metaphor of LEGO bricks wh ere programming functions snap together in a manner that prohibits programming errors and thus avoids the steep learning curve that can be a barrier to many students in traditional programming environments. The authors frame the goal of Scratch as providing “tinkerability” for learners that will allow them to experiment and redesign their creations in a manner that is analogous to physical elements, albeit with greater combinatorial sophistication.
Scratch has been deployed in a number of educational settings [25, 61]. In addition to focused research efforts to evaluate its impact, a growing Scratch community website, wh ere authors can publish their work, provides mounting evidence that it is a powerful tool for fostering meaningful participation for a broad and diverse population.
Scratch incorporates multimodality through the integration of sound player modules within the primarily visual environment. However, it provides only a lim ited set of available tools for sound transformation (e.g., soundfile playback, speech synthesis) and as a consequence, authors are not able to achieve the multimodal sophistication that is possible within SMALLab. Similarly, Scratch addresses the theme of embodiment in the sense that authors and users can represent themselves as avatars within the digital realm. However, Scratch exists within the standard desktop computing paradigm and students cannot interact through other more physically embodied mechanisms.
2.4. Defining Play
With a focus on play in the context of games, Salen and Zimmerman [62] summarize amultitude of definitions. First they consider the diverse meanings and contexts of the very term “play.” They further articulate multiple scopes for the term, proposing a hierarchy comprised of three broad types. The most open sense is “being playful,” such as teasing or wordplay. Next is “ludic activity,” such as playing with a ball, but without the formal structure of a game. The most focused type is “game play,” wh ere players adhere to rigid rules that define a particular game space.
Play and game play in particular have been shown to be an important motivational tool [63], and as Salen and Zimmerman note, play can be transformative as, “it can overflow and overwhelm the more rigid structure in which it is taking place, generating emergent, unpredictable results.” Our work is informed by these broad conceptions of play that are applied to the implementation of game-like learning scenarios for K-12 content learning [62].
Jenkins offers an expansive definition of play as “the capacity to experiment with one’s surroundings as a form of problem-solving” [64]. Students engaged in this type of play exhibit the same transformative effects as described by Salen and Zimmerman. We apply this definition of play as collaborative problem solving in our work with students in formal learning contexts.
2.5. Toward a Theoretical and Technological Integration
As described above, there has been extensive theoretical and practice-based research across Education and HCI that is aimed at improving learning through the use of embodiment, multimodality, and compositional frameworks. We have described examples of prior projects, each of which strongly emphasizes one or two of these concepts. This prior work has yielded significant results that demonstrate the powerful impact of educational research that is aligned with emerging HCI practices. However, while there are some prior examples of interactive platforms that integrate these principles [65], there are few prior efforts to-date that do so while leveraging the powerful affordances of mixed reality for content learning. As such there is an important opportunity to improve upon prior work.
In addition, many technologically driven efforts are lim ited by the use of leading edge technologies that are prohibitively expensive and/or too fragile for most realworld learning situations. As a consequence, many promising initiatives do not make a broad impact on students and cannot be properly evaluated owing to a failure to address the practical constraints of today’s classrooms and informal learning contexts. Specifically, in order to see large-scale deployment on a two- to five-year horizon, learning environments must be inexpensive, mindful of typical site constraints (e.g., space, connectivity, infrastructure support), robust, and easily maintainable. It is essential to reach a balance between reliance upon leading-edge technologies and consideration of the real-world context in order to collect longitudinal data over a broad population of learners that will demonstrate the efficacy of these approaches.
Our own efforts are focused on advancing research at the intersection of HCI and Education. We next describe a new mixed-reality environment for learning, a series of formative pilot studies, and two recent in-school programs that illustrate the implementation and demonstrate the impact of our work.