Snapshot

Problem Context

Learning languages is often tough because it is broken into parts- you learn how to write and how to speak separately. With mobile based solutions running out of engagement, and classroom spaces inhabited by hesitations of being incorrect, we wanted to design an intervention that allows the learner to immerse themselves in a language and get opportunities to practice it, in a manner that will engross them in the process.

Intervention Designed

Using an immersive VR system, we make the process of learning a language playful, interactive and experiential. This system provides independence to the learners to learn and practice languages. It is a safe environment where learners can afford autonomy and can shed fears of being wrong. The learning system integrates reading, listening and speaking activities. We provide for learning through vocabulary building and pronunciation practice. We leverage interactivity, spatio-visual learning and voice user interfaces to help people with the process of learning of a language and make it engaging.

[4 mins] This video below provides an overview of the intervention designed:

The key interaction of the system is the ability to create an object out of nowhere, like a spell.

Technology

We use HTC Vive and the Unity engine to revamp the IBM Watson Speech Sandbox and provide language learners a highly interactive platform to learn a language. The setup consists of a Headset and mic mounted on the user, a physical space marked by HTC Vive lighthouses and, controllers with which the users interact with the virtual world. We augment immersion through the use of background sounds and music via earphones.

Learning in VR

We use a creation-based approach for learning, where objects appear when you invoke their names through speech in the language you want to learn. The creation of the object at successful pronunciation is an intrinsic feedback mechanism. Learning is hence supported with active feedback and delight on seeing objects appear at commands. We use a discovery oriented approach that encourages speech practice and voacbulary acquisition via repetition and knowledge transfer. The learning is further scaffolded through spatio-visual learning.

Vocabulary Acquistion

The learner acquires new vocabulary through a list of objects and their translations. For each word, we provide the pronunciations, which can be heard to learn how that word is spoken in the new language. This facilitates retention and vocabulary acquisition. This method creates strong associations between a word, its translation, its pronunciation and the object created. This is boosted by the immersive context of a VR system.

Pronunciations

The system provides for a natural voice user interface designed to faciliate learning and practice via repetition. The learner can simply speak up what object they want to create in the new language. If the pronunciation is acceptable, the object is created. If the pronunciation is not acceptable, the absence of creation notifies the learner that they need to practice more.

Key Concepts

Active Feedback-based Learning

We use a creation-based approach for learning, where objects appear when you invoke their names through speech in the language you want to learn. The creation of the object at successful pronunciation is an intrinsic feedback mechanism. Learning is hence supported with active feedback and delight on seeing objects appear at commands. This interaction also creates strong associations between the object, the word and its pronunciation. We use a discovery oriented approach that encourages speech practice and vocabulary acquisition via repetition and knowledge transfer. The learning is further scaffolded through spatio-visual learning.

Immersive Experience

The nature of the activity and the surrealism of VR provides for immersion into the system. The playful visual setting, physical interactions- walking around in physically mapped virtual space and leaning forward to pick objects, and the discovery oriented learning adds towards the immersion. Thus, the learner spends more time in the system, getting ore exposure to the language.

Interactions in VR

We spent a lot of time mapping meaningful physical interactions to virtual interactions to levarge users' mental models. Thus, the user uses the grip mechanisms of the controller to grab and throw the objects, can move around in the space and can speak freely without pressing any buttons. Physics, shadows, evelavtions was simulated in the virtual world, but at the same time, we provided for surrealism as well, through mechanisms like strength and teleportation. The controllers had pointer rays which acted as the visual guide for sight, reach and intent.

Making the Learner Independent

The interactions and space were designed to provide a learner with a safe space, where they could train themselves without hesitations or fear of being wrong. They become independent for the parts of language learning faciliated by the system, and need not rely on external agents. By encouraging them to read, listen, absorb and then speak out, we simulate the activities necessary to acquire a language.

Process

Understanding the problem

Learning languages is often tough because it comprises of many parts- scripts, vocabulary, pronunciation, listening, speaking, forming sentences, communicating and even culture! We constructed a mind map to chart out these various aspects of language.

Following this, we narrowed down our to providing for acquisition of vocabulary and pronunciation of foreign words in a safe, hesitation free environment.

Gathering Requirements

Recognising that the the standard design process does not fit this type of a problem domain owing to its reliance on primary research, we instead gathered requirements through extensive secondary research: academic literature that revolved around pedagogical startegies, challenges and technology driven solutions. Thus, we adopted the learnings from this corpora of knowledge of researchers and practioners of language instruction. I do not discuss these here.

Further, with an ambition to provide for an immersive learning experience, we also looked into literature on how this was already adopted in research and industry. This helped us speed up our process of interaction design and content. We also looked at many VR applications to experience the affordances of immersive, interactive VR enabled by HTC Vive.

Key Requirements of the solution

An virtual environment to hold the learning experience.
Providing the learners vocabulary words and their pronunciation as comprehensible learning inputs.
Understand the user through multiple forms of interactions- spatial, controller driven, line of sight tracking, movement, triggers, speech and audio.
The system should be able to give feedback to the user on their input, both language based and interaction based
There needs to be a separation of tasks, scenarios, and intensities based on the current skill level of the user.
Must have visual to word mapping for Vocabulary Training so the user can formulate small sentences; pronunciation training so the user is adept to communicate verbally in the target language.

Capturing Context

We then use the PACT framework to define the constraints of our requirements.

The PACT Framework to constrain the problem.

Key Interactions

Impact

This was the first interaction design project around VR in the college! We had a poster presentation where >30 students and >6 faculty showed up to test the device hands on!

This project inspired the junior design batch to go ahead and experiment with Mixed Reality solutions. I am currently mentoring 67 of these students across 17 teams, out of which 14 are designing MR interactive solutions!

We also evaluated the project with 6 learners which I always discuss in person. We also submitted the project to CHI Late Breaking Works 2020. While the work did not make it to the conference, we got some great feedback to incorporate. This project is the inspiration behind my BTech Thesis.