A Human Interface for Mixed Reality

Recently we went to Interaction 17’ and presented our speculative concept on free form gesture inputs. If you haven’t seen the talk yet, here is the link. It was impossible to unpack the entire concept in 8 minutes. So, we're sharing a bit of our process and thinking behind the talk here.

Mixed Reality has recently become a popular buzzword in design, but underlying interaction paradigms still need refinement. We are familiar with using trackpads and VIVE controllers. However, when we use Hololens, the lack of affordance makes free form gestures still hard to approach. In this work, we focus on free-form gesture patterns and how we can utilize our own body when interacting in the Mixed Reality context.

Finger pointing at purple geometric form — The lack of affordances makes free form gestures difficult to approach.

Body interface and proprioception.

We have an innate understanding of our limbs, joints, and muscles without needing to rely on our five senses. This is called proprioception. We hypothesize that we should rely on proprioception for mixed reality input because it creates affordances by using the body without introducing external hardware interfaces. We see great potential for a more natural and flexible body interface in this approach.

Hand on top of layered geometric forms — Use proprioception as affordance for free form gestures.

Seamful vs. Seamless.

Traditionally, we appreciate seamless design: design that is invisible and is effortless to use. However, to approach free-form gestures, we think the affordance has to be obvious for users to pick up. We need to better define the interaction space and range when designing gesture inputs. So here we argue for Seamful Gestures.

There is no better way to think and learn than making things. So we made several prototypes, tested them out with colleagues, and quickly improved the design based on feedback.

Graph showing fidelity of controls increased from a single variable to a set of 3D variables — Prototypes explored: the fidelity of controls increased from a single variable to a set of 3D variables.

Two people with a computer testing prototypes to learn, fix, and increase fidelity — Testing prototypes to learn, fix and increase fidelity.

1. Seam

We define Seam as a clear affordance for an interaction boundary. For example, the distance in between our thumb and index fingers is a seamful proprioceptive interaction, since we intrinsically know the maximum and minimum of the span. This type of interaction is perfect for controlling transparency or other controls that have properties with upper and lower limits.

Hand showing the distance in between thumb and index finger — The distance in between thumb and index fingers is a seamful proprioceptive gesture with a clear affordance and an interaction boundary.

Hand performing pinch gesture — Assign the gesture to a continuous value with upper and lower limits.

2. Hierarchy

Our finger muscles are trained differently on a daily basis: thumb and index are more frequently used than pinky. As a consequence, our fingers have a hierarchy in terms of how powerful and sensitive they are. For example, the affordance between thumb and index feels easier to use and has a higher fidelity than the affordance between thumb and pinky. There is an opportunity to utilize this existing hierarchy of fidelity. For example: the click between thumb and middle/ring fingers can be used for step controls of the object’s transparency.

Hand with line showing hierarchy across fingers related to power and sensitivity — Hierarchy across fingers in terms of how powerful and sensitive they are.

Hand making a click gesture — Apply lower fidelity step controls to less powerful and sensitive fingers.

We have also thought about assigning less frequently used tasks or even mission critical to the affordance between thumb and pinky, because it is not a common gesture and it takes more effort to activate. In addition, we had a prototype using the less dominant hand for binary controls to fulfill complementary functions such as activation/deactivation, while the dominant hand is assigned to more precise scrolling selection. The principle behind these two concepts is the same: our body interface has a hierarchy. Think about the level of fidelity when designing your input system.

3. Directionality

We had a prototype utilizing the orientation of the wrist to control which axis(x, y, or z) to manipulate when scaling an object in 3D. Another attempt was using the vector reading between two fingers to select the axis. Both didn’t turn out effective. It feels unnatural to force the fingers into a precise x, y, or z direction.

Hand showing vector reading between thumb and index fingers to control directionality — Use the vector reading between thumb and index fingers to control directionality feels unnatural.

We feel proprioceptive gesture is powerful for fine tuning interactions, but not for directionality in space. For directionality controls, we still prefer the solution below: using the tilting of the hand to fly in Google map.