B1: Gestural interaction - A memory of my childhood

Gestural interactions are becoming more and more common in our daily lives with technology and smart devices. From automatic soap dispensers, the pinch zoom on our smartphones, to automatic soap dispensers, to VR climbing games. While the pinch zoom gestures feel surprisingly natural to me by now, climbing in VR, “holding” on to rocks that aren’t there and my feet safely grounded to the floor feels rather confusing.

But what exactly are gestural interactions? They are interactions with computing devices using body movement or gestures, instead the conventional tools such as physical buttons, keys, or a mouse.

In this blog post, I will share with you my experience with gestural interfaces using one specific example, how the user experience of it is/was, and how the product worked technically.

My experience

While reading the article Natural user interfaces are not natural by Norman, D. A. I immediately had to think about one of the first experiences I had with gestural interactions as a child. It was when I was 2004 when I was around 9 years old when I received an EyeToy camera for my Playstation 2 with the accompanying game EyeToy: Play. It quickly became one of my favorite games and further editions of the game, EyeToy: Play 2 and 3 followed. While I was super enthusiastic about these games, my below neighbors weren't.

How “natural” are these interactions?

With the rise of digital systems and devices the question of whether the interaction with their interfaces is “natural” arises. What does “natural” even mean? According to the Cambridge dictionary natural means “as found in nature and not involving anything made or done by people”. This already contradicts itself in this context, because interfaces, devices, and the interactions with them are all made up of people. Therefore I’d rather ask the question “how natural do these interactions feel?”

Focusing on my example above of EyeToy, I’d say the overall interactions feel quite “natural”. The main reason for that is that you can see yourself on the screen moving your hand to the object you want to navigate to, additionally, it’s showing immediate feedback by starting to fill up the circle around the word. Besides that, it is also easy to learn. However, I wouldn’t say waving at something for a few seconds feels natural. It requires quite some effort because most of the navigation is above the head so your arm might get tired of waving. An improvement for navigating different games might be to precisely track the position of the hand and make the triggering area around the selection bigger. If the hand is in that field for three seconds the user makes the selection.

Besides the navigation between the games, I would also like to talk about the interactions between the games. I’d say that there are some games that feel more “natural” when playing, such as the window cleaning game than other games, such as running and jumping through a parkour. I think the reason for this is that window cleaning is something you usually also do mainly with your hands, but in the game, the window is cleaned way faster than in real life. This also makes it more fun. On the other hand, the parkour game feels less realistic than running a parkour outside of the game. Since the game focuses on the upper body the movement of your legs doesn’t get taken into account and you stay in one spot while playing the game, while your character on the screen actually runs and moves forward. It doesn’t fit the mental modal and muscle memory your body has of doing parkour in the physical world. An improvement for this might be to make use of the whole body movement and get the whole body into the frame of the camera.

How does EyeToy:Play track your gestures?

The EyeToy color webcam with integrated microphone is the hardware, which represents kind of the eye and ears of the Playstation. While the software of the game EyeToy:Play uses gesture recognition, a subdiscipline of computer vision. It can be seen as the brain of the Playstation. Gesture recognition interprets gestures and movements people make through mathematical algorithms. This way the game knows how the player moves and which actions should be triggered.

Conclusion

Considering the time when EyeToy was released, it was pretty novel in creating games that don’t require an extra remote for playing. It is definitely challenging to make interactions only through the body movement of the user - no button that can be pressed for selection and no physical feedback for the user either, but heavily depending on visual feedback. While writing this blog post I was reflecting about that EyeToy is a game, but how would I feel about gestural interactions in daily life? In the game, I can accept if something goes wrong, but in a busy daily life, traffic, cooking,... if something goes wrong it could have serious consequences.

My questions

What effects might gestural interactions have on human well-being?

How might gestural interactions influence our society? (if everything goes automatic, like an automatic door, will we become ignorant towards other people because we don’t need to watch out for anything anymore?)

How inclusive and accessible are gestural interactions? How could they be made inclusive?

Sources:
https://dictionary.cambridge.org/dictionary/english/natural
https://www.igi-global.com/dictionary/improved-interaction-for-mid-air-projection-screen-technology/12176
https://en.wikipedia.org/wiki/EyeToy
https://en.wikipedia.org/wiki/Gesture_recognition
https://www.gamedeveloper.com/pc/in-depth-eye-to-eye---the-history-of-eyetoy