Resumen de Interactive sonification of images in serious games as an education aid for visually impaired children
Andrzej Radecki, Michał Bujacz, Piotr Skulimowski, Paweł Strumiłło
The paper presents an application for interactive sonification of images intended for use on mobile devices in education of blind children at elementary school level. The paper proposes novel sonification algorithms for converting colour and grayscale images into sound. The blind user can interactively explore image content through multi-touch gestures and select image sub-regions for sonification with real-time control of the synthesized sound parameters such as amplitude, frequency and timbre. Additionally, images may contain text fields read by text-to-speech synthesizer. The usability of one of the proposed sonification schemes is tested by collecting data on the tracking accuracy and recognition speed of basic shapes, such as lines, curves, as well as figures and simple functions. In order to facilitate the learning process for blind children, a number of games were proposed that use the devised sonification schemes. The first game??Hear the Invisible??is intended for two players?one child draws a shape and the task of the other one is to guess the displayed shape by means of the available sonification methods. The second proposed game is ?Follow the Rabbit? and is intended for a single player who tracks a colourful ?Rabbit? that runs along a path representing a given geometric shape. The obtained results show that the proposed sonification methods allow to reach previously unattainable levels of flexibility in exploration of shapes and colours. The main application of the described interactive tool is to teach geometry and mathematics in schools for blind children. The developed games are meant to enhance the learning process and motivate the children. Practitioner Notes What is known about the topic Sonification is a generic term for data-based non-speech sound generation. Interactive sonification is a relatively fresh area of study, where the listener heavily influences the process of sound synthesis. Earlier studies on sonification and educational computer games have shown that it is possible to demonstrate shapes such as plots and geometry using only non-verbal sounds. What the paper adds The paper proposes an interactive sonification algorithm that can be used on greyscale or colour images using additive synthesis and the HSV colour representation. The presented software also allows to record and analyse user interaction with the images, making it a good research tool for current and future sonification algorithms. The paper presents a pilot study with blind children using two games that improve the learning process of the proposed algorithms. Implications for practice and/or policy Once the software is openly released it can be easily installed on any Android OS device to allow blind users to analyse images by touch. The proposed games train the users the basics of the sonification algorithm in a competitive context. The software can be especially useful in education of blind children, eg, in teaching geometry, math and in training visual and spatial imagination.
