Novel interfaces for motor assessment and learning support of violin playing and cursive handwriting

Playing violin and handwriting are two complex motor tasks that require many hours of exercise and practice. The teacher plays a crucial role in the progress of the learning process. Indeed, the teacher’s instructions allow the student to be conscious of his mistakes and to correct them. Nowadays, in the music scenario the teaching methods include exercises at different level of complexity guided by verbal instructions, visual demonstration, and physical guidance (i.e., when teachers drive students’ movements with their hands). Unfortunately, this teacher-student interaction requires the constant supervision of a teacher who should be physically present at the time of the exercise and can be prone to an ambiguous interpretation of the teacher’ cues and/or to delayed feedback (i.e., feedback provided at the end of the exercise). Learning handwriting requires children to properly master the tool (i.e., a pen or a pencil), acquire basic shapes necessary for letter forming, and automatize this process. To develop these skills, children are exposed to several exercises with different level of complexity and different styles: upper case, script, and cursive. The latter is the most challenging style, usually introduced at the end of the second year of the primary school in Italy with several difficulties. Indeed, recent data collected by the Italian Ministry of Education highlights an increasing of the numbers of children assessed as “dysgraphic” or “poor writers”. This evidence points out the need of screening tools available also to teachers for a timely evaluation of possible problems which can impact the learning process. Unfortunately, nowadays the handwriting evaluation is based on pen and paper tests (i.e., human-based coding - HBC) that are time-consuming and focus only on the handwriting product (i.e., child’s handwriting readability) providing a poor assessment of the handwriting process (i.e., fine- and gross- motor skills implemented by the child during the task). The ambition of this PhD thesis is the design and development of new methods and interfaces for the assessment of bowing gesture and cursive handwriting, providing novel tools for supporting both students and teachers. Technologies and feedback exploited to monitor and support playing violin were reviewed to identify the most appropriate ones for unstructured environments, like the home of a violin student. Among these, magneto-inertial measurement unit (MIMU) and optical sensors were investigated to monitor the first motor skill that a violin student must learn: i.e., mastery the bow-violin orientation and the bow section in contact with the string. A feasibility study on 9 violin beginners confirmed the choice of these technologies. The results supported the design and development of a modular platform that includes two MIMUs, two optical sensors and an ad-hoc software to monitor the bowing gesture and to provide real-time visual feedback to the student. A study on 24 violin beginners confirmed the positive effect of the visual feedback in reducing the bow-violin orientation errors performed by the subjects. Moreover, a questionnaire administered to the subjects exposed to the visual feedback, showed as the feedback has been assessed useful to improve the bowing movement and clear in the presentation of data. However, even if useful for controlling the movement, the subjects perceived as more challenging the execution of the exercise with feedback. The latter outcome is in accordance with the results already available in literature. Preliminary results about the use of optical sensors for estimating the bow section encouraged the use of this type of technology. Data from optical sensors were used to train a fine-tree classifier and return which section of the bow has been used to play the violin. According to violin teachers the bow has been divided into three sections: a proximal one, near the frog; a central one; and a distal one, near the tip. However, since the same expert musician was used for the training and testing of the classifier, a collection of data considering different expert musicians, bows, and different bow-hair extension is needed to assess the actual accuracy of the method proposed. The technology developed can be embedded into any bow, but it has been tested only on a 4/4 bow for adults. Overall, the application of this technology to bow moved its center of mass 8 mm towards the frog. This feature has a major impact in advanced music techniques like spiccato that are not taught to beginner in the first period of the learning process. However, further analysis is needed to investigate if the displacement of the bow center of mass could impact the bow control in beginners. The static tests on the hardware demonstrated the ability of the platform to provide bowing angles with error below 3°. This result is in accordance with the outcome of the feasibility study. The main grapho-motor parameters (GMPs) useful for cursive handwriting evaluation have been identified from a comparison among the gold standard pen and paper tests currently used for handwriting assessment. In parallel, an analysis of the technologies used in literature for handwriting assessment allowed to identify the functional and technical requirements of a custom platform enabling the measure of the GMPs of interest. Results of this preliminary research allowed developing a new tool called Grapho-motor Handwriting Evaluation and Exercise – GHEE composed of both a hardware and software module. The former is composed of a Wacom Cintiq 16 interactive display and its stylus Pro Pen 2. The software components include: i) the Eye and Pen Software for the stimuli presentation and data acquisition, and ii) a custom App developed in Matlab for the handwriting assessment. A feasibility study on 6 adults volunteers allowed to test the reliability of GHEE in comparison with a human coder. Results showed the ability of GHEE to provide a completely automatic assessment (machine-based coding – MBC) of GMPs relying quantitative aspects of handwriting and to reduce coding time. However, the feasibility study suggested the introduction of a human-machine interaction approach for the assessment of qualitative aspects of handwriting. Indeed, this approach will allow to reduce the workload on the human coder without losing important information. A study on 10 children allowed to test GHEE in an operative environment. The tested platform provided a completely automatic assessment of some quantitative GMPs (i.e., Fluctuations, Space, Margin alignment, Dimension and Number of Inversion of Velocities) and proposed a human-machine interaction approach for some qualitative GMPs (i.e., Connections and Directions). The comparison between GHEE MBC and HBC showed as the platform is able to compute 6 of the 7 considered quantitative indices with an accuracy equivalent to the standardized tests. Results on the human-machine based approach show as the new screen-based technologies could improve handwriting assessment providing teachers with useful information on qualitative aspects of handwriting. Moreover, these new technologies allow to assess the handwriting process providing an analysis of the kinematic. However, the use of these new screen-based technologies points out the need to evaluate the influence of these new tools on the handwriting performance in order to identify the eventual necessity of new normative data (not available for screen-based technologies). A study on 40 children collected using the developed platform is investigating these aspects and preliminary results are presented in the work. Finally, the development of a custom app integrating both the stimuli presentation, GMPs extraction and assessment is presented. Such tool will foster the diffusion of the GHEE all among different schools in different learning scenarios.