Embedded Real-time Musical Pattern Detection for Smart Musical Instruments

This thesis explores the development of embedded real-time musical pattern detection (RTPD) systems for smart musical instruments (SMI), a class of musical instruments which integrate sensors, embedded intelligence, and wireless connectivity. We focused on enabling SMIs to detect pre-defined musical patterns in real-time, which can then be used as triggers to control external devices such as stage lights, smoke machines, haptic devices and mixed reality headsets. This work is situated within the broader context of the Internet of Musical Things (IoMusT), a networked ecosystem of interconnected musical devices and systems. Most existing pattern detection methods are designed for offline contexts - i.e., to detect all occurrences of a pattern in an entire composition. Before the beginning of this thesis work, real-time implementations of such systems did not exist. This thesis fills this research gap by introducing a real-time algorithm optimized for embedded systems with constrained computational resources. This thesis begins by discussing the challenges of RTPD, especially in live performance where low latency and minimal memory consumption are paramount. Then three datasets tailored for RTPD research are introduced. The Dataset of Musical Patterns, the Dataset of Drum Patterns, and the Dataset of Polyphonic Patterns were developed with contributions from professional musicians across diverse backgrounds. They capture expressive and artistic variations a performer might do during a live performance, and they address gaps in existing resources by focusing on intentional variations in pattern repetitions. These datasets allowed for the development of Hot Licks, an open-source RTPD algorithm that can be integrated into SMIs. Hot Licks was deployed on different SMIs prototypes, and it demonstrated high accuracy in detection as well as a minimal latency. Hot Licks was recognized as a finalist in the prestigious MIDI awards 2023. User studies involving professional musicians evaluated the SMIs prototypes' usability and effectiveness. Results indicated positive reception, highlighting the potential of RTPD-enabled SMIs to enhance creativity and audience engagement. Furthermore, multisensory performances were conducted using these instruments, showcasing their ability to synchronize music with visual and tactile stimuli in real-time. The findings underscore the potential of RTPD in live music contexts, paving the way for new artistic possibilities and interactive experiences. This thesis concludes by reflecting on its contributions to the fields of music technology and IoMusT while outlining future research directions, including advancements in polyphonic detection algorithms, integration with virtual reality environments, and broader applications in education and therapy.