RETHINKING CORE ARCHITECTURAL BLOCKS FOR ADVANCING DEEP LEARNING PERFORMANCE

RECENT PROGRESS IN DEEP LEARNING HAS LARGELY BEEN ACHIEVED BY SCALING MODEL CAPACITY AND TRAINING DATA, LEVERAGING EVER-INCREASING COMPUTATIONAL RESOURCES TO OBTAIN INCREMENTAL PERFORMANCE GAINS. IN PARALLEL WITH THIS TREND, A COMPLEMENTARY DIRECTION IS OPENED BY STRUCTURALLY IMPROVING THE ATOMIC COMPONENTS THAT CONSTITUTE DEEP ARCHITECTURES, SO THAT EACH LOCAL ENHANCEMENT CAN PROPAGATE THROUGH THE ENTIRE NETWORK AND YIELD GLOBAL BENEFITS IN EFFICIENCY, STABILITY, AND ACCURACY. IN THIS THESIS, ATTENTION IS DIRECTED TO TWO FUNDAMENTAL CLASSES OF ARCHITECTURAL BLOCKS: POOLING OPERATORS AND CONVOLUTIONAL MODULES. FIRST, A FAMILY OF TRAINABLE POOLING FUNCTIONS INSPIRED BY FUZZY AGGREGATION THEORY IS INTRODUCED, IN WHICH FEATURE AGGREGATION IS ADAPTIVELY MODULATED RATHER THAN FIXED. WHEN THESE OPERATORS ARE EMBEDDED IN AUTOENCODER-STYLE ARCHITECTURES FOR BOTH IMAGES AND TIME SERIES, MORE INFORMATIVE LATENT REPRESENTATIONS ARE OBTAINED AND RECONSTRUCTION ERROR IS REDUCED WITH RESPECT TO TRADITIONAL AND CONTEMPORARY POOLING STRATEGIES, WITHOUT A SUBSTANTIAL INCREASE IN PARAMETER COUNT OR COMPUTATIONAL COST. SECOND, THE ROLE OF CONVOLUTIONAL LAYERS IN AUDIO SOURCE SEPARATION IS RECONSIDERED BY DESIGNING AN ENCODER–DECODER ARCHITECTURE BASED ON MULTI-PATH RESIDUAL PROCESSING. PARALLEL CONVOLUTIONAL PATHWAYS OPERATING AT MULTIPLE TEMPORAL SCALES ARE EMPLOYED, WHOSE OUTPUTS ARE COMBINED THROUGH LEARNABLE FUSION MECHANISMS THAT EMPHASIZE CONTEXT-RELEVANT INFORMATION. THIS DESIGN LEADS TO MORE STABLE TRAINING DYNAMICS AND IMPROVED SEPARATION QUALITY OF HUMAN VOICES FROM COMPLEX ACOUSTIC BACKGROUNDS WHEN COMPARED UNDER CONTROLLED CONDITIONS TO BOTH CONVOLUTIONAL AND TRANSFORMER-BASED BASELINES. OVERALL, THE THESIS ARGUES THAT SYSTEMATIC INNOVATION AT THE LEVEL OF THESE MICROSCOPIC ARCHITECTURAL PRIMITIVES PROVIDES A POWERFUL AND SCALABLE ROUTE FOR ADVANCING DEEP LEARNING PERFORMANCE, COMPLEMENTING LARGE-SCALE MODEL AND DATASET GROWTH AND ENABLING MORE EFFICIENT EXPLOITATION OF AVAILABLE COMPUTATIONAL BUDGETS ACROSS DIVERSE DOMAINS.