Engaging with malware: coverage-guided fuzzing and dynamic introspection to analyze environment sensitive and obfuscated windows malware

This thesis develops automated techniques for the analysis of Windows malware that employs anti-analysis defenses and environment-dependent, multi-path behavior. It introduces two complementary systems that transform adversarial logic from an obstacle to a systematic source of insight. Rather than pursuing full transparency of execution, the dissertation treats evasive behavior as an observable signal. This engagement-based perspective contrasts with traditional approaches that seek to conceal analysis artifacts and forms the conceptual basis for both contributions. The first contribution is an automated deobfuscation pipeline for API hashing, a widely used technique that conceals dependencies by replacing plaintext API names with digests. A slice-guided static and dynamic analysis identifies comparison sites and state-update instructions, reconstructing hashing logic and hash-to-API mappings. Each specimen is then repurposed as a hash oracle: by injecting chosen strings and emulating only the recovered slice, the pipeline derives ground-truth digests using the malware’s own code, avoiding manual re-implementation and improving robustness to routine variants. On a labeled set of eight Windows samples, the pipeline recovers hashing routines in 87.5% of cases and precisely identifies the hashing function in 62.5%. Applied to 21,991 samples exhibiting indicators of API hiding, it identifies 1,686 strong candidates for API hashing, demonstrating that the technique is both prevalent in the wild and amenable to automated deobfuscation. The second and main contribution of this thesis is PFuzzer, a coverage-guided fuzzing framework for systematically exploring environment-sensitive execution paths. PFuzzer models environmental conditions as fuzzable inputs, automatically mutating system responses to induce latent behaviors. Evaluated on a curated corpus of 1,078 Windows malware samples, PFuzzer exposes additional conspicuous behavior in 42.39% of specimens and consistently outperforms state-of-the-art systems such as BluePill and Enviral in head-to-head comparisons.