An audio inpainting system that reconstructs missing or corrupted regions of music. The core question is the audio inverse problem: given the audio before and after a missing region, can we reconstruct what plausibly belongs in the gap? Current audio inpainting tools and models are cumbersome for the average user and largely lack the ability to make use of a dedicated knowledge base. This project addresses both problems by pairing an inpainting model with a knowledge base that uses hybrid search to supply relevant reference samples, all behind a simple frontend. The work builds on recent research in token-based audio inpainting using discrete diffusion (Dror et al.).
If you are a CS student and get selected for this project, you will need to transfer to the ECE section of Capstone by end of Week 1.
Audio inpainting is an active research area, but most existing tools are built for researchers rather than everyday users. Letting the model pull relevant samples from a knowledge base is a practical way to improve reconstruction quality without retraining the model every time new data is added. The project covers deep learning, retrieval and search, and full stack development, which makes it a good way to build a complete machine learning system with real use in restoring and editing audio.
Familiarity with Python, comfortable with the command line, and able to work through complex multistep problems. Team members should be willing to learn PyTorch and collaborative development with Git. No prior experience with audio machine learning is required, but willingness to learn audio processing concepts and either web development or database and search concepts (depending on assigned block) is essential.
Interest in music, audio, or creative technology. Prior coursework in machine learning, data structures, or algorithms. Experience with web development for the frontend block, or with databases, vector search, or information retrieval for the knowledge base block. Familiarity with linear algebra, signal processing, or music theory. Most importantly, willingness to learn new tools and work through hard technical problems through research and experimentation.
