Artificial Intelligence (AI) systems are becoming part of everyday life, utilized for generating and editing text, recommending content, and recently in exploring mathematical and scientific data. However, AI tools that memorize, that is, store and reproduce outcomes directly seen in their training sets, pose several risks related to security, privacy, and reliability. Through a $1M multi-year National Science Foundation’s Mathematical Foundations of Artificial Intelligence (MFAI) grant, Hayden Schaeffer (Professor of Mathematics, UCLA), Abolfazl Hashemi (Assistant Professor in Electrical and Computer Engineering at Purdue University), and Kaushik Roy (Edward G. Tiedemann Jr. Distinguished Professor of Electrical and Computer Engineering at Purdue University) will investigate how and why AI models memorize data.
When AI models are trained, they learn patterns from large datasets. The hope is that with a large and rich enough training dataset, the learned model will be able to generalize, that is, will be able to predict outcomes from unseen settings. However, with larger models a potential risk is that the AI systems memorize parts of their training data, which can lead to serious issues. For example, if a model retains sensitive data, it could potentially reveal private or secure information through its outputs. Memorization may also contribute to models that perform poorly when applied to new (unseen) inputs. Although observed in large language models (LLMs), the mathematical, statistical, and computational underpinnings of memorization remain poorly understood.
This newly funded research effort aims to address memorization by developing mathematical and statistical techniques to detect and quantify memorization during training. The research team will formalize the quantitative aspects of memorization and construct computationally tractable indicators to detect when and how memorization occurs. This work is at the intersection of mathematics, statistics, and computer science, utilizing and developing methods in optimization, information theory, and dynamics. By understanding subtleties in the training of AI systems, this research could help to ensure that these systems lead to positive societal and scientific outcomes.
Learn more about the program here.
The award is split into two parts, learn more below:
