Differential Privacy

Differential Privacy (DP) is a framework for using datasets without revealing information about individual data points. It is designed to protect the privacy of individuals within a dataset while enabling meaningful data analysis. DP achieves this by introducing controlled statistical noise to the data or model outputs, ensuring that the presence or absence of any single data point does not significantly affect the results. Originally developed for statistical analysis, DP has since been adapted to machine learning models, safeguarding against data leakage even during model training. In Machine Learning, this approach has been primarily applied to tabular datasets and classification systems . Recent Research has proposed extensions for Large Language Models.
DP does not change the original data points but modifies the behavior when a dataset is queried. Instead of returning the raw distribution of values, noise is added to change the exact values. The addition of noise implies that adding or removing specific data points will not change the returned results. Finding the right amount of "noise" is a trade-off between stronger privacy guarantees and keeping meaningful signals in the data.
DP principles have been extended to Machine Learning because models can overfit to a specific dataset and be manipulated to reveal information about specific datapoints. A broad model-agnostic approach involves training multiple copies of the model on different subsets of the data, pooling the votes, adding noise, and returning the top predicted class. Other approaches modify specific algorithms to similarly dilute the signal from individual training datapoints.

DP mitigates the risk of re-identification attacks, data leakage, and membership inference attacks by ensuring that sensitive information about individuals cannot be reverse-engineered from datasets or trained models. By adding noise to query outputs or gradients during model training, DP prevents adversaries from distinguishing whether specific data points were included in the dataset. This approach reduces risks such as:

- Membership Inference Attacks: Attackers cannot confidently determine if a particular individual's data was used to train the model.
- Re-identification Risks: Adversaries cannot link anonymized data back to individuals even with auxiliary datasets.
- Data Leakage: Protects against both direct and indirect data leaks from aggregated outputs or ML models.

- Example Code Using Differential Privacy
- Screenshots or code snippets showing how DP libraries like PyDP, Opacus, or TensorFlow Privacy are used in AI models or data analysis.
- Privacy Settings Documentation
- Internal documents explaining how privacy metrics are chosen and why they are set at those levels.
- Records of DP-Protected Data Queries
- Logs or reports showing how data queries were modified using DP techniques to protect individual privacy.