# Privacy-Preserving Machine Learning ## Protected information and trust models - **Differential privacy** - Bounds how output distributions change between neighboring datasets. The adjacency relation defines the protected unit; epsilon and delta describe a mechanism's guarantee, not the probability that an individual record is safe. [Dwork and Roth 2014]( "The Algorithmic Foundations of Differential Privacy; Found. Trends TCS"). - **Record-level versus user-level privacy** - Changing one record and changing all records of one person are different adjacency relations. Group privacy degrades with group size; a record-level guarantee cannot be reported unchanged as protection for a whole user. [Dwork and Roth 2014]( "The Algorithmic Foundations of Differential Privacy; Found. Trends TCS"). - **Local differential privacy: RAPPOR** - Randomizes client reports using randomized response and Bloom filters for string statistics. Repeated-report protection needs separate accounting; local privacy trades statistical efficiency for reduced trust in a collector. [Erlingsson et al. 2014]( "RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response; CCS"). - **Shuffle-model amplification** - Anonymously shuffling locally randomized reports can strengthen the central DP guarantee. Amplification relies on the shuffling protocol and adversary's view; removing names from otherwise revealing records is not the same mechanism. [Erlingsson et al. 2019]( "Amplification by Shuffling: From Local to Central Differential Privacy via Anonymity; SODA"). - **Secure aggregation** - Lets a server recover aggregate client updates without seeing individual updates under stated protocol assumptions. Protects messages during computation; the aggregate or released model may still reveal training information. [Bonawitz et al. 2017]( "Practical Secure Aggregation for Privacy-Preserving Machine Learning; CCS"). - **Encrypted inference: CryptoNets** - Evaluates a transformed neural network on homomorphically encrypted inputs and returns encrypted predictions. Protects the inference input from the evaluator; this is a different target from DP protection for training examples. [Gilad-Bachrach et al. 2016]( "CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy; ICML"). ## Noise mechanisms and privacy accounting - **Sensitivity-calibrated mechanisms** - Laplace and Gaussian mechanisms calibrate noise to how much a query can change on neighboring data. Bounding contributions is essential; adding noise without sensitivity analysis does not establish differential privacy. [Dwork and Roth 2014]( "The Algorithmic Foundations of Differential Privacy; Found. Trends TCS"). - **Composition and post-processing** - Repeated private accesses consume privacy budget; computation using only a private release preserves its guarantee. New access to raw private data is not post-processing, even when it occurs during model selection. [Dwork and Roth 2014]( "The Algorithmic Foundations of Differential Privacy; Found. Trends TCS"). - **Zero-concentrated differential privacy** - Controls privacy-loss tails through Rényi-divergence bounds across orders. Gives convenient composition and group-privacy analysis; its parameter must be converted before comparison with epsilon-delta DP. [Bun and Steinke 2016]( "Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds; TCC"). - **Rényi differential privacy** - Bounds Rényi divergence between outputs on neighboring datasets. Privacy costs add at each order under composition, then convert to an epsilon-delta guarantee; the selected order affects the resulting bound. [Mironov 2017]( "Renyi Differential Privacy; CSF"). - **Subsampling amplification** - Bounds privacy after sampling a subset before applying a private mechanism. Sampling probability and scheme enter the theorem; an accountant must match how the implementation actually forms training batches. [Wang et al. 2019]( "Subsampled Rényi Differential Privacy and Analytical Moments Accountant; AISTATS"). - **Privacy-loss random-variable accounting** - Numerically composes privacy-loss distributions with bounded approximation error. Tightens accounting for repeated mechanisms; evaluates a specified mechanism without auditing whether code implements it. [Gopi et al. 2021]( "Numerical Composition of Differential Privacy; NeurIPS"). - **Gaussian differential privacy** - Describes privacy through hypothesis-testing tradeoffs between neighboring outputs. Gaussian DP provides a one-parameter family and composition approximations; asymptotic approximations and finite-composition bounds should be distinguished. [Dong et al. 2022]( "Gaussian Differential Privacy; JRSS B"). - **Private hyperparameter selection** - Non-private tuning can leak information even when the final training run is private. Papernot and Steinke analyze search procedures under Rényi DP; reported budgets must account for the selection procedure and its assumptions. [Papernot and Steinke 2022]( "Hyperparameter Tuning with Renyi Differential Privacy; ICLR"). - **Adaptive f-DP filter failure** - Tensor products of realized privacy trade-off curves can underestimate loss when mechanism choices depend on prior outputs, even for subsampled Gaussian mechanisms. Standard composition accounting need not yield a valid adaptive filter. [Tran et al. 2026]( "$f$-Differential Privacy Filters: Validity and Approximate Solutions; NeurIPS"). ## Private optimization and knowledge transfer - **Output and objective perturbation** - Adds randomness to parameters or the optimization objective. Chaudhuri et al. prove privacy for regularized ERM under convexity, smoothness and sensitivity conditions; arbitrary neural objectives need separate analysis. [Chaudhuri et al. 2011]( "Differentially Private Empirical Risk Minimization; JMLR"). - **DP-SGD** - Clips per-example gradients, adds Gaussian noise and accounts for sampled iterations. Clipping bounds each example's contribution; noise scale, sample rate and training steps jointly determine the privacy guarantee. [Abadi et al. 2016]( "Deep Learning with Differential Privacy; CCS"). - **Private Aggregation of Teacher Ensembles** - Trains teachers on disjoint private partitions and labels public inputs through noisy voting. A student learns from these labels; privacy belongs to the aggregation mechanism, not merely to using an ensemble. [Papernot et al. 2017]( "Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data; ICLR"). - **Selective teacher queries** - Adds Gaussian noise to teacher votes and a noisy consensus threshold that can withhold labels. Selective querying reduces disclosure on disagreements; data-dependent privacy accounting includes the threshold test and answered queries. [Papernot et al. 2018]( "Scalable Private Learning with PATE; ICLR"). - **User-level private federated learning** - Clips and noises user contributions rather than individual-example gradients. McMahan et al. train recurrent language models with user-level DP; grouping, participation sampling and contribution limits define the protected unit. [McMahan et al. 2018]( "Learning Differentially Private Recurrent Language Models; ICLR"). - **DP-FTRL** - Uses tree aggregation to perturb cumulative clipped gradients with noise correlated across steps. Avoids reliance on sampling/shuffling amplification; repeated contributions still need accounting. Trades noise structure against optimization and compute. [Kairouz et al. 2021]( "Practical and Private (Deep) Learning without Sampling or Shuffling; ICML"). - **Private synthetic data: AIM** - Adaptively selects and privately measures statistical queries, then fits synthetic data to noisy answers. Utility depends on the query workload; synthetic appearance alone is no privacy guarantee, and downstream fidelity needs evaluation. [McKenna et al. 2022]( "AIM: An Adaptive and Iterative Mechanism for Differentially Private Synthetic Data; arXiv"). ## Pretraining and empirical utility - **Private language-model fine-tuning** - Li et al. combine public pretraining, matched objectives and DP-specific tuning; ghost clipping avoids materializing full per-example gradients. Privacy covers the private fine-tuning data, not an unaccounted pretraining corpus. [Li et al. 2022]( "Large Language Models Can Be Strong Differentially Private Learners; ICLR"). - **Private vision at scale** - Averages gradients over multiple augmentations before clipping each image; large batches and normalization improve tested DP-SGD accuracy. Reported privacy budgets exclude hyperparameter-search costs, so they do not cover the full tuning procedure. [De et al. 2022]( "Unlocking High-Accuracy Differentially Private Image Classification through Scale; arXiv"). ## Attacks, memorization and empirical auditing - **Membership inference** - Infers whether a candidate record was used for training from model responses. Shokri et al. train shadow-model attacks; success depends on access and data assumptions, while a failed attack does not prove privacy. [Shokri et al. 2017]( "Membership Inference Attacks against Machine Learning Models; IEEE S&P"). - **Canary exposure** - Inserts rare synthetic sequences and measures how unusually highly the model ranks them. Exposure quantifies tested memorization and extraction risk; it is not a universal certificate for all private information. [Carlini et al. 2018]( "The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks; arXiv"). - **Deep Leakage from Gradients** - Optimizes dummy inputs to match shared gradients and reconstructs training data in studied image/text settings. Keeping raw records on-device does not ensure that transmitted gradients conceal them. [Zhu et al. 2019]( "Deep Leakage from Gradients; NeurIPS"). - **Training-text extraction** - Carlini et al. recover verbatim training sequences from GPT-2, including examples appearing in one document. Generation can reveal memorized data; membership classification and extracting an unknown sequence are distinct attack targets. [Carlini et al. 2020]( "Extracting Training Data from Large Language Models; arXiv"). - **Gradient inversion beyond toy settings** - Uses magnitude-invariant gradient matching to reconstruct high-resolution images from trained networks. Architecture, batch size and attacker knowledge matter; the experiments refute treating gradient sharing as inherently private. [Geiping et al. 2020]( "Inverting Gradients -- How easy is it to break privacy in federated learning?; NeurIPS"). - **Likelihood Ratio Attack** - Fits per-example distributions of model confidence with and without membership. Evaluates true positives at low false-positive rates, exposing risks hidden by average attack accuracy; shadow-model assumptions affect the attack. [Carlini et al. 2022]( "Membership Inference Attacks From First Principles; IEEE S&P"). - **One-run privacy auditing** - Uses many independently included or excluded examples to audit a mechanism in one training run. Produces empirical privacy lower bounds under the audit's assumptions; it can expose a claimed bound's failure, not certify every possible attack. [Steinke et al. 2023]( "Privacy Auditing with One (1) Training Run; NeurIPS"). - **Data laundering detection** - Rewritten training texts can evade tests that query originals. SDR searches synthetic rewrites to restore detector separation; tests use fine-tuned MIMIR models, likelihood-based scores and known nonmember references, not arbitrary API-only provenance. [Li et al. 2026]( "Combating Data Laundering in LLM Training; NeurIPS"). ## Software - **Opacus** - Implements DP-SGD in PyTorch. Its validator rejects BatchNorm, whose batch statistics couple records; alternatives include GroupNorm. Sampling and accounting must still match the executed training procedure. [docs](). ## Related topics - [Federated learning](https://mlpapers.org/federated-learning/) - [Distributed learning](https://mlpapers.org/distributed-learning/) - [Machine unlearning](https://mlpapers.org/machine-unlearning/) - [Evaluation](https://mlpapers.org/evaluation/) - [Generative models](https://mlpapers.org/generative-models/)