Merge branch 'main' into main

README.md

@@ -2,39 +2,54 @@
 
 <hr/>
 
-[![CircleCI](https://dl.circleci.com/status-badge/img/gh/pytorch/opacus/tree/main.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/pytorch/opacus/tree/main)
+[![GitHub Actions](https://github.com/pytorch/opacus/actions/workflows/ci_cpu.yml/badge.svg)](https://github.com/pytorch/opacus/actions/workflows/ci_cpu.yml)
 [![Coverage Status](https://coveralls.io/repos/github/pytorch/opacus/badge.svg?branch=main)](https://coveralls.io/github/pytorch/opacus?branch=main)
 [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
 [![License](https://img.shields.io/badge/license-apache2-green.svg)](LICENSE)
 
-[Opacus](https://opacus.ai) is a library that enables training PyTorch models with differential privacy.
-It supports training with minimal code changes required on the client, has little impact on training performance, and allows the client to online track the privacy budget expended at any given moment.
+[Opacus](https://opacus.ai) is a library that enables training PyTorch models
+with differential privacy. It supports training with minimal code changes
+required on the client, has little impact on training performance, and allows
+the client to online track the privacy budget expended at any given moment.
+
 
 ## Target audience
+
 This code release is aimed at two target audiences:
-1. ML practitioners will find this to be a gentle introduction to training a model with differential privacy as it requires minimal code changes.
-2. Differential Privacy researchers will find this easy to experiment and tinker with, allowing them to focus on what matters.
 
+1. ML practitioners will find this to be a gentle introduction to training a
+   model with differential privacy as it requires minimal code changes.
+2. Differential Privacy researchers will find this easy to experiment and tinker
+   with, allowing them to focus on what matters.
 
 ## Installation
+
 The latest release of Opacus can be installed via `pip`:
+
 ```bash
 pip install opacus
 ```
+
 OR, alternatively, via `conda`:
+
 ```bash
 conda install -c conda-forge opacus
 ```
 
-You can also install directly from the source for the latest features (along with its quirks and potentially occasional bugs):
+You can also install directly from the source for the latest features (along
+with its quirks and potentially occasional bugs):
+
 ```bash
 git clone https://github.com/pytorch/opacus.git
 cd opacus
 pip install -e .
 ```
 
 ## Getting started
-To train your model with differential privacy, all you need to do is to instantiate a `PrivacyEngine` and pass your model, data_loader, and optimizer to the engine's `make_private()` method to obtain their private counterparts.
+
+To train your model with differential privacy, all you need to do is to
+instantiate a `PrivacyEngine` and pass your model, data_loader, and optimizer to
+the engine's `make_private()` method to obtain their private counterparts.
 
 ```python
 # define your components as usual
@@ -54,21 +69,25 @@ model, optimizer, data_loader = privacy_engine.make_private(
 # Now it's business as usual
 ```
 
-The [MNIST example](https://github.com/pytorch/opacus/tree/main/examples/mnist.py) shows an end-to-end run using Opacus. The [examples](https://github.com/pytorch/opacus/tree/main/examples/) folder contains more such examples.
+The
+[MNIST example](https://github.com/pytorch/opacus/tree/main/examples/mnist.py)
+shows an end-to-end run using Opacus. The
+[examples](https://github.com/pytorch/opacus/tree/main/examples/) folder
+contains more such examples.
 
 ### Migrating to 1.0
 
-Opacus 1.0 introduced many improvements to the library, but also some breaking changes.
-If you've been using Opacus 0.x and want to update to the latest release,
-please use this [Migration Guide](https://github.com/pytorch/opacus/blob/main/Migration_Guide.md)
-
+Opacus 1.0 introduced many improvements to the library, but also some breaking
+changes. If you've been using Opacus 0.x and want to update to the latest
+release, please use this
+[Migration Guide](https://github.com/pytorch/opacus/blob/main/Migration_Guide.md)
 
 ## Learn more
 
 ### Interactive tutorials
 
-We've built a series of IPython-based tutorials as a gentle introduction to training models
-with privacy and using various Opacus features.
+We've built a series of IPython-based tutorials as a gentle introduction to
+training models with privacy and using various Opacus features.
 
 - [Building an Image Classifier with Differential Privacy](https://github.com/pytorch/opacus/blob/main/tutorials/building_image_classifier.ipynb)
 - [Training a differentially private LSTM model for name classification](https://github.com/pytorch/opacus/blob/main/tutorials/building_lstm_name_classifier.ipynb)
@@ -78,9 +97,13 @@ with privacy and using various Opacus features.
 - [Opacus Guide: Module Validator and Fixer](https://github.com/pytorch/opacus/blob/main/tutorials/guide_to_module_validator.ipynb)
 
 ## Technical report and citation
-The technical report introducing Opacus, presenting its design principles, mathematical foundations, and benchmarks can be found [here](https://arxiv.org/abs/2109.12298).
+
+The technical report introducing Opacus, presenting its design principles,
+mathematical foundations, and benchmarks can be found
+[here](https://arxiv.org/abs/2109.12298).
 
 Consider citing the report if you use Opacus in your papers, as follows:
+
 ```
 @article{opacus,
   title={Opacus: {U}ser-Friendly Differential Privacy Library in {PyTorch}},
@@ -92,21 +115,29 @@ Consider citing the report if you use Opacus in your papers, as follows:
 
 ### Blogposts and talks
 
-If you want to learn more about DP-SGD and related topics, check out our series of blogposts and talks:
+If you want to learn more about DP-SGD and related topics, check out our series
+of blogposts and talks:
 
 - [Differential Privacy Series Part 1 | DP-SGD Algorithm Explained](https://medium.com/pytorch/differential-privacy-series-part-1-dp-sgd-algorithm-explained-12512c3959a3)
 - [Differential Privacy Series Part 2 | Efficient Per-Sample Gradient Computation in Opacus](https://medium.com/pytorch/differential-privacy-series-part-2-efficient-per-sample-gradient-computation-in-opacus-5bf4031d9e22)
 - [PriCon 2020 Tutorial: Differentially Private Model Training with Opacus](https://www.youtube.com/watch?v=MWPwofiQMdE&list=PLUNOsx6Az_ZGKQd_p4StdZRFQkCBwnaY6&index=52)
 - [Differential Privacy on PyTorch | PyTorch Developer Day 2020](https://www.youtube.com/watch?v=l6fbl2CBnq0)
 - [Opacus v1.0 Highlights | PyTorch Developer Day 2021](https://www.youtube.com/watch?v=U1mszp8lzUI)
-
+- [Enabling Fast Gradient Clipping and Ghost Clipping in Opacus](https://pytorch.org/blog/clipping-in-opacus/)
 
 ## FAQ
-Check out the [FAQ](https://opacus.ai/docs/faq) page for answers to some of the most frequently asked questions about differential privacy and Opacus.
+
+Check out the [FAQ](https://opacus.ai/docs/faq) page for answers to some of the
+most frequently asked questions about differential privacy and Opacus.
 
 ## Contributing
-See the [CONTRIBUTING](https://github.com/pytorch/opacus/tree/main/CONTRIBUTING.md) file for how to help out.
-Do also check out the README files inside the repo to learn how the code is organized.
+
+See the
+[CONTRIBUTING](https://github.com/pytorch/opacus/tree/main/CONTRIBUTING.md) file
+for how to help out. Do also check out the README files inside the repo to learn
+how the code is organized.
 
 ## License
-This code is released under Apache 2.0, as found in the [LICENSE](https://github.com/pytorch/opacus/tree/main/LICENSE) file.
+
+This code is released under Apache 2.0, as found in the
+[LICENSE](https://github.com/pytorch/opacus/tree/main/LICENSE) file.

docs/faq.md

@@ -108,7 +108,8 @@ Opacus computes and stores *per-sample* gradients under the hood. What this mean
 
 Although we report expended privacy budget using the (epsilon, delta) language, internally, we track it using Rényi Differential Privacy (RDP) [[Mironov 2017](https://arxiv.org/abs/1702.07476), [Mironov et al. 2019](https://arxiv.org/abs/1908.10530)]. In short, (alpha, epsilon)-RDP bounds the [Rényi divergence](https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy#R%C3%A9nyi_divergence) of order alpha between the distribution of the mechanism’s outputs on any two datasets that differ in a single element. An (alpha, epsilon)-RDP statement is a relaxation of epsilon-DP but retains many of its important properties that make RDP particularly well-suited for privacy analysis of DP-SGD. The `alphas` parameter instructs the privacy engine what RDP orders to use for tracking privacy expenditure.
 
-When the privacy engine needs to bound the privacy loss of a training run using (epsilon, delta)-DP for a given delta, it searches for the optimal order from among `alphas`. There’s very little additional cost in expanding the list of orders. We suggest using a list `[1 + x / 10.0 for x in range(1, 100)] + list(range(12, 64))`. You can pass your own alphas by passing `alphas=custom_alphas` when calling `privacy_engine.make_private_with_epsilon`.
+When the privacy engine needs to bound the privacy loss of a training run using (epsilon, delta)-DP for a given delta, it searches for the optimal order from among `alphas`. There’s very little additional cost in expanding the list of orders. We suggest using a list `[1 + x / 10.0 for x in range(1, 100)] + list(range(12, 64))`.
+<!-- You can pass your own alphas by passing `alphas=custom_alphas` when calling `privacy_engine.make_private_with_epsilon`. -->
 
 A call to `privacy_engine.get_epsilon(delta=delta)` returns a pair: an epsilon such that the training run satisfies (epsilon, delta)-DP and an optimal order alpha. An easy diagnostic to determine whether the list of `alphas` ought to be expanded is whether the returned value alpha is one of the two boundary values of `alphas`.
 

opacus/accountants/gdp.py

@@ -44,7 +44,7 @@ def step(self, *, noise_multiplier: float, sample_rate: float):
         else:
             self.history = [(noise_multiplier, sample_rate, 1)]
 
-    def get_epsilon(self, delta: float, poisson: bool = True) -> float:
+    def get_epsilon(self, delta: float, poisson: bool = True, **kwargs) -> float:
         """
         Return privacy budget (epsilon) expended so far.
 

opacus/accountants/prv.py

@@ -81,7 +81,12 @@ def step(self, *, noise_multiplier: float, sample_rate: float):
             self.history.append((noise_multiplier, sample_rate, 1))
 
     def get_epsilon(
-        self, delta: float, *, eps_error: float = 0.01, delta_error: float = None
+        self,
+        delta: float,
+        *,
+        eps_error: float = 0.01,
+        delta_error: float = None,
+        **kwargs,
     ) -> float:
         """
         Return privacy budget (epsilon) expended so far.

opacus/accountants/rdp.py

@@ -68,7 +68,10 @@ def get_privacy_spent(
         return float(eps), float(best_alpha)
 
     def get_epsilon(
-        self, delta: float, alphas: Optional[List[Union[float, int]]] = None
+        self,
+        delta: float,
+        alphas: Optional[List[Union[float, int]]] = None,
+        **kwargs,
     ):
         """
         Return privacy budget (epsilon) expended so far.

opacus/optimizers/optimizer_fast_gradient_clipping.py

@@ -14,6 +14,7 @@
 
 from __future__ import annotations
 
+import copy
 import logging
 from typing import Callable, Optional
 
@@ -112,9 +113,9 @@ def accumulate(self):
         """
         for p in self.params:
             if p.summed_grad is not None:
-                p.summed_grad += p.grad
+                p.summed_grad.add_(p.grad.data)
             else:
-                p.summed_grad = p.grad
+                p.summed_grad = copy.deepcopy(p.grad.data)
 
     def zero_grad(self, set_to_none: bool = False):
         """

opacus/privacy_engine.py

@@ -136,7 +136,6 @@ def _prepare_optimizer(
             loss_reduction=loss_reduction,
             generator=generator,
             secure_mode=self.secure_mode,
-            **kwargs,
         )
 
     def _prepare_data_loader(
@@ -212,6 +211,26 @@ def _prepare_model(
                     loss_reduction=loss_reduction,
                 )
 
+    def _prepare_criterion(
+        self,
+        *,
+        module: GradSampleModule,
+        optimizer: DPOptimizer,
+        criterion=nn.CrossEntropyLoss(),
+        loss_reduction: str = "mean",
+        **kwargs,
+    ) -> DPLossFastGradientClipping:
+        """
+        Args:
+            module: GradSampleModule used for training,
+            optimizer: DPOptimizer used for training,
+            criterion: Loss function used for training,
+            loss_reduction: "mean" or "sum", indicates if the loss reduction (for aggregating the gradients)
+
+        Prepare the DP loss class, which packages the two backward passes for fast gradient clipping.
+        """
+        return DPLossFastGradientClipping(module, optimizer, criterion, loss_reduction)
+
     def is_compatible(
         self,
         *,
@@ -403,9 +422,14 @@ def make_private(
             self.accountant.get_optimizer_hook_fn(sample_rate=sample_rate)
         )
         if grad_sample_mode == "ghost":
-            criterion = DPLossFastGradientClipping(
-                module, optimizer, criterion, loss_reduction
+            criterion = self._prepare_criterion(
+                module=module,
+                optimizer=optimizer,
+                criterion=criterion,
+                loss_reduction=loss_reduction,
+                **kwargs,
             )
+
             return module, optimizer, criterion, data_loader
 
         return module, optimizer, data_loader