QBI: Quantile-based Bias Initialization for Efficient Private Data Reconstruction in Federated Learning

In this scenario, the attacker is assumed to have no control over the model’s architecture, its parameters, or the specific FL protocol that is being used. The gradients created through a standard FL protocol could either be obtained by an honest but curious central entity, or an external actor via a man-in-the-middle (MITM) attack. A long line of work has demonstrated, that simply by obtaining these gradients, an attacker could learn about properties of the data [7, 8], data membership [8], and even partially reconstruct user data [9, 10, 11].

The second major threat model assumes a malicious central entity, that has the ability to modify the model’s architecture or parameters in an attempt to break the user’s privacy. A broad range of research has outlined a multitude of possible active attacks, that often leverage induced gradient sparsity [12, 9, 13] or sample disaggregation [14, 15] to recover user data.

Each of these approaches strikes a different balance between computational overhead, quality of the reconstructed data, client-side detectability and the requirement for knowledge about the user data, including certain features and their distributions.

In this work, we propose a novel method of maliciously initializing a model, by only modifying the bias values of a fully connected layer, while leaving the weights completely randomly initialized. This reduces client-side detectability in parameter space compared to methods that either maliciously train the full model to compromise privacy [16] or introduce a shift in the magnitude of all weight values [12]. Additionally our method removes the need for experimentally determined hyperparameters [12] or handpicked target features or classes [17]. We provide two variants of our approach: Quantile-Based Bias Initialization (QBI), which directly determines the optimal bias with near-zero computational cost, and Pattern-Aware Iterative Random Search (PAIRS) which builds on QBI, but further enhances reconstruction success by incorporating auxiliary data and incurring marginally higher computational overhead. Additionally, we derive boundaries for the expected success of attacks leveraging stochastic gradient sparsity, which enables us to identify the inherent constraints of such attacks. Using the findings described in this paper, we propose a novel defensive framework – AGGP – aimed at preventing gradient sparsity attacks, contributing to the development of more secure and private FL systems.

FL is a decentralized approach to machine learning that enables multiple parties to collaboratively train a shared model on their local data without sharing the data itself. Each client has a local dataset and computes an update to the model parameters based on their local data. The update is typically computed as the gradient of the local loss function with respect to the model parameters. The local updates are then aggregated to update the global model parameters. One common aggregation method is federated averaging, which computes the weighted average of the local updates. This process is repeated over multiple rounds to train the global model [6].

As demonstrated by Geiping et al. [18], it is feasible to extract a single input from the gradients of a fully-connected layer, given that the layer is preceded only by fully connected layers, has a bias $b$, uses the ReLU activation function, and the gradient of the loss with respect to the layer’s output contains at least one non-zero entry (see Proposition D.1 in Geiping et al. [18] for a full proof). If this layer is placed at the beginning of the network, this corresponds to reconstructing the original input data point $x$. As shown in Section 5.1 of Boenisch et al. [12], for any non-zero output $y_{i}$, the gradient of the corresponding weight row directly contains the input scaled by the gradient of the loss with respect to the bias. In practice, gradients are typically computed as the average over an entire batch of samples. Since a single neuron is often activated by multiple samples, the gradients of its weight row contain the average of multiple input data points, effectively obscuring them and preventing individual extraction. To extract an input sample $x$, there has to exist a neuron $n_{i}$ such that $L(x)_{i}>0$ and $L(x^{\prime})_{i}<0$ for all $x^{\prime}\neq x$, where $L(x)_{i}$ denotes the activation of the $i$-th neuron for sample $x$. Since the samples that are to be extracted are unknown, initializing a layer to achieve this specific activation pattern is challenging. Therefore, the goal of stochastic gradient sparsity attacks is to increase the probability that a neuron is activated only by a single sample, while producing a variety of neurons with diverse activation patterns, to capture as much data from the target domain as possible.

Boenisch et al. [12] extend the gradient sparsity attack to Convolutional Neural Networks (CNNs), where one or more convolutional layers precede the first linear layer. By utilizing zero-padding, a stride of one, and maliciously initialized filters, convolutional layers can effectively be transformed into identity functions, which perfectly transmit the inputs deeper into the network, allowing them to be extracted once they pass through a fully connected layer. For a detailed description, including visualizations, see Appendix B of Boenisch et al. [12].

Given a linear layer $L$ of shape $N\times M$, that uses the ReLU activation function, and a batch $\tilde{X}$ of $B$ samples $x_{1},\cdots,x_{B}$, the objective is that for every $x$ there exists one and only one neuron $n_{i}$ such that $L(x)_{i}>0$ and $L(x^{\prime})_{i}<0$ for all $x^{\prime}\neq x$, where $L(x)_{i}$ denotes the activation of the $i$-th neuron for sample $x$. This ensures that neuron $n_{i}$ allows for perfect reconstruction of $x$ from the gradients of its weight row, while producing zero gradients for all other samples. Therefore, for every neuron $n_{i}$ the desired probability to activate for any sample in the batch is $1/B$. We take the model’s weights and our approximated normalized features to be independently and identically distributed (i.i.d.) random variables sampled from a normal distribution:

Using this assumption, the probability of a single neuron, with corresponding weight row $w_{i}$ and bias $b_{i}$, activating for a sample $x_{i}\in\mathbb{R}^{M}$ can be expanded as:

Now the left-hand side is a sum of i.i.d. random variables, each of which has characteristic function:

In turn, the characteristic function of the entire sum is:

This immediately identifies it as distributed like:

where $Q_{1}$ and $Q_{2}$ are independent and both of the chi-square distribution with $M$ degrees of freedom. Note that the variance of each addend is $V(w_{ij}x_{ij})=1$ (derived by evaluating the second derivative of Equation 3 at 0). We can drop the assumption that our features are normally-distributed and instead assume that the data has been normalized, as we do in our experiments. Even with this change, the variance of each addend still evaluates to $1$ as we maintain the assumption that the weights are independently- and normally-distributed. This enables us to apply the Central Limit Theorem to the original sum in Equation 2. As a sum of i.i.d. random variables with existing second moments, we can make a statement about convergence in distribution:

where $\Phi$ is the cumulative distribution function of a standard normal distribution. Meaning we can solve for the asymptotically optimal bias $b_{*}$ by using the inverse cdf or quantile function:

It can be helpful to conceptualize the relationship between this linear layer and a set of samples as a bipartite random graph. The nodes of one partition class are neurons from the neuron set $\smash{\tilde{N}}=\{n_{1},\ldots,n_{n}\}$, and those in the other partition are from the sample set $\smash{\tilde{X}}=\{x_{1},\ldots,x_{B}\}$. A vertex $(n_{i},x_{j})\in\smash{\tilde{N}}\times\smash{\tilde{X}}$ is said to exist if $n_{i}$ is activated by $x_{j}$. If the malicious actor wants to reconstruct a sample $x_{j}$, that sample needs to have a neighbor $n_{i}$ with degree $\delta(n_{i})=1$.

We closely follow the extraction metrics proposed by Boenisch et al. [12]. Given a linear layer with $N$ output neurons and a batch $\tilde{X}$ of $B$ samples $x_{1},\cdots,x_{B}$, we define the following metrics:

1. 1.

   Active neurons ($A$): This metric represents the percentage of neurons in layer $L$ that activate for at least one of the $B$ samples. Formally, let $N_{A}$ be the number of neurons $n_{i}$ that satisfy the condition that there exists at least one input $x$ in $\tilde{X}$ such that $L(x)_{i}>0$, where $L(x)_{i}$ denotes the activation of the $i$-th neuron in $L$ for sample $x$. For the neuron conceptualized as a graph node, this condition is equivalent to $\delta(n_{i})\geq 1$. The active neurons metric $A$ is therefore defined as the ratio of $N_{A}$ to the total number of neurons $N$:

   $$A=\frac{N_{A}}{N}$$

   (8)

2. 2.

   Extraction-Precision ($P$): This metric measures the percentage of neurons that allow for the extraction of individual data points. Specifically, let $N_{u}$ be the number of neurons $n_{i}$ in layer $L$ with unique activations, i.e. those that satisfy the following condition: $L(x)_{i}>0$ for one input $x$ in $\tilde{X}$, and $L(x^{\prime})_{i}<0$ for all other inputs $x^{\prime}\neq x$. This condition is equivalent to $\delta(n_{i})=1$, which effectively counts neuron nodes that are leaves of their graphs. The extraction-precision $P$ is defined as the following ratio:

   $$P=\frac{N_{u}}{N}$$

   (9)

3. 3.

   Extraction-Recall ($R$): The extraction-recall measures the percentage of input data points that can be perfectly reconstructed from any gradient row. Let $B_{0}$ be the number of data points that can be extracted with an $l_{2}$-error of zero, then $R$ is denoted as:

   $$R=\frac{B_{0}}{B}$$

   (10)

Notably, $R$ is the most significant metric, as neither $A$ nor $P$ can be used in isolation to meaningfully assess the effectiveness of an attack. A high $A$ value could lead to overlapping activations that prevent individual extraction, while a high $P$ value could be observed in a scenario where all neurons activated for the same sample. If the true probability of a neuron activating is $1/B$, we can derive the explicit probabilities $p_{A;B}$ and $p_{u;B}$ for a neuron to be counted as a success in the context of the $A$ and $P$ metrics, respectively. Since the success of one neuron does not influence the remaining neurons, the entire batch follows a binomial distribution: $N_{A}\sim\mathcal{B}(N,p_{A;B})$ and $N_{u}\sim\mathcal{B}(N,p_{u;B})$. Consequently, the expected activation share $A$ and the expected extraction-precision $P$ are:

For growing batch sizes, these converge to:

From a graph theory perspective, $R\cdot B=B_{0}$ denotes the size of the largest neuron subset $V\subset\tilde{N}$, such that the subgraph induced by the union of $V$ and all neighbors of vertices in $V$ forms a perfect matching. However in contrast to the previous metrics, the expected extraction recall $R$ exhibits a crucial difference. Our assumptions state that existence of a particular edge (representing the activation of a neuron by a data point) is independent of the existence of any other edges. Due to this, and because neurons do not have edges in common with one another, the event of inclusion of any neuron in the count for $N_{A}$ and $N_{u}$ is also independent of the inclusion of any other neuron therein. This allows us to assert the success probabilities (see Equation 11) and that $N_{A}$ and $N_{u}$ will both follow a binomial distribution. This binomial approach breaks down for $R$, however. We can and will still derive the non-zero probability (see Equation 13) for one specific data point to be included in the count (for $N_{A}$ and $N_{u}$ we were counting neurons, for $B_{0}$ we count data points). This value will depend on the size of both partition classes of the graph, not just the one being counted. Namely, the expected share of data points that the malicious actor can perfectly reconstruct is:

See Section B.3 for a more detailed explanation. To check that this does not yield the exact distribution of $R$ and that the binomial approach is no longer relevant, consider a graph with more data points than neurons, i.e. $B>N$. It is clear that $P(R=100\%)=0\neq(p_{R;B;N})^{\textsc{B}}$, since there are not enough neurons to cover each data point. As Equation 13 assumes the optimal scenario, only obtainable with per-neuron activation probabilities of $1/B$ and truly normally distributed data, it provides an upper limit for the expected success of stochastic gradient sparsity attacks on real-world data, which will yield lower expected results, the further the data deviates from being normally distributed. In Section A.3 we assess these boundaries by evaluating the extraction success on synthetic truly random datasets.

QBI maliciously initializes a linear layer $L$ before sending it to a client $k$ targeted for extraction. The weight values $w$ of $L$ are initialized from a standard normal distribution. Given a batch size $B$ used on the client side and the number of input features $M$, QBI determines the bias value $b^{*}$ using Equation 7, which approximately leads to an activation probability of $1/B$ for each neuron in $L$. Even though the true distribution of features on the user side is unknown and the features are neither independent nor truly normally-distributed, our approximation is effective in practice.

We propose the PAIRS algorithm that further adapts the malicious QBI linear layer to the target domain when auxiliary data is available. By acknowledging that real-world data, such as images, rarely exhibits the assumed i.i.d. properties, PAIRS iteratively searches the weight space to better capture the underlying patterns. The procedure, outlined in Algorithm 1, begins by performing a forward pass with a batch of auxiliary data. It then identifies and re-initializes the weight rows of neurons that are either overactive or underactive, as well as those that exhibit redundant activation patterns. Through this process, PAIRS builds neuron-sample pairs, iteratively searching the weight space until all samples are covered or a fixed number of iterations is reached. Specifically, with an output shape of $N$ and a batch size of $B$, PAIRS uses $N/B$ batches of $B$ samples for groups of $B$ neurons each. By randomly re-initializing weight values, PAIRS avoids detectability in weight space while increasing the percentage of data that can be perfectly reconstructed, surpassing the performance of plain QBI.

We evaluated our method on two benchmark vision datasets: ImageNet [21] at a resolution of $224\times 224$, and CIFAR-10 [22] at a resolution of $32\times 32$. As our method requires normalized data, we used publically available normalization parameters for both datasets. The model we used consisted of convolutional layers maliciously initialized to transfer the input further into the network, followed by a linear layer initialized with either QBI or PAIRS. The rate of perfectly reconstructed images $R$ (see Equation 10) was evaluated using batch sizes of 20, 50, 100, and 200, and layer sizes of 200, 500, and 1000. Our results, presented in Table 1, surpass those reported by Boenisch et al. [12] using their trap weights approach. Our method yields consistent improvements for both the CIFAR-10 and ImageNet datasets across various layer size and batch size combinations. Notably, the most significant gains are observed on ImageNet with smaller batch sizes, where our method achieves up to $50\%$ higher reconstruction rates. Figure 2 in Appendix A.1 presents an example batch of 20 images and the perfectly reconstructed subset of 16 images, obtained from a layer size of 200. We also conducted experiments using unnormalized data, inserting either a batch normalization layer or a layer normalization layer before the maliciously initialized linear layer. See Section B.1, B.2 and Table 6 for more details.

The IMDB sentiment analysis dataset [23] was used to evaluate the extraction of text data. Our model operates on 250-token sentences with an embedding dimension of 250, where the embedded tokens are directly fed to a fully connected layer of size 1000. We re-trained the bert-base-uncased tokenizer [24] on the IMDB dataset, resulting in a vocabulary size of $10,000$. The extraction was evaluated on batch sizes of 20, 50, 100, and 200. The results, presented in Table 2, are compared to those reported by Boenisch et al. [12]. Our approach performs similarly across batch sizes 20 and 50, but achieves performance gains of $25\%$ and $47\%$ on batch sizes 100 and 200, respectively.

The advantage of our method can be explained by examining the secondary metrics precision $P$ and activation value $A$ (see Equation 11). As established in Equation 12, the theoretical optimum for these values lies at $A=1-1/e\approx 63.2\%$ and $P=1/e\approx 36.8\%$. Although these can only be achieved if the target data is truly normally distributed (see Table 5 in the Appendix), values that lie closer to this optimum will lead to better reconstruction rates. Table 4 in the Appendix compares the $A$ and $P$ values achieved using QBI and PAIRS to those obtained using trap weights [12] on image data. Specifically, on the ImageNet dataset, our $A$ values range from $72\%$ to $87\%$, whereas those reported in Boenisch et al. [12] show a stronger variance across batch sizes, ranging from $9\%$ to $89\%$. Similarly, our precision values range from $26\%$ to $34\%$ on ImageNet, while those reported in [12] vary from $23\%$ to $94\%$.

We find that our proposed defense framework reduces the percentage of perfectly reconstructable samples obtained via gradient sparsity in linear layers to zero. This occurs as any neuron that meets the condition for perfect extraction ($a_{n}=1$), triggers the pruning of $1-0.25\cdot p_{l}$ percent of gradient values of its corresponding weight row (see Equation 14). The effect of our defense is best presented visually – the left-hand side of Figure 1, displays the data that is leaked passively from the first 20 neurons of a benign network after a single training step with a batch of 20 samples from the ImageNet dataset. The right-hand side depicts the impact of AGGP on the same scenario, where gradients of neurons with low activation counts are aggressively pruned, while those with high activation counts remain unaffected. Further visualizations of AGGP’s impact on a maliciously initialized model, along with preliminary observations of its effect on training performance, are provided in Section A.5.

Secure aggregation (SA) [25] and distributed differential privacy (DDP) [26] are proposed modifications to the traditional FL protocol, designed to decrease the amount of trust users have to place in the central entity. SA employs a multiparty computation protocol to perform decentralized gradient aggregation, while in DDP users locally add a small amount of noise to their gradient updates. Boenisch et al. [27] demonstrated that attacks leveraging gradient sparsity can undermine these protocols if the server is able to introduce malicious nodes in the form of so-called sybil devices. Since their work uses the previously described trap weights [12], replacing the model initialization with our QBI or PAIRS approach could significantly improve the performance of this attack vector.

Our method leaves all weight values completely randomly initialized, making it virtually undetectable in weight space. However, it introduces a negative shift in the bias values, which could potentially be detected by the client. Additionally, as our method relies on gradient sparsity, it would be feasible to detect it in gradient space, e.g., by leveraging measures like the disaggregation signal-to-noise ratio [14].

Boenisch et al. [12] and our attack rely on the existence of a fully connected layer, either at the beginning of the network or positioned such that preceding layers can be maliciously initialized to perfectly transmit the input deeper into the network. Preceding layers that reduce dimensionality, for example, through pooling operations, will diminish fidelity and thereby prevent perfect data extraction. We conduct preliminary tests on the impact of AGGP on training performance that indicate little-to-no adverse effects, however, a comprehensive evaluation across a wider range of architectures, datasets, and pruning functions is needed to achieve generalizable insights.