An Efficient Framework for Crediting Data Contributors of Diffusion Models

Abstract

As diffusion models are deployed in real-world settings, and their performance is driven by training data, appraising the contribution of data contributors is crucial to creating incentives for sharing quality data and to implementing policies for data compensation. Depending on the use case, model performance corresponds to various global properties of the distribution learned by a diffusion model (e.g., overall aesthetic quality). Hence, here we address the problem of attributing global properties of diffusion models to data contributors. The Shapley value provides a principled approach to valuation by uniquely satisfying game-theoretic axioms of fairness. However, estimating Shapley values for diffusion models is computationally impractical because it requires retraining on many training data subsets corresponding to different contributors and rerunning inference. We introduce a method to efficiently retrain and rerun inference for Shapley value estimation, by leveraging model pruning and fine-tuning. We evaluate the utility of our method with three use cases: (i) image quality for a DDPM trained on a CIFAR dataset, (ii) demographic diversity for an LDM trained on CelebA-HQ, and (iii) aesthetic quality for a Stable Diffusion model LoRA-finetuned on Post-Impressionist artworks. Our results empirically demonstrate that our framework can identify important data contributors across models' global properties, outperforming existing attribution methods for diffusion models.

Method

We propose a scalable framework for estimating Shapley values to quantify the contribution of training data contributors to diffusion models. Instead of retraining a full model for each subset, we introduce sparsified fine-tuning: a pretrained model is pruned to create a sparse proxy, which is then fine-tuned on subsets sampled from the Shapley distribution. Global properties are evaluated via inference to approximate full retraining:

\[ \mathcal{F}(\tilde{\theta}^{\mathrm{ft}}_{S_j, k}) \approx \mathcal{F}(\theta^*_{S_j}), \]

where \(\theta^*_{S_j}\) is a model retrained on subset \(S_j\), and \(\tilde{\theta}^{\mathrm{ft}}_{S_j, k}\) is the corresponding fine-tuned sparse model after \(k\) steps. We provide a formal guarantee that the Shapley estimate remains close to the ground truth. Let \(\beta^*\) denote the true Shapley value, and \(\tilde{\beta}^{\mathrm{ft}}_k\) be its approximation. Then:

\[ \mathbb{E} \left[ \left\| \tilde{\beta}_k^{\mathrm{ft}} - \beta^* \right\|_2 \right] \leq 2 \sqrt{n} C \quad \text{as } k \to \infty, \]

where \(n\) is the number of contributors and \(C\) is a constant depending on the model's Lipschitz properties. This result justifies using sparsified fine-tuning in place of expensive full retraining. See Appendix A for details.

Results

We evaluate our approach across three datasets—CIFAR-20, CelebA-HQ, and ArtBench (Post-Impressionism)—each with a distinct global property: Inception Score, demographic diversity, and aesthetic quality, respectively. Our method, sparsified-FT Shapley, consistently outperforms existing attribution methods across all tasks.

As shown below, our method achieves significantly higher Linear Datamodeling Scores (LDS) than baselines such as TRAK, influence functions, and pixel similarity—demonstrating more faithful alignment with global model behavior. Additionally, sparsified fine-tuning reduces attribution runtime by up to 18× while maintaining high attribution accuracy, enabling evaluation over more Shapley subsets under the same compute budget.

Table 1: Contributor attribution performance measured by LDS (%), evaluated at \(\alpha = 0.5\). Our method (sparsified-FT Shapley) achieves the highest LDS across all datasets. Error bars indicate 95% confidence intervals over three runs.

Figure 2 highlights how our method compares under computational constraints. Given the same budget (defined as the time required to retrain and evaluate one full model), sparsified fine-tuning achieves significantly better LDS performance than standard fine-tuning or full retraining. This demonstrates that sparsified-FT enables broader coverage of the Shapley sampling space, resulting in more faithful contributor valuation under limited resources.

Figure 2: LDS (%) as a function of computational budget. Sparsified-FT Shapley achieves the best attribution performance under limited compute, outperforming both fine-tuning and full retraining.

In Figure 3, we perform counterfactual evaluations by removing or retaining top-ranked contributors. Removing contributors identified by our method leads to the largest performance drop across datasets (e.g., −23.2% on CIFAR-20), while retaining them yields the highest gains (e.g., +20.0% on CelebA-HQ). Compared to baselines like CLIP similarity, LOO, and influence functions, our method consistently identifies the most impactful data.

Figure 3: Counterfactual evaluation of top contributors. We compare model performance before and after removing (top) or retaining (bottom) the top 40–60% contributors, as ranked by different attribution methods. Our method leads to the most significant improvements or degradation, confirming more accurate contributor valuation.

BibTeX


@inproceedings{luefficient,
  title={An Efficient Framework for Crediting Data Contributors of Diffusion Models},
  author={Lu, MingYu and Lin, Chris and Kim, Chanwoo and Lee, Su-In},
  booktitle={The Thirteenth International Conference on Learning Representations}
}