Dataset Distillation by Adversarial Prediction Matching
Authors: Mingyang Chen, Bo Huang, Junda Lu, Bing Li, Yi Wang, Minhao Cheng, Wei Wang
Abstract: Dataset distillation is the technique of synthesizing smaller condensed datasets from large distinctive datasets whereas retaining wanted data to persist the affect. On this paper, we technique the dataset distillation disadvantage from a novel perspective: we regard minimizing the prediction discrepancy on the true data distribution between fashions, which can be respectively expert on the large distinctive dataset and on the small distilled dataset, as a conduit for condensing data from the raw data into the distilled mannequin. An adversarial framework is proposed to resolve the problem successfully. In distinction to current distillation methods involving nested optimization or long-range gradient unrolling, our technique hinges on single-level optimization. This ensures the memory effectivity of our methodology and provides a flexible tradeoff between time and memory budgets, allowing us to distil ImageNet-1K using a minimal of solely 6.5GB of GPU memory. Beneath the optimum tradeoff approach, it requires solely 2.5× a lot much less memory and 5× a lot much less runtime compared with the state-of-the-art. Empirically, our methodology can produce synthetic datasets merely 10% the size of the distinctive, however receive, on widespread, 94% of the test accuracy of fashions expert on the overall distinctive datasets along with ImageNet-1K, significantly surpassing state-of-the-art. Furthermore, in depth assessments reveal that our distilled datasets excel in cross-architecture generalization capabilities.
