文章速览 | 联邦学习 x AAAI'2023 （上）

本文是由白小鱼博主整理的AAAI 2023会议中，与联邦学习相关的论文合集及摘要翻译。

Delving into the Adversarial Robustness of Federated Learning

• Authors: Jie Zhang; Bo Li; Chen Chen; Lingjuan Lyu; Shuang Wu; Shouhong Ding; Chao Wu

• Journal: Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26331

Abstract: In Federated Learning (FL), models are as fragile as centrally trained models against adversarial examples. However, the adversarial robustness of federated learning remains largely unexplored. This paper casts light on the challenge of adversarial robustness of federated learning. To facilitate a better understanding of the adversarial vulnerability of the existing FL methods, we conduct comprehensive robustness evaluations on various attacks and adversarial training methods. Moreover, we reveal the negative impacts induced by directly adopting adversarial training in FL, which seriously hurts the test accuracy, especially in non-IID settings. In this work, we propose a novel algorithm called Decision Boundary based Federated Adversarial Training (DBFAT), which consists of two components (local re-weighting and global regularization) to improve both accuracy and robustness of FL systems. Extensive experiments on multiple datasets demonstrate that DBFAT consistently outperforms other baselines under both IID and non-IID settings.

abstractTranslation: 在联邦学习（FL）中，模型与针对对抗性示例的集中训练模型一样脆弱。然而，联邦学习的对抗鲁棒性在很大程度上仍未得到探索。本文揭示了联邦学习的对抗鲁棒性挑战。为了更好地理解现有 FL 方法的对抗漏洞，我们对各种攻击和对抗训练方法进行了全面的鲁棒性评估。此外，我们还揭示了在 FL 中直接采用对抗性训练所带来的负面影响，这严重损害了测试准确性，尤其是在非独立同分布环境中。在这项工作中，我们提出了一种称为基于决策边界的联邦对抗训练（DBFAT）的新颖算法，它由两个部分组成（局部重新加权和全局正则化），以提高 FL 系统的准确性和鲁棒性。对多个数据集的大量实验表明，在 IID 和非 IID 设置下，DBFAT 始终优于其他基线。

Notes:

[PDF](https://arxiv.org/abs/2302.09479)

FedALA: Adaptive Local Aggregation for Personalized Federated Learning

• Authors: Jianqing Zhang; Yang Hua; Hao Wang; Tao Song; Zhengui Xue; Ruhui Ma; Haibing Guan

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26330

Abstract: A key challenge in federated learning (FL) is the statistical heterogeneity that impairs the generalization of the global model on each client. To address this, we propose a method Federated learning with Adaptive Local Aggregation (FedALA) by capturing the desired information in the global model for client models in personalized FL. The key component of FedALA is an Adaptive Local Aggregation (ALA) module, which can adaptively aggregate the downloaded global model and local model towards the local objective on each client to initialize the local model before training in each iteration. To evaluate the effectiveness of FedALA, we conduct extensive experiments with five benchmark datasets in computer vision and natural language processing domains. FedALA outperforms eleven state-of-the-art baselines by up to 3.27% in test accuracy. Furthermore, we also apply ALA module to other federated learning methods and achieve up to 24.19% improvement in test accuracy. Code is available at https://github.com/TsingZ0/FedALA.

abstractTranslation: 联邦学习 (FL) 的一个关键挑战是统计异质性，它会损害每个客户端的全局模型的泛化能力。为了解决这个问题，我们提出了一种采用自适应本地聚合（FedALA）的联邦学习方法，通过捕获个性化 FL 中客户端模型的全局模型中所需的信息。FedALA 的关键组件是自适应本地聚合（ALA）模块，它可以针对每个客户端上的本地目标自适应聚合下载的全局模型和本地模型，以便在每次迭代训练之前初始化本地模型。为了评估 FedALA 的有效性，我们使用计算机视觉和自然语言处理领域的五个基准数据集进行了广泛的实验。FedALA 的测试准确度比 11 个最先进的基线高出 3.27%。此外，我们还将ALA模块应用到其他联邦学习方法中，测试精度提高了24.19%。代码可在 https://github.com/TsingZ0/FedALA 获取。

Notes:

[PDF](https://arxiv.org/abs/2212.01197) 

[code](https://github.com/tsingz0/fedala)

Untargeted Attack against Federated Recommendation Systems via Poisonous Item Embeddings and the Defense

• Authors: Yang Yu; Qi Liu; Likang Wu; Runlong Yu; Sanshi Lei Yu; Zaixi Zhang

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/25611

Abstract: Federated recommendation (FedRec) can train personalized recommenders without collecting user data, but the decentralized nature makes it susceptible to poisoning attacks. Most previous studies focus on the targeted attack to promote certain items, while the untargeted attack that aims to degrade the overall performance of the FedRec system remains less explored. In fact, untargeted attacks can disrupt the user experience and bring severe financial loss to the service provider. However, existing untargeted attack methods are either inapplicable or ineffective against FedRec systems. In this paper, we delve into the untargeted attack and its defense for FedRec systems. (i) We propose ClusterAttack, a novel untargeted attack method. It uploads poisonous gradients that converge the item embeddings into several dense clusters, which make the recommender generate similar scores for these items in the same cluster and perturb the ranking order. (ii) We propose a uniformity-based defense mechanism (UNION) to protect FedRec systems from such attacks. We design a contrastive learning task that regularizes the item embeddings toward a uniform distribution. Then the server filters out these malicious gradients by estimating the uniformity of updated item embeddings. Experiments on two public datasets show that ClusterAttack can effectively degrade the performance of FedRec systems while circumventing many defense methods, and UNION can improve the resistance of the system against various untargeted attacks, including our ClusterAttack.

abstractTranslation: 联邦推荐（FedRec）可以在不收集用户数据的情况下训练个性化推荐器，但去中心化的性质使其容易受到中毒攻击。之前的大多数研究都集中在促进某些项目的有针对性的攻击上，而旨在降低 FedRec 系统整体性能的非针对性攻击仍然很少被探讨。事实上，无针对性的攻击可能会破坏用户体验并给服务提供商带来严重的经济损失。然而，现有的非针对性攻击方法对于 FedRec 系统要么不适用，要么无效。在本文中，我们深入研究了 FedRec 系统的非针对性攻击及其防御。(i)我们提出ClusterAttack，一种新颖的非目标攻击方法。它上传有毒梯度，将项目嵌入收敛到几个密集的集群中，这使得推荐器为同一集群中的这些项目生成相似的分数，并扰乱排名顺序。(ii) 我们提出了一种基于一致性的防御机制（UNION）来保护 FedRec 系统免受此类攻击。我们设计了一个对比学习任务，将项目嵌入规范化为均匀分布。然后，服务器通过估计更新项目嵌入的均匀性来过滤掉这些恶意梯度。在两个公共数据集上的实验表明，ClusterAttack可以有效降低FedRec系统的性能，同时规避许多防御方法，而UNION可以提高系统对各种非针对性攻击的抵抗力，包括我们的ClusterAttack。

Notes:

[PDF](https://arxiv.org/abs/2212.05399) 

[code](https://github.com/yflyl613/fedrec)

DeFL: Defending against Model Poisoning Attacks in Federated Learning via Critical Learning Periods Awareness

• Authors: Gang Yan; Hao Wang; Xu Yuan; Jian Li

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26271

Abstract: Federated learning (FL) is known to be susceptible to model poisoning attacks in which malicious clients hamper the accuracy of the global model by sending manipulated model updates to the central server during the FL training process. Existing defenses mainly focus on Byzantine-robust FL aggregations, and largely ignore the impact of the underlying deep neural network (DNN) that is used to FL training. Inspired by recent findings on critical learning periods (CLP) in DNNs, where small gradient errors have irrecoverable impact on the final model accuracy, we propose a new defense, called a CLP-aware defense against poisoning of FL (DeFL). The key idea of DeFL is to measure fine-grained differences between DNN model updates via an easy-to-compute federated gradient norm vector (FGNV) metric. Using FGNV, DeFL simultaneously detects malicious clients and identifies CLP, which in turn is leveraged to guide the adaptive removal of detected malicious clients from aggregation. As a result, DeFL not only mitigates model poisoning attacks on the global model but also is robust to detection errors. Our extensive experiments on three benchmark datasets demonstrate that DeFL produces significant performance gain over conventional defenses against state-of-the-art model poisoning attacks.

abstractTranslation: 众所周知，联邦学习 (FL) 容易受到模型中毒攻击，其中恶意客户端通过在 FL 训练过程中向中央服务器发送受操纵的模型更新来妨碍全局模型的准确性。现有的防御主要集中在拜占庭鲁棒的 FL 聚合上，很大程度上忽略了用于 FL 训练的底层深度神经网络 (DNN) 的影响。受最近 DNN 关键学习周期 (CLP) 发现的启发，其中小的梯度误差会对最终模型的准确性产生不可恢复的影响，我们提出了一种新的防御措施，称为 CLP 感知防御 FL 中毒 (DeFL)。DeFL 的关键思想是通过易于计算的联邦梯度范数向量 (FGNV) 指标来测量 DNN 模型更新之间的细粒度差异。使用 FGNV，DeFL 可同时检测恶意客户端并识别 CLP，进而利用 CLP 来指导从聚合中自适应删除检测到的恶意客户端。因此，DeFL 不仅可以减轻对全局模型的模型中毒攻击，而且对于检测错误也具有鲁棒性。我们对三个基准数据集进行的广泛实验表明，与针对最先进模型中毒攻击的传统防御相比，DeFL 具有显着的性能提升。

Federated Generative Model on Multi-Source Heterogeneous Data in IoT

• Authors: Zuobin Xiong; Wei Li; Zhipeng Cai

• Journal: Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26252

Abstract: The study of generative models is a promising branch of deep learning techniques, which has been successfully applied to different scenarios, such as Artificial Intelligence and the Internet of Things. While in most of the existing works, the generative models are realized as a centralized structure, raising the threats of security and privacy and the overburden of communication costs. Rare efforts have been committed to investigating distributed generative models, especially when the training data comes from multiple heterogeneous sources under realistic IoT settings. In this paper, to handle this challenging problem, we design a federated generative model framework that can learn a powerful generator for the hierarchical IoT systems. Particularly, our generative model framework can solve the problem of distributed data generation on multi-source heterogeneous data in two scenarios, i.e., feature related scenario and label related scenario. In addition, in our federated generative models, we develop a synchronous and an asynchronous updating methods to satisfy different application requirements. Extensive experiments on a simulated dataset and multiple real datasets are conducted to evaluate the data generation performance of our proposed generative models through comparison with the state-of-the-arts.

abstractTranslation: 生成模型的研究是深度学习技术的一个有前途的分支，已成功应用于人工智能和物联网等不同场景。而在大多数现有工作中，生成模型都是作为中心化结构实现的，这增加了安全和隐私的威胁以及通信成本的负担。人们很少致力于研究分布式生成模型，特别是当训练数据来自现实物联网设置下的多个异构源时。在本文中，为了解决这个具有挑战性的问题，我们设计了一个联邦生成模型框架，可以为分层物联网系统学习强大的生成器。特别是，我们的生成模型框架可以解决特征相关场景和标签相关场景两种场景下多源异构数据的分布式数据生成问题。此外，在我们的联邦生成模型中，我们开发了同步和异步更新方法来满足不同的应用需求。对模拟数据集和多个真实数据集进行了广泛的实验，通过与最先进的技术进行比较来评估我们提出的生成模型的数据生成性能。

CDMA: A Practical Cross-Device Federated Learning Algorithm for General Minimax Problems

• Authors: Jiahao Xie; Chao Zhang; Zebang Shen; Weijie Liu; Hui Qian

• Journal: Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26246

Abstract: Minimax problems arise in a wide range of important applications including robust adversarial learning and Generative Adversarial Network (GAN) training. Recently, algorithms for minimax problems in the Federated Learning (FL) paradigm have received considerable interest. Existing federated algorithms for general minimax problems require the full aggregation (i.e., aggregation of local model information from all clients) in each training round. Thus, they are inapplicable to an important setting of FL known as the cross-device setting, which involves numerous unreliable mobile/IoT devices. In this paper, we develop the first practical algorithm named CDMA for general minimax problems in the cross-device FL setting. CDMA is based on a Start-Immediately-With-Enough-Responses mechanism, in which the server first signals a subset of clients to perform local computation and then starts to aggregate the local results reported by clients once it receives responses from enough clients in each round. With this mechanism, CDMA is resilient to the low client availability. In addition, CDMA is incorporated with a lightweight global correction in the local update steps of clients, which mitigates the impact of slow network connections. We establish theoretical guarantees of CDMA under different choices of hyperparameters and conduct experiments on AUC maximization, robust adversarial network training, and GAN training tasks. Theoretical and experimental results demonstrate the efficiency of CDMA.

abstractTranslation: 极小极大问题出现在广泛的重要应用中，包括鲁棒的对抗性学习和生成对抗性网络（GAN）训练。最近，联邦学习（FL）范式中的极小极大问题的算法引起了相当大的兴趣。针对一般极小极大问题的现有联邦算法需要在每轮训练中进行完全聚合（即来自所有客户端的本地模型信息的聚合）。因此，它们不适用于 FL 的重要设置（称为跨设备设置），该设置涉及大量不可靠的移动/物联网设备。在本文中，我们针对跨设备 FL 设置中的一般极小极大问题开发了第一个名为 CDMA 的实用算法。CDMA 基于“立即启动足够的响应”机制，其中服务器首先向客户端的子集发出信号以执行本地计算，然后一旦收到每个客户端中足够多的客户端的响应，就开始聚合客户端报告的本地结果。圆形的。通过这种机制，CDMA 能够适应低客户端可用性。此外，CDMA在客户端本地更新步骤中加入了轻量级全局校正，从而减轻了网络连接速度慢的影响。我们在不同超参数选择下建立了 CDMA 的理论保证，并对 AUC 最大化、鲁棒对抗网络训练和 GAN 训练任务进行了实验。理论和实验结果证明了CDMA 的效率。

Notes:

[PDF](https://arxiv.org/abs/2105.14216) 

[CODE](https://github.com/xjiajiahao/federated-minimax)

Bayesian Federated Neural Matching That Completes Full Information

• Authors: Peng Xiao; Samuel Cheng

• Journal: Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26245

Abstract: Federated learning is a contemporary machine learning paradigm where locally trained models are distilled into a global model. Due to the intrinsic permutation invariance of neural networks, Probabilistic Federated Neural Matching (PFNM) employs a Bayesian nonparametric framework in the generation process of local neurons, and then creates a linear sum assignment formulation in each alternative optimization iteration. But according to our theoretical analysis, the optimization iteration in PFNM omits global information from existing. In this study, we propose a novel approach that overcomes this flaw by introducing a Kullback-Leibler divergence penalty at each iteration. The effectiveness of our approach is demonstrated by experiments on both image classification and semantic segmentation tasks.

abstractTranslation: 联邦学习是一种当代机器学习范式，其中本地训练的模型被提炼为全局模型。由于神经网络固有的排列不变性，概率联邦神经匹配（PFNM）在局部神经元的生成过程中采用贝叶斯非参数框架，然后在每次替代优化迭代中创建线性和分配公式。但根据我们的理论分析，PFNM 中的优化迭代忽略了现有的全局信息。在这项研究中，我们提出了一种新颖的方法，通过在每次迭代中引入 Kullback-Leibler 散度惩罚来克服这一缺陷。我们的方法的有效性通过图像分类和语义分割任务的实验得到了证明。

FedNP: Towards Non-IID Federated Learning via Federated Neural Propagation

• Authors: Xueyang Wu; Hengguan Huang; Youlong Ding; Hao Wang; Ye Wang; Qian Xu

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26237

Abstract: Traditional federated learning (FL) algorithms, such as FedAvg, fail to handle non-i.i.d data because they learn a global model by simply averaging biased local models that are trained on non-i.i.d local data, therefore failing to model the global data distribution. In this paper, we present a novel Bayesian FL algorithm that successfully handles such a non-i.i.d FL setting by enhancing the local training task with an auxiliary task that explicitly estimates the global data distribution. One key challenge in estimating the global data distribution is that the data are partitioned in FL, and therefore the ground-truth global data distribution is inaccessible. To address this challenge, we propose an expectation-propagation-inspired probabilistic neural network, dubbed federated neural propagation (FedNP), which efficiently estimates the global data distribution given non-i.i.d data partitions. Our algorithm is sampling-free and end-to-end differentiable, can be applied with any conventional FL frameworks and learns richer global data representation. Experiments on both image classification tasks with synthetic non-i.i.d image data partitions and real-world non-i.i.d speech recognition tasks demonstrate that our framework effectively alleviates the performance deterioration caused by non-i.i.d data.

abstractTranslation: 传统的联邦学习 (FL) 算法（例如 FedAvg）无法处理非独立同分布数据，因为它们通过简单地平均在非独立同分布本地数据上训练的有偏差的本地模型来学习全局模型，因此无法对全局数据分布进行建模。在本文中，我们提出了一种新颖的贝叶斯 FL 算法，该算法通过使用显式估计全局数据分布的辅助任务来增强局部训练任务，成功地处理了这种非独立独立的 FL 设置。估计全局数据分布的一个关键挑战是数据在 FL 中进行分区，因此无法访问真实的全局数据分布。为了应对这一挑战，我们提出了一种受期望传播启发的概率神经网络，称为联邦神经传播（FedNP），它可以有效地估计给定非独立同分布数据分区的全局数据分布。我们的算法是免采样且端到端可微分的，可以应用于任何传统的 FL 框架并学习更丰富的全局数据表示。对具有合成非独立同分布图像数据分区的图像分类任务和现实世界非独立同分布语音识别任务的实验表明，我们的框架有效减轻了非独立同分布数据引起的性能恶化。

Notes:

[CODE](https://github.com/CodePothunter/fednp) 

[VIDEO](https://www.youtube.com/watch?v=3XM_NNvXCBo) 

[SUPP](https://github.com/CodePothunter/fednp/blob/main/appendix.pdf)

Faster Adaptive Federated Learning

• Authors: Xidong Wu; Feihu Huang; Zhengmian Hu; Heng Huang

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26235

Abstract:

Federated learning has attracted increasing attention with the emergence of distributed data. While extensive federated learning algorithms have been proposed for the non-convex distributed problem, the federated learning in practice still faces numerous challenges, such as the large training iterations to converge since the sizes of models and datasets keep increasing, and the lack of adaptivity by SGD-based model updates. Meanwhile, the study of adaptive methods in federated learning is scarce and existing works either lack a complete theoretical convergence guarantee or have slow sample complexity. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on the momentum-based variance reduced technique in cross-silo FL. We first explore how to design the adaptive algorithm in the FL setting. By providing a counter-example, we prove that a simple combination of FL and adaptive methods could lead to divergence. More importantly, we provide a convergence analysis for our method and prove that our algorithm is the first adaptive FL algorithm to reach the best-known samples O(epsilon(-3)) and O(epsilon(-2)) communication rounds to find an epsilon-stationary point without large batches. The experimental results on the language modeling task and image classification task with heterogeneous data demonstrate the efficiency of our algorithms.

abstractTranslation: 随着分布式数据的出现，联邦学习越来越受到人们的关注。虽然针对非凸分布式问题提出了广泛的联邦学习算法，但联邦学习在实践中仍然面临着许多挑战，例如由于模型和数据集的大小不断增加而需要大量的训练迭代才能收敛，以及联邦学习缺乏适应性。基于 SGD 的模型更新。同时，联邦学习中自适应方法的研究很少，现有的工作要么缺乏完整的理论收敛保证，要么样本复杂度较低。在本文中，我们提出了一种基于跨筒仓 FL 中基于动量的方差减少技术的高效自适应算法（即 FAFED）。我们首先探讨如何在 FL 设置中设计自适应算法。通过提供一个反例，我们证明了 FL 和自适应方法的简单组合可能会导致发散。更重要的是，我们为我们的方法提供了收敛分析，并证明我们的算法是第一个达到最著名样本 O(epsilon(-3)) 和 O(epsilon(-2)) 通信轮数的自适应 FL 算法，以找到没有大批量的 epsilon 驻点。在异构数据的语言建模任务和图像分类任务上的实验结果证明了我们算法的效率。

Notes:

[PDF](https://arxiv.org/abs/2212.00974)

FedGS: Federated Graph-Based Sampling with Arbitrary Client Availability

• Authors: Zheng Wang; Xiaoliang Fan; Jianzhong Qi; Haibing Jin; Peizhen Yang; Siqi Shen; Cheng Wang

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26223

Abstract: While federated learning has shown strong results in opti- mizing a machine learning model without direct access to the original data, its performance may be hindered by in- termittent client availability which slows down the conver- gence and biases the final learned model. There are significant challenges to achieve both stable and bias-free training un- der arbitrary client availability. To address these challenges, we propose a framework named Federated Graph-based Sam- pling (FEDGS), to stabilize the global model update and mitigate the long-term bias given arbitrary client availabil- ity simultaneously. First, we model the data correlations of clients with a Data-Distribution-Dependency Graph (3DG) that helps keep the sampled clients data apart from each other, which is theoretically shown to improve the approximation to the optimal model update. Second, constrained by the far- distance in data distribution of the sampled clients, we fur- ther minimize the variance of the numbers of times that the clients are sampled, to mitigate long-term bias. To validate the effectiveness of FEDGS, we conduct experiments on three datasets under a comprehensive set of seven client availability modes. Our experimental results confirm FEDGS’s advantage in both enabling a fair client-sampling scheme and improving the model performance under arbitrary client availability. Our code is available at https://github.com/WwZzz/FedGS.

abstractTranslation: 虽然联邦学习在无需直接访问原始数据的情况下优化机器学习模型方面显示出强大的成果，但其性能可能会受到间歇性客户端可用性的阻碍，这会减慢收敛速度并使最终学习模型产生偏差。在任意客户端可用性下实现稳定且无偏差的训练存在重大挑战。为了应对这些挑战，我们提出了一个名为基于联邦图的采样（FEDGS）的框架，以稳定全局模型更新并减轻同时给定任意客户端可用性的长期偏差。首先，我们使用数据分布依赖图（3DG）对客户端的数据相关性进行建模，该图有助于将采样的客户端数据彼此分开，理论上表明这可以提高最佳模型更新的近似度。其次，受样本客户数据分布距离较远的限制，我们进一步最小化样本样本次数的方差，以减轻长期偏差。为了验证 FEDGS 的有效性，我们在七种客户端可用性模式下对三个数据集进行了实验。我们的实验结果证实了 FEDGS 在实现公平的客户端采样方案和提高任意客户端可用性下的模型性能方面的优势。我们的代码可在 https://github.com/WwZzz/FedGS 获取。

Notes:

[PDF](https://arxiv.org/abs/2211.13975) 

[code](https://github.com/wwzzz/fedgs)

Beyond ADMM: A Unified Client-Variance-Reduced Adaptive Federated Learning Framework

• Authors: Shuai Wang; Yanqing Xu; Zhiguo Wang; Tsung-Hui Chang; Tony Q. S. Quek; Defeng Sun

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26212

Abstract: As a novel distributed learning paradigm, federated learning (FL) faces serious challenges in dealing with massive clients with heterogeneous data distribution and computation and communication resources. Various client-variance-reduction schemes and client sampling strategies have been respectively introduced to improve the robustness of FL. Among others, primal-dual algorithms such as the alternating direction of method multipliers (ADMM) have been found being resilient to data distribution and outperform most of the primal-only FL algorithms. However, the reason behind remains a mystery still. In this paper, we firstly reveal the fact that the federated ADMM is essentially a client-variance-reduced algorithm. While this explains the inherent robustness of federated ADMM, the vanilla version of it lacks the ability to be adaptive to the degree of client heterogeneity. Besides, the global model at the server under client sampling is biased which slows down the practical convergence. To go beyond ADMM, we propose a novel primal-dual FL algorithm, termed FedVRA, that allows one to adaptively control the variance-reduction level and biasness of the global model. In addition, FedVRA unifies several representative FL algorithms in the sense that they are either special instances of FedVRA or are close to it. Extensions of FedVRA to semi/un-supervised learning are also presented. Experiments based on (semi-)supervised image classification tasks demonstrate superiority of FedVRA over the existing schemes in learning scenarios with massive heterogeneous clients and client sampling.

abstractTranslation: 作为一种新颖的分布式学习范式，联邦学习（FL）在处理具有异构数据分布以及计算和通信资源的海量客户端时面临着严峻的挑战。分别引入了各种客户端方差减少方案和客户端采样策略来提高 FL 的鲁棒性。其中，方法乘数交替方向 (ADMM) 等原始对偶算法被发现对数据分布具有弹性，并且优于大多数仅原始 FL 算法。然而，背后的原因仍然是个谜。在本文中，我们首先揭示了联邦 ADMM 本质上是一种客户端方差减少算法。虽然这解释了联邦 ADMM 固有的稳健性，但它的普通版本缺乏适应客户端异构程度的能力。此外，客户端采样下服务器端的全局模型存在偏差，这会减慢实际收敛速度。为了超越 ADMM，我们提出了一种新颖的原始对偶 FL 算法，称为 FedVRA，它允许人们自适应地控制全局模型的方差减少水平和偏差。此外，FedVRA 统一了几种具有代表性的 FL 算法，因为它们要么是 FedVRA 的特殊实例，要么与其接近。还介绍了 FedVRA 对半监督/无监督学习的扩展。基于（半）监督图像分类任务的实验证明了 FedVRA 在具有大量异构客户端和客户端采样的学习场景中相对于现有方案的优越性。

Notes:

[pdf](https://arxiv.org/abs/2212.01519)

FedABC: Targeting Fair Competition in Personalized Federated Learning

• Authors: Dui Wang; Li Shen; Yong Luo; Han Hu; Kehua Su; Yonggang Wen; Dacheng Tao

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26203

Abstract: Federated learning aims to collaboratively train models without accessing their client's local private data. The data may be Non-IID for different clients and thus resulting in poor performance. Recently, personalized federated learning (PFL) has achieved great success in handling Non-IID data by enforcing regularization in local optimization or improving the model aggregation scheme on the server. However, most of the PFL approaches do not take into account the unfair competition issue caused by the imbalanced data distribution and lack of positive samples for some classes in each client. To address this issue, we propose a novel and generic PFL framework termed Federated Averaging via Binary Classification, dubbed FedABC. In particular, we adopt the ``one-vs-all'' training strategy in each client to alleviate the unfair competition between classes by constructing a personalized binary classification problem for each class. This may aggravate the class imbalance challenge and thus a novel personalized binary classification loss that incorporates both the under-sampling and hard sample mining strategies is designed. Extensive experiments are conducted on two popular datasets under different settings, and the results demonstrate that our FedABC can significantly outperform the existing counterparts.

abstractTranslation: 联邦学习旨在协作训练模型，而无需访问客户的本地私有数据。对于不同的客户端，数据可能是非独立同分布的，从而导致性能不佳。最近，个性化联邦学习（PFL）通过在本地优化中强制正则化或改进服务器上的模型聚合方案，在处理非独立同分布数据方面取得了巨大成功。然而，大多数PFL方法没有考虑到由于数据分布不平衡以及每个客户端中某些类别缺乏正样本而导致的不公平竞争问题。为了解决这个问题，我们提出了一种新颖且通用的 PFL 框架，称为通过二元分类进行联邦平均，称为 FedABC。特别是，我们在每个客户端中采用“一对一”训练策略，通过为每个类别构建个性化的二元分类问题来缓解类别之间的不公平竞争。这可能会加剧类别不平衡的挑战，因此设计了一种结合了欠采样和硬样本挖掘策略的新颖的个性化二元分类损失。在不同设置下对两个流行的数据集进行了广泛的实验，结果表明我们的 FedABC 可以显着优于现有的同行。

Federated Learning on Non-IID Graphs via Structural Knowledge Sharing

• Authors: Yue Tan; Yixin Liu; Guodong Long; Jing Jiang; Qinghua Lu; Chengqi Zhang

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26187

Abstract:  Graph neural networks (GNNs) have shown their superiority in modeling graph data. Owing to the advantages of federated learning, federated graph learning (FGL) enables clients to train strong GNN models in a distributed manner without sharing their private data. A core challenge in federated systems is the non-IID problem, which also widely exists in real-world graph data. For example, local data of clients may come from diverse datasets or even domains, e.g., social networks and molecules, increasing the difficulty for FGL methods to capture commonly shared knowledge and learn a generalized encoder. From real-world graph datasets, we observe that some structural properties are shared by various domains, presenting great potential for sharing structural knowledge in FGL. Inspired by this, we propose FedStar, an FGL framework that extracts and shares the common underlying structure information for inter-graph federated learning tasks. To explicitly extract the structure information rather than encoding them along with the node features, we define structure embeddings and encode them with an independent structure encoder. Then, the structure encoder is shared across clients while the feature-based knowledge is learned in a personalized way, making FedStar capable of capturing more structure-based domain-invariant information and avoiding feature misalignment issues. We perform extensive experiments over both cross-dataset and cross-domain non-IID FGL settings, demonstrating the superiority of FedStar.

abstractTranslation:  图神经网络（GNN）在图数据建模方面表现出了其优越性。由于联邦学习的优势，联邦图学习（FGL）使客户能够以分布式方式训练强大的 GNN 模型，而无需共享其私有数据。联邦系统的核心挑战是非独立同分布问题，该问题也广泛存在于现实世界的图数据中。例如，客户端的本地数据可能来自不同的数据集甚至领域，例如社交网络和分子，这增加了 FGL 方法捕获共同共享知识和学习通用编码器的难度。从现实世界的图数据集中，我们观察到一些结构属性是由不同领域共享的，这为 FGL 中共享结构知识展现了巨大的潜力。受此启发，我们提出了 FedStar，一个 FGL 框架，它提取并共享图间联邦学习任务的公共底层结构信息。为了显式提取结构信息而不是将它们与节点特征一起编码，我们定义了结构嵌入并使用独立的结构编码器对其进行编码。然后，结构编码器在客户端之间共享，同时以个性化方式学习基于特征的知识，使 FedStar 能够捕获更多基于结构的域不变信息并避免特征错位问题。我们对跨数据集和跨域非 IID FGL 设置进行了广泛的实验，证明了 FedStar 的优越性。

Notes:

[PDF](https://arxiv.org/abs/2211.13009) 

[code](https://github.com/yuetan031/fedstar)

DPAUC: Differentially Private AUC Computation in Federated Learning

• Authors: Jiankai Sun; Xin Yang; Yuanshun Yao; Junyuan Xie; Di Wu; Chong Wang

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26770

Abstract:  Federated learning (FL) has gained significant attention recently as a privacy-enhancing tool to jointly train a machine learning model by multiple participants. The prior work on FL has mostly studied how to protect label privacy during model training. However, model evaluation in FL might also lead to the potential leakage of private label information. In this work, we propose an evaluation algorithm that can accurately compute the widely used AUC (area under the curve) metric when using the label differential privacy (DP) in FL. Through extensive experiments, we show our algorithms can compute accurate AUCs compared to the ground truth. The code is available at https://github.com/bytedance/fedlearner/tree/master/example/privacy/DPAUC

abstractTranslation: 联邦学习（FL）作为一种由多个参与者联邦训练机器学习模型的隐私增强工具，最近受到了极大的关注。FL 之前的工作主要研究如何在模型训练过程中保护标签隐私。然而，FL 中的模型评估也可能导致私有标签信息的潜在泄漏。在这项工作中，我们提出了一种评估算法，可以在 FL 中使用标签差分隐私（DP）时准确计算广泛使用的 AUC（曲线下面积）指标。通过大量的实验，我们证明我们的算法可以计算出与真实情况相比准确的 AUC。该代码位于https://github.com/bytedance/fedlearner/tree/master/example/privacy/DPAUC

Notes:

[PDF](https://arxiv.org/abs/2208.12294) 

[code](https://github.com/bytedance/fedlearner)

项目链接: https://github.com/youngfish42/Awesome-FL

作者: 白小鱼（上海交通大学计算机系博士生）

END

往期推荐