文章速览 | 联邦学习 x ICML'2023（下）

本文是由白小鱼博主整理的ICML 2023会议中，与联邦学习相关的论文合集及摘要翻译，关于上一章的整理，请读者们详阅文章速览 | 联邦学习 x ICML'2023（上）。

Revisiting Weighted Aggregation in Federated Learning with Neural Networks

Authors: Zexi Li; Tao Lin; Xinyi Shang; Chao Wu

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/li23s.html

Abstract: In federated learning (FL), weighted aggregation of local models is conducted to generate a global model, and the aggregation weights are normalized (the sum of weights is 1) and proportional to the local data sizes. In this paper, we revisit the weighted aggregation process and gain new insights into the training dynamics of FL. First, we find that the sum of weights can be smaller than 1, causing global weight shrinking effect (analogous to weight decay) and improving generalization. We explore how the optimal shrinking factor is affected by clients’ data heterogeneity and local epochs. Second, we dive into the relative aggregation weights among clients to depict the clients’ importance. We develop client coherence to study the learning dynamics and find a critical point that exists. Before entering the critical point, more coherent clients play more essential roles in generalization. Based on the above insights, we propose an effective method for Federated Learning with Learnable Aggregation Weights, named as FedLAW. Extensive experiments verify that our method can improve the generalization of the global model by a large margin on different datasets and models.

ISSN: 2640-3498 abstractTranslation: 在联邦学习（FL）中，对局部模型进行加权聚合以生成全局模型，聚合权重被归一化（权重之和为1）并与局部数据大小成正比。在本文中，我们重新审视了加权聚合过程，并对 FL 的训练动态有了新的见解。首先，我们发现权重之和可以小于1，从而产生全局权重收缩效应（类似于权重衰减）并提高泛化能力。我们探讨了最佳收缩因子如何受到客户数据异质性和本地时代的影响。其次，我们深入研究客户之间的相对聚合权重来描述客户的重要性。我们培养客户的一致性来研究学习动态并找到存在的关键点。在进入临界点之前，更一致的客户在泛化中发挥着更重要的作用。基于上述见解，我们提出了一种有效的具有可学习聚合权重的联邦学习方法，命名为FedLAW。大量的实验验证了我们的方法可以在不同的数据集和模型上大幅提高全局模型的泛化能力。

Notes:

PUB (https://openreview.net/forum?id=FuDAjnWhrQ)

PDF (https://arxiv.org/abs/2302.10911)

CODE (https://github.com/zexilee/icml-2023-fedlaw)

Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression: Fast Convergence and Partial Participation

Authors: Xiaoyun Li; Ping Li

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/li23o.html

Abstract: In practical federated learning (FL) systems, the communication cost between the clients and the central server can often be a bottleneck. In this paper, we focus on biased gradient compression in non-convex FL problems. In the classical distributed learning, the method of error feedback (EF) is a common technique to remedy the downsides of biased gradient compression, but the performance of EF in FL still lacks systematic investigation. In this work, we study a compressed FL scheme with error feedback, named Fed-EF, with two variants depending on the global model optimizer. While directly applying biased compression in FL leads to poor convergence, we show that Fed-EF is able to match the convergence rate of the full-precision FL counterpart with a linear speedup w.r.t. the number of clients. Experiments verify that Fed-EF achieves the same performance as the full-precision FL approach, at the substantially reduced communication cost. Moreover, we develop a new analysis of the EF under partial participation (PP), an important scenario in FL. Under PP, the convergence rate of Fed-EF exhibits an extra slow-down factor due to a so-called “stale error compensation” effect, which is also justified in our experiments. Our results provide insights on a theoretical limitation of EF, and possible directions for improvements.

ISSN: 2640-3498 abstractTranslation: 在实际的联邦学习（FL）系统中，客户端和中央服务器之间的通信成本通常是一个瓶颈。在本文中，我们重点研究非凸 FL 问题中的偏置梯度压缩。在经典的分布式学习中，误差反馈（EF）方法是弥补有偏梯度压缩缺点的常用技术，但 EF 在 FL 中的性能仍然缺乏系统的研究。在这项工作中，我们研究了一种带有错误反馈的压缩 FL 方案，名为 Fed-EF，具有两种取决于全局模型优化器的变体。虽然在 FL 中直接应用偏置压缩会导致收敛性较差，但我们表明 Fed-EF 能够以线性加速比与全精度 FL 对应的收敛速度相匹配。客户数量。实验验证了 Fed-EF 能够以大幅降低的通信成本实现与全精度 FL 方法相同的性能。此外，我们对部分参与（PP）下的 EF 进行了新的分析，这是 FL 的一个重要场景。在 PP 下，由于所谓的“过时误差补偿”效应，Fed-EF 的收敛速度表现出额外的减速因素，这在我们的实验中也是合理的。我们的结果提供了有关 EF 理论局限性的见解以及可能的改进方向。

Notes:

PUB (https://openreview.net/forum?id=wbs1fKLfOe)

PDF (https://arxiv.org/abs/2211.14292)

Federated Adversarial Learning: A Framework with Convergence Analysis

Authors: Xiaoxiao Li; Zhao Song; Jiaming Yang

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/li23z.html

Abstract: Federated learning (FL) is a trending training paradigm to utilize decentralized training data. FL allows clients to update model parameters locally for several epochs, then share them to a global model for aggregation. This training paradigm with multi-local step updating before aggregation exposes unique vulnerabilities to adversarial attacks. Adversarial training is a popular and effective method to improve the robustness of networks against adversaries. In this work, we formulate a general form of federated adversarial learning (FAL) that is adapted from adversarial learning in the centralized setting. On the client side of FL training, FAL has an inner loop to generate adversarial samples for adversarial training and an outer loop to update local model parameters. On the server side, FAL aggregates local model updates and broadcast the aggregated model. We design a global robust training loss and formulate FAL training as a min-max optimization problem. Unlike the convergence analysis in classical centralized training that relies on the gradient direction, it is significantly harder to analyze the convergence in FAL for three reasons: 1) the complexity of min-max optimization, 2) model not updating in the gradient direction due to the multi-local updates on the client-side before aggregation and 3) inter-client heterogeneity. We address these challenges by using appropriate gradient approximation and coupling techniques and present the convergence analysis in the over-parameterized regime. Our main result theoretically shows that the minimum loss under our algorithm can converge to ϵϵ\epsilon small with chosen learning rate and communication rounds. It is noteworthy that our analysis is feasible for non-IID clients.

ISSN: 2640-3498 abstractTranslation: 联邦学习（FL）是一种利用去中心化训练数据的趋势训练范式。FL 允许客户端在本地更新几个时期的模型参数，然后将它们共享到全局模型进行聚合。这种在聚合之前进行多本地步骤更新的训练范例暴露了对抗性攻击的独特漏洞。对抗性训练是一种流行且有效的方法，可以提高网络对抗对手的鲁棒性。在这项工作中，我们制定了一种联邦对抗性学习（FAL）的通用形式，它改编自集中式环境中的对抗性学习。在 FL 训练的客户端，FAL 有一个内循环来生成用于对抗训练的对抗样本，还有一个外循环来更新本地模型参数。在服务器端，FAL聚合本地模型更新并广播聚合模型。我们设计了全局鲁棒训练损失，并将 FAL 训练制定为最小-最大优化问题。与依赖梯度方向的经典集中训练中的收敛分析不同，FAL 中的收敛分析要困难得多，原因有以下三个：1）最小-最大优化的复杂性，2）由于以下原因，模型不会在梯度方向上更新：聚合前客户端的多本地更新以及 3) 客户端间的异构性。我们通过使用适当的梯度近似和耦合技术来解决这些挑战，并在过度参数化的情况下进行收敛分析。我们的主要结果从理论上表明，在选择学习率和通信轮次的情况下，我们算法下的最小损失可以收敛到较小的 εε\epsilon。值得注意的是，我们的分析对于非 IID 客户是可行的。

Notes:

FedVS: Straggler-Resilient and Privacy-Preserving Vertical Federated Learning for Split Models

Authors: Songze Li; Duanyi Yao; Jin Liu

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/li23an.html

Abstract: In a vertical federated learning (VFL) system consisting of a central server and many distributed clients, the training data are vertically partitioned such that different features are privately stored on different clients. The problem of split VFL is to train a model split between the server and the clients. This paper aims to address two major challenges in split VFL: 1) performance degradation due to straggling clients during training; and 2) data and model privacy leakage from clients’ uploaded data embeddings. We propose FedVS to simultaneously address these two challenges. The key idea of FedVS is to design secret sharing schemes for the local data and models, such that information-theoretical privacy against colluding clients and curious server is guaranteed, and the aggregation of all clients’ embeddings is reconstructed losslessly, via decrypting computation shares from the non-straggling clients. Extensive experiments on various types of VFL datasets (including tabular, CV, and multi-view) demonstrate the universal advantages of FedVS in straggler mitigation and privacy protection over baseline protocols.

ISSN: 2640-3498 abstractTranslation: 在由中央服务器和许多分布式客户端组成的纵向联邦学习（VFL）系统中，训练数据被纵向分区，以便不同的特征私有地存储在不同的客户端上。分割 VFL 的问题是训练服务器和客户端之间的模型分割。本文旨在解决分割 VFL 中的两个主要挑战：1）由于训练过程中客户端落后而导致性能下降；2）客户上传的数据嵌入导致数据和模型隐私泄露。我们建议 FedVS 同时应对这两个挑战。FedVS 的关键思想是为本地数据和模型设计秘密共享方案，从而保证针对合谋客户端和好奇服务器的信息论隐私，并且通过解密计算共享来无损地重建所有客户端嵌入的聚合。不落后的客户。对各种类型的 VFL 数据集（包括表格、CV 和多视图）进行的广泛实验证明了 FedVS 在落后者缓解和隐私保护方面相对于基线协议的普遍优势。

Notes:

PUB (https://openreview.net/forum?id=7aqVcrXjxa)

PDF (https://arxiv.org/abs/2304.13407)

Adversarial Collaborative Learning on Non-IID Features

Authors: Qinbin Li; Bingsheng He; Dawn Song

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/li23j.html

Abstract: Federated Learning (FL) has been a popular approach to enable collaborative learning on multiple parties without exchanging raw data. However, the model performance of FL may degrade a lot due to non-IID data. While many FL algorithms focus on non-IID labels, FL on non-IID features has largely been overlooked. Different from typical FL approaches, the paper proposes a new learning concept called ADCOL (Adversarial Collaborative Learning) for non-IID features. Instead of adopting the widely used model-averaging scheme, ADCOL conducts training in an adversarial way: the server aims to train a discriminator to distinguish the representations of the parties, while the parties aim to generate a common representation distribution. Our experiments show that ADCOL achieves better performance than state-of-the-art FL algorithms on non-IID features.

ISSN: 2640-3498 abstractTranslation: 联邦学习 (FL) 是一种流行的方法，可以在不交换原始数据的情况下实现多方协作学习。然而，由于非独立同分布数据，FL 的模型性能可能会下降很多。虽然许多 FL 算法专注于非 IID 标签，但非 IID 特征上的 FL 在很大程度上被忽视了。与典型的 FL 方法不同，本文针对非 IID 特征提出了一种新的学习概念，称为 ADCOL（对抗性协作学习）。ADCOL 没有采用广泛使用的模型平均方案，而是以对抗方式进行训练：服务器的目标是训练鉴别器来区分各方的表示，而各方的目标是生成共同的表示分布。我们的实验表明，ADCOL 在非 IID 特征上实现了比最先进的 FL 算法更好的性能。

Notes:

PUB (https://openreview.net/forum?id=DVF7gEQQf7)

Cocktail Party Attack: Breaking Aggregation-Based Privacy in Federated Learning Using Independent Component Analysis

Authors: Sanjay Kariyappa; Chuan Guo; Kiwan Maeng; Wenjie Xiong; G. Edward Suh; Moinuddin K. Qureshi; Hsien-Hsin S. Lee

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/kariyappa23a.html

Abstract: Federated learning (FL) aims to perform privacy-preserving machine learning on distributed data held by multiple data owners. To this end, FL requires the data owners to perform training locally and share the gradients or weight updates (instead of the private inputs) with the central server, which are then securely aggregated over multiple data owners. Although aggregation by itself does not offer provable privacy protection, prior work suggested that if the batch size is sufficiently large the aggregation may be secure enough. In this paper, we propose the Cocktail Party Attack (CPA) that, contrary to prior belief, is able to recover the private inputs from gradients/weight updates aggregated over as many as 1024 samples. CPA leverages the crucial insight that aggregate gradients from a fully connected (FC) layer is a linear combination of its inputs, which allows us to frame gradient inversion as a blind source separation (BSS) problem. We adapt independent component analysis (ICA)—a classic solution to the BSS problem—to recover private inputs for FC and convolutional networks, and show that CPA significantly outperforms prior gradient inversion attacks, scales to ImageNet-sized inputs, and works on large batch sizes of up to 1024.

ISSN: 2640-3498 abstractTranslation: 联邦学习（FL）旨在对多个数据所有者持有的分布式数据执行保护隐私的机器学习。为此，FL 要求数据所有者在本地执行训练，并与中央服务器共享梯度或权重更新（而不是私有输入），然后将其安全地聚合到多个数据所有者。尽管聚合本身并不能提供可证明的隐私保护，但先前的工作表明，如果批量大小足够大，则聚合可能足够安全。在本文中，我们提出了鸡尾酒会攻击（CPA），与之前的看法相反，它能够从多达 1024 个样本聚合的梯度/权重更新中恢复私有输入。CPA 利用了一个重要的见解，即来自全连接 (FC) 层的聚合梯度是其输入的线性组合，这使我们能够将梯度反演视为盲源分离 (BSS) 问题。我们采用独立分量分析 (ICA)（BSS 问题的经典解决方案）来恢复 FC 和卷积网络的私有输入，并表明 CPA 显着优于先前的梯度反转攻击、可扩展到 ImageNet 大小的输入，并且适用于大批量最大尺寸为 1024。

Notes:

PUB (https://openreview.net/forum?id=Ai1TyAjZt9)

PDF (https://arxiv.org/abs/2209.05578)

One-Shot Federated Conformal Prediction

Authors: Pierre Humbert; Batiste Le Bars; Aurélien Bellet; Sylvain Arlot

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/humbert23a.html

Abstract: In this paper, we present a Conformal Prediction method that computes prediction sets in a one-shot Federated Learning (FL) setting. More specifically, we introduce a novel quantile-of-quantiles estimator and prove that for any distribution, it is possible to compute prediction sets with desired coverage in only one round of communication. To mitigate privacy issues, we also describe a locally differentially private version of our estimator. Finally, over a wide range of experiments, we show that our method returns prediction sets with coverage and length very similar to those obtained in a centralized setting. These results demonstrate that our method is well-suited for one-shot Federated Learning.

ISSN: 2640-3498 abstractTranslation: 在本文中，我们提出了一种保形预测方法，该方法在一次性联邦学习（FL）设置中计算预测集。更具体地说，我们引入了一种新颖的分位数估计器，并证明对于任何分布，都可以仅在一轮通信中计算出具有所需覆盖范围的预测集。为了缓解隐私问题，我们还描述了估计器的局部差分隐私版本。最后，通过广泛的实验，我们表明我们的方法返回的预测集的覆盖范围和长度与集中设置中获得的预测集非常相似。这些结果表明我们的方法非常适合一次性联邦学习。

Notes:

PUB (https://openreview.net/forum?id=SZJGIWe1Ag)

PDF (https://arxiv.org/abs/2302.06322)

CODE (https://github.com/pierreHmbt/FedCP-QQ)

Federated Linear Contextual Bandits with User-level Differential Privacy

Authors: Ruiquan Huang; Huanyu Zhang; Luca Melis; Milan Shen; Meisam Hejazinia; Jing Yang

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/huang23q.html

Abstract: This paper studies federated linear contextual bandits under the notion of user-level differential privacy (DP). We first introduce a unified federated bandits framework that can accommodate various definitions of DP in the sequential decision-making setting. We then formally introduce user-level central DP (CDP) and local DP (LDP) in the federated bandits framework, and investigate the fundamental trade-offs between the learning regrets and the corresponding DP guarantees in a federated linear contextual bandits model. For CDP, we propose a federated algorithm termed as ROBIN and show that it is near-optimal in terms of the number of clients MMM and the privacy budget ε by deriving nearly-matching upper and lower regret bounds when user-level DP is satisfied. For LDP, we obtain several lower bounds, indicating that learning under user-level (ε,δ)-LDP must suffer a regret blow-up factor at least min{1/ε,M} or under different conditions.

ISSN: 2640-3498 abstractTranslation: 本文研究了用户级差分隐私（DP）概念下的联邦线性上下文强盗。我们首先引入一个统一的联邦老虎机框架，该框架可以在顺序决策设置中容纳 DP 的各种定义。然后，我们在联邦强盗框架中正式引入用户级中央DP（CDP）和本地DP（LDP），并研究联邦线性上下文强盗模型中学习遗憾和相应DP保证之间的基本权衡。对于 CDP，我们提出了一种称为 ROBINROBIN\texttt{ROBIN} 的联邦算法，并通过推导出几乎匹配的后悔上限和下限，表明该算法在客户端 MMM 数量和隐私预算 εε\varepsilon 方面接近最优。用户级DP满意。对于LDP，我们得到了几个下界，这表明在用户级别(ε,δ)(ε,δ)(\varepsilon,\delta)-LDP下学习必须遭受至少min{1/ε的遗憾爆炸因子,M}min{1/ε,M}\min{1/\varepsilon,M} 或 min{1/ε√,M−−√}min{1/ε,M}\min{1/不同条件下的\sqrt{\varepsilon},\sqrt{M}}。

Notes:

PUB (https://openreview.net/forum?id=b9opfVNw6O)

PDF (https://arxiv.org/abs/2306.05275)

Achieving Linear Speedup in Non-IID Federated Bilevel Learning

Authors: Minhui Huang; Dewei Zhang; Kaiyi Ji

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/huang23p.html

Abstract: Federated bilevel learning has received increasing attention in various emerging machine learning and communication applications. Recently, several Hessian-vector-based algorithms have been proposed to solve the federated bilevel optimization problem. However, several important properties in federated learning such as the partial client participation and the linear speedup for convergence (i.e., the convergence rate and complexity are improved linearly with respect to the number of sampled clients) in the presence of non-i.i.d. datasets, still remain open. In this paper, we fill these gaps by proposing a new federated bilevel algorithm named FedMBO with a novel client sampling scheme in the federated hypergradient estimation. We show that FedMBO achieves a convergence rate of on non-i.i.d. datasets, where n is the number of participating clients in each round, and K is the total number of iteration. This is the first theoretical linear speedup result for non-i.i.d. federated bilevel optimization. Extensive experiments validate our theoretical results and demonstrate the effectiveness of our proposed method.

ISSN: 2640-3498 abstractTranslation: 联邦双层学习在各种新兴的机器学习和通信应用中受到越来越多的关注。最近，已经提出了几种基于 Hessian 向量的算法来解决联邦双层优化问题。然而，在存在非独立同分布的情况下，联邦学习中的几个重要属性，例如部分客户端参与和收敛的线性加速（即，收敛速度和复杂性相对于采样客户端的数量线性提高）。数据集，仍然保持开放。在本文中，我们通过提出一种名为 FedMBO 的新联邦双层算法来填补这些空白，并在联邦超梯度估计中采用新颖的客户端采样方案。我们证明 FedMBO 的收敛速度为  在非独立同分布上数据集，其中 n 是每轮参与的客户端数量，K 是迭代总数。这是非独立同分布的第一个理论线性加速结果。联邦双层优化。大量的实验验证了我们的理论结果并证明了我们提出的方法的有效性。

Notes:

FeDXL: Provable Federated Learning for Deep X-Risk Optimization

Authors: Zhishuai Guo; Rong Jin; Jiebo Luo; Tianbao Yang

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/guo23c.html

Abstract: In this paper, we tackle a novel federated learning (FL) problem for optimizing a family of X-risks, to which no existing FL algorithms are applicable. In particular, the objective has the form of , where two sets of data are distributed over multiple machines, is a pairwise loss that only depends on the prediction outputs of the input data pairs . This problem has important applications in machine learning, e.g., AUROC maximization with a pairwise loss, and partial AUROC maximization with a compositional loss. The challenges for designing an FL algorithm for X-risks lie in the non-decomposability of the objective over multiple machines and the interdependency between different machines. To this end, we propose an active-passive decomposition framework that decouples the gradient’s components with two types, namely active parts and passive parts, where the active parts depend on local data that are computed with the local model and the passive parts depend on other machines that are communicated/computed based on historical models and samples. Under this framework, we design two FL algorithms (FeDXL) for handling linear and nonlinear f, respectively, based on federated averaging and merging and develop a novel theoretical analysis to combat the latency of the passive parts and the interdependency between the local model parameters and the involved data for computing local gradient estimators. We establish both iteration and communication complexities and show that using the historical samples and models for computing the passive parts do not degrade the complexities. We conduct empirical studies of FeDXL for deep AUROC and partial AUROC maximization, and demonstrate their performance compared with several baselines.

ISSN: 2640-3498 abstractTranslation: 在本文中，我们解决了一个新的联邦学习（FL）问题，用于优化一系列 X 风险，现有的 FL 算法不适用于该问题。特别地，目标的形式为 Ez∼S1f(Ez′∼S2l(w;z,z′))Ez∼S1f(Ez′∼S2l(w;z,z′))\mathbb{E}_{ \mathbf{z}\sim \mathcal{S}1} f(\mathbb{E}{\mathbf{z}'\sim\mathcal{S}_2} \ell(\mathbf{w}; \mathbf{ z}, \mathbf{z}'))，其中两组数据 S1,S2S1,S2\mathcal S_1, \mathcal S_2 分布在多台机器上，l(⋅;⋅,⋅)l(⋅;⋅,⋅) )\ell(\cdot; \cdot,\cdot) 是一个成对损失，仅取决于输入数据对 (z,z′)(z,z′)(\mathbf{z}, \mathbf 的预测输出{z}'）。这个问题在机器学习中具有重要的应用，例如，具有成对损失的 AUROC 最大化，以及具有组合损失的部分 AUROC 最大化。设计针对 X 风险的 FL 算法的挑战在于目标在多台机器上的不可分解性以及不同机器之间的相互依赖性。为此，我们提出了一种主动-被动分解框架，将梯度分量解耦为两种类型，即主动部分和被动部分，其中主动部分依赖于使用局部模型计算的局部数据，被动部分依赖于其他部分基于历史模型和样本进行通信/计算的机器。在此框架下，我们设计了两种基于联邦平均和合并的 FL 算法（FeDXL），分别用于处理线性和非线性 fff，并开发了一种新颖的理论分析来应对无源部件的延迟以及局部模型参数和局部模型参数之间的相互依赖性。用于计算局部梯度估计器的相关数据。我们建立了迭代和通信复杂性，并表明使用历史样本和模型来计算无源部件不会降低复杂性。我们对 FeDXL 进行深度 AUROC 和部分 AUROC 最大化的实证研究，并证明其与几个基线相比的性能。

Notes:

PUB (https://openreview.net/forum?id=C7fNCYdptO)

PDF (https://arxiv.org/abs/2210.14396)

CODE (https://github.com/optimization-ai/icml2023_fedxl)

FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reduction

Authors: Yongxin Guo; Xiaoying Tang; Tao Lin

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/guo23g.html

Abstract: Federated Learning (FL) is a way for machines to learn from data that is kept locally, in order to protect the privacy of clients. This is typically done using local SGD, which helps to improve communication efficiency. However, such a scheme is currently constrained by slow and unstable convergence due to the variety of data on different clients’ devices. In this work, we identify three under-explored phenomena of biased local learning that may explain these challenges caused by local updates in supervised FL. As a remedy, we propose FedBR, a novel unified algorithm that reduces the local learning bias on features and classifiers to tackle these challenges. FedBR has two components. The first component helps to reduce bias in local classifiers by balancing the output of the models. The second component helps to learn local features that are similar to global features, but different from those learned from other data sources. We conducted several experiments to test FedBR and found that it consistently outperforms other SOTA FL methods. Both of its components also individually show performance gains. Our code is available at https://github.com/lins-lab/fedbr.

ISSN: 2640-3498 abstractTranslation: 联邦学习（FL）是机器从本地保存的数据中学习的一种方式，以保护客户的隐私。这通常是使用本地 SGD 完成的，这有助于提高通信效率。然而，由于不同客户端设备上的数据多种多样，目前这种方案受到收敛速度慢且不稳定的限制。在这项工作中，我们确定了三种尚未充分探索的局部学习偏差现象，这可能解释了监督式 FL 中局部更新带来的这些挑战。作为补救措施，我们提出了 FedBR，这是一种新颖的统一算法，可以减少特征和分类器的本地学习偏差，以应对这些挑战。FedBR 有两个组成部分。第一个组件通过平衡模型的输出来帮助减少局部分类器的偏差。第二个组件有助于学习与全局特征相似但与从其他数据源学习的特征不同的局部特征。我们进行了多项实验来测试 FedBR，发现它始终优于其他 SOTA FL 方法。它的两个组件也分别显示出性能提升。我们的代码可在 https://github.com/lins-lab/fedbr 获取。

Notes:

PUB (https://openreview.net/forum?id=nDKoVwNjMH)

PDF (https://arxiv.org/abs/2205.13462)

CODE (https://github.com/lins-lab/fedbr)

Out-of-Distribution Generalization of Federated Learning via Implicit Invariant Relationships

Authors: Yaming Guo; Kai Guo; Xiaofeng Cao; Tieru Wu; Yi Chang

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/guo23b.html

Abstract: Out-of-distribution generalization is challenging for non-participating clients of federated learning under distribution shifts. A proven strategy is to explore those invariant relationships between input and target variables, working equally well for non-participating clients. However, learning invariant relationships is often in an explicit manner from data, representation, and distribution, which violates the federated principles of privacy-preserving and limited communication. In this paper, we propose FedIIR, which implicitly learns invariant relationships from parameter for out-of-distribution generalization, adhering to the above principles. Specifically, we utilize the prediction disagreement to quantify invariant relationships and implicitly reduce it through inter-client gradient alignment. Theoretically, we demonstrate the range of non-participating clients to which FedIIR is expected to generalize and present the convergence results for FedIIR in the massively distributed with limited communication. Extensive experiments show that FedIIR significantly outperforms relevant baselines in terms of out-of-distribution generalization of federated learning.

ISSN: 2640-3498 abstractTranslation: 对于分布变化下联邦学习的非参与客户来说，分布外泛化具有挑战性。一个行之有效的策略是探索输入变量和目标变量之间的不变关系，对于非参与客户同样有效。然而，学习不变关系通常是以显式的方式从数据、表示和分布中学习，这违反了隐私保护和有限通信的联邦原则。在本文中，我们提出了 FedIIR，它遵循上述原则，隐式地从参数中学习不变关系以进行分布外泛化。具体来说，我们利用预测不一致来量化不变关系，并通过客户端间梯度对齐隐式减少它。从理论上讲，我们展示了 FedIIR 预计将推广到的非参与客户的范围，并呈现了 FedIIR 在大规模分布式且通信有限的情况下的收敛结果。大量实验表明，FedIIR 在联邦学习的分布外泛化方面显着优于相关基线。

Notes:

PUB (https://openreview.net/forum?id=JC05k0E2EM)

CODE (https://github.com/YamingGuo98/FedIIR)

Privacy-Aware Compression for Federated Learning Through Numerical Mechanism Design

Authors: Chuan Guo; Kamalika Chaudhuri; Pierre Stock; Michael Rabbat

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/guo23a.html

Abstract: In private federated learning (FL), a server aggregates differentially private updates from a large number of clients in order to train a machine learning model. The main challenge in this setting is balancing privacy with both classification accuracy of the learnt model as well as the number of bits communicated between the clients and server. Prior work has achieved a good trade-off by designing a privacy-aware compression mechanism, called the minimum variance unbiased (MVU) mechanism, that numerically solves an optimization problem to determine the parameters of the mechanism. This paper builds upon it by introducing a new interpolation procedure in the numerical design process that allows for a far more efficient privacy analysis. The result is the new Interpolated MVU mechanism that is more scalable, has a better privacy-utility trade-off, and provides SOTA results on communication-efficient private FL on a variety of datasets.

ISSN: 2640-3498 abstractTranslation: 在私有联邦学习（FL）中，服务器聚合来自大量客户端的差异私有更新，以训练机器学习模型。此设置中的主要挑战是平衡隐私与学习模型的分类准确性以及客户端和服务器之间通信的位数。先前的工作通过设计一种隐私感知压缩机制（称为最小方差无偏（MVU）机制）实现了良好的权衡，该机制通过数值方式解决优化问题以确定机制的参数。本文在此基础上通过在数值设计过程中引入一种新的插值程序来实现更有效的隐私分析。结果是新的插值 MVU 机制更具可扩展性，具有更好的隐私与实用性权衡，并在各种数据集上提供了通信高效的私有 FL 的 SOTA 结果。

Notes:

PUB (https://openreview.net/forum?id=Otdp5SGQMr)

PDF (https://arxiv.org/abs/2211.03942)

CODE (https://github.com/facebookresearch/dp_compression)

Federated Heavy Hitter Recovery under Linear Sketching

Authors: Adria Gascon; Peter Kairouz; Ziteng Sun; Ananda Theertha Suresh

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/gascon23a.html

Abstract: Motivated by real-life deployments of multi-round federated analytics with secure aggregation, we investigate the fundamental communication-accuracy tradeoffs of the heavy hitter discovery and approximate (open-domain) histogram problems under a linear sketching constraint. We propose efficient algorithms based on local subsampling and invertible bloom look-up tables (IBLTs). We also show that our algorithms are information-theoretically optimal for a broad class of interactive schemes. The results show that the linear sketching constraint does increase the communication cost for both tasks by introducing an extra linear dependence on the number of users in a round. Moreover, our results also establish a separation between the communication cost for heavy hitter discovery and approximate histogram in the multi-round setting. The dependence on the number of rounds is at most logarithmic for heavy hitter discovery whereas that of approximate histogram is . We also empirically demonstrate our findings.

ISSN: 2640-3498 abstractTranslation: 在具有安全聚合的多轮联邦分析的现实部署的推动下，我们研究了线性草图约束下的重磅发现和近似（开放域）直方图问题的基本通信准确性权衡。我们提出了基于局部子采样和可逆布隆查找表（IBLT）的高效算法。我们还表明，我们的算法对于广泛的交互方案来说在信息理论上是最优的。结果表明，线性草图约束确实通过引入对一轮中用户数量的额外线性依赖而增加了这两项任务的通信成本。此外，我们的结果还建立了多轮设置中重击者发现的通信成本与近似直方图之间的分离。对于重量级发现而言，对 RRR 轮数的依赖性至多是对数，而近似直方图的依赖性是 θ(R−−√)θ(R)\Theta(\sqrt{R})。我们还凭经验证明了我们的发现。

Notes:

PUB (https://openreview.net/forum?id=zN4oRCrlnM)

PDF (https://arxiv.org/abs/2307.13347)

CODE (https://github.com/google-research/federated)

DoCoFL: Downlink Compression for Cross-Device Federated Learning

Authors: Ron Dorfman; Shay Vargaftik; Yaniv Ben-Itzhak; Kfir Yehuda Levy

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/dorfman23a.html

Abstract: Many compression techniques have been proposed to reduce the communication overhead of Federated Learning training procedures. However, these are typically designed for compressing model updates, which are expected to decay throughout training. As a result, such methods are inapplicable to downlink (i.e., from the parameter server to clients) compression in the cross-device setting, where heterogeneous clients may appear only once during training and thus must download the model parameters. Accordingly, we propose DoCoFL – a new framework for downlink compression in the cross-device setting. Importantly, DoCoFL can be seamlessly combined with many uplink compression schemes, rendering it suitable for bi-directional compression. Through extensive evaluation, we show that DoCoFL offers significant bi-directional bandwidth reduction while achieving competitive accuracy to that of a baseline without any compression.

ISSN: 2640-3498 abstractTranslation: 人们提出了许多压缩技术来减少联邦学习训练过程的通信开销。然而，这些通常是为了压缩模型更新而设计的，预计模型更新会在整个训练过程中衰减。因此，此类方法不适用于跨设备设置中的下行链路（即从参数服务器到客户端）压缩，其中异构客户端在训练期间可能只出现一次，因此必须下载模型参数。因此，我们提出了 DoCoFL——一种跨设备设置中下行链路压缩的新框架。重要的是，DoCoFL可以与许多上行链路压缩方案无缝结合，使其适合双向压缩。通过广泛的评估，我们表明 DoCoFL 可以显着降低双向带宽，同时在没有任何压缩的情况下实现与基线相比具有竞争力的精度。

Notes:

PUB (https://openreview.net/forum?id=VxKr51JjWC)

PDF (https://arxiv.org/abs/2302.00543)

Chameleon: Adapting to Peer Images for Planting Durable Backdoors in Federated Learning

Authors: Yanbo Dai; Songze Li

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/dai23a.html

Abstract: In a federated learning (FL) system, distributed clients upload their local models to a central server to aggregate into a global model. Malicious clients may plant backdoors into the global model through uploading poisoned local models, causing images with specific patterns to be misclassified into some target labels. Backdoors planted by current attacks are not durable, and vanish quickly once the attackers stop model poisoning. In this paper, we investigate the connection between the durability of FL backdoors and the relationships between benign images and poisoned images (i.e., the images whose labels are flipped to the target label during local training). Specifically, benign images with the original and the target labels of the poisoned images are found to have key effects on backdoor durability. Consequently, we propose a novel attack, Chameleon, which utilizes contrastive learning to further amplify such effects towards a more durable backdoor. Extensive experiments demonstrate that Chameleon significantly extends the backdoor lifespan over baselines by 1.2×∼4×, for a wide range of image datasets, backdoor types, and model architectures.

ISSN: 2640-3498 abstractTranslation: 在联邦学习（FL）系统中，分布式客户端将其本地模型上传到中央服务器以聚合成全局模型。恶意客户端可能通过上传中毒的本地模型在全局模型中植入后门，导致特定模式的图像被错误分类到某些目标标签中。目前的攻击所植入的后门并不持久，一旦攻击者停止模型中毒，后门就会很快消失。在本文中，我们研究了 FL 后门的持久性与良性图像和中毒图像（即在本地训练期间标签翻转到目标标签的图像）之间的关系之间的联系。具体来说，发现具有原始图像和中毒图像的目标标签的良性图像对后门耐久性具有关键影响。因此，我们提出了一种新颖的攻击，Chameleon，它利用对比学习来进一步放大这种效果，从而形成更持久的后门。大量实验表明，对于各种图像数据集、后门类型和模型架构，Chameleon 将后门寿命显着延长了基线 1.2×∼4×。

Notes:

PUB (https://openreview.net/forum?id=HtHFnHrZXu)

PDF (https://arxiv.org/abs/2304.12961)

CODE (https://github.com/ybdai7/chameleon-durable-backdoor)

From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning

Authors: Edwige Cyffers; Aurélien Bellet; Debabrota Basu

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/cyffers23a.html

Abstract: We study differentially private (DP) machine learning algorithms as instances of noisy fixed-point iterations, in order to derive privacy and utility results from this well-studied framework. We show that this new perspective recovers popular private gradient-based methods like DP-SGD and provides a principled way to design and analyze new private optimization algorithms in a flexible manner. Focusing on the widely-used Alternating Directions Method of Multipliers (ADMM) method, we use our general framework derive novel private ADMM algorithms for centralized, federated and fully decentralized learning. We establish strong privacy guarantees for these algorithms, leveraging privacy amplification by iteration and by subsampling. Finally, we provide utility guarantees for the three algorithms using a unified analysis that exploits a recent linear convergence result for noisy fixed-point iterations.

ISSN: 2640-3498 abstractTranslation: 我们研究差分隐私（DP）机器学习算法作为噪声定点迭代的实例，以便从这个经过充分研究的框架中获得隐私和实用结果。我们证明，这种新视角恢复了流行的基于私有梯度的方法，如 DP-SGD，并提供了一种以灵活的方式设计和分析新的私有优化算法的原则方法。专注于广泛使用的交替方向乘子法（ADMM）方法，我们使用我们的通用框架衍生出新颖的私有 ADMM 算法，用于集中式、联邦式和完全分散式学习。我们为这些算法建立了强有力的隐私保证，通过迭代和二次采样来利用隐私放大。最后，我们使用统一分析为三种算法提供效用保证，该分析利用了噪声定点迭代的最新线性收敛结果。

Notes:

PUB (https://openreview.net/forum?id=CBLDv6SFMn)

PDF (https://arxiv.org/abs/2302.12559)

CODE (https://github.com/totilas/padadmm)

On the Convergence of Federated Averaging with Cyclic Client Participation

Authors: Yae Jee Cho; Pranay Sharma; Gauri Joshi; Zheng Xu; Satyen Kale; Tong Zhang

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/cho23b.html

Abstract: Federated Averaging (FedAvg) and its variants are the most popular optimization algorithms in federated learning (FL). Previous convergence analyses of FedAvg either assume full client participation or partial client participation where the clients can be uniformly sampled. However, in practical cross-device FL systems, only a subset of clients that satisfy local criteria such as battery status, network connectivity, and maximum participation frequency requirements (to ensure privacy) are available for training at a given time. As a result, client availability follows a natural cyclic pattern. We provide (to our knowledge) the first theoretical framework to analyze the convergence of FedAvg with cyclic client participation with several different client optimizers such as GD, SGD, and shuffled SGD. Our analysis discovers that cyclic client participation can achieve a faster asymptotic convergence rate than vanilla FedAvg with uniform client participation under suitable conditions, providing valuable insights into the design of client sampling protocols.

ISSN: 2640-3498 abstractTranslation: 联邦平均 (FedAvg) 及其变体是联邦学习 (FL) 中最流行的优化算法。之前的 FedAvg 收敛分析要么假设全部客户参与，要么假设部分客户参与，其中客户可以进行统一抽样。然而，在实际的跨设备 FL 系统中，只有满足本地标准（例如电池状态、网络连接和最大参与频率要求（以确保隐私））的客户端子集可在给定时间进行训练。因此，客户可用性遵循自然的循环模式。我们提供（据我们所知）第一个理论框架来分析 FedAvg 与循环客户参与与几种不同的客户优化器（例如 GD、SGD 和洗牌 SGD）的收敛性。我们的分析发现，在适当的条件下，与统一客户参与的普通 FedAvg 相比，循环客户参与可以实现更快的渐近收敛速度，为客户抽样协议的设计提供了宝贵的见解。

Notes:

PUB (https://openreview.net/forum?id=d8LTNXt97w)

PDF (https://arxiv.org/abs/2302.03109)

GuardHFL: Privacy Guardian for Heterogeneous Federated Learning

Authors: Hanxiao Chen; Meng Hao; Hongwei Li; Kangjie Chen; Guowen Xu; Tianwei Zhang; Xilin Zhang

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/chen23j.html

Abstract: Heterogeneous federated learning (HFL) enables clients with different computation and communication capabilities to collaboratively train their own customized models via a query-response paradigm on auxiliary datasets. However, such a paradigm raises serious privacy concerns due to the leakage of highly sensitive query samples and response predictions. We put forth GuardHFL, the first-of-its-kind efficient and privacy-preserving HFL framework. GuardHFL is equipped with a novel HFL-friendly secure querying scheme built on lightweight secret sharing and symmetric-key techniques. The core of GuardHFL is two customized multiplication and comparison protocols, which substantially boost the execution efficiency. Extensive evaluations demonstrate that GuardHFL significantly outperforms the alternative instantiations based on existing state-of-the-art techniques in both runtime and communication cost.

ISSN: 2640-3498 abstractTranslation: 异构联邦学习 (HFL) 使具有不同计算和通信能力的客户能够通过辅助数据集上的查询响应范例协作训练自己的定制模型。然而，由于高度敏感的查询样本和响应预测的泄露，这种范例引起了严重的隐私问题。我们提出了 GuardHFL，这是首个高效且保护隐私的 HFL 框架。GuardHFL 配备了一种基于轻量级秘密共享和对称密钥技术的新型 HFL 友好安全查询方案。GuardHFL的核心是两个定制的乘法和比较协议，极大地提高了执行效率。广泛的评估表明，GuardHFL 在运行时间和通信成本方面显着优于基于现有最先进技术的替代实例。

Notes:

PUB (https://openreview.net/forum?id=iASUTBGw07)

Efficient Personalized Federated Learning via Sparse Model-Adaptation

Authors: Daoyuan Chen; Liuyi Yao; Dawei Gao; Bolin Ding; Yaliang Li

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/chen23aj.html

Abstract: Federated Learning (FL) aims to train machine learning models for multiple clients without sharing their own private data. Due to the heterogeneity of clients’ local data distribution, recent studies explore the personalized FL that learns and deploys distinct local models with the help of auxiliary global models. However, the clients can be heterogeneous in terms of not only local data distribution, but also their computation and communication resources. The capacity and efficiency of personalized models are restricted by the lowest-resource clients, leading to sub-optimal performance and limited practicality of personalized FL. To overcome these challenges, we propose a novel approach named pFedGate for efficient personalized FL by adaptively and efficiently learning sparse local models. With a lightweight trainable gating layer, pFedGate enables clients to reach their full potential in model capacity by generating different sparse models accounting for both the heterogeneous data distributions and resource constraints. Meanwhile, the computation and communication efficiency are both improved thanks to the adaptability between the model sparsity and clients’ resources. Further, we theoretically show that the proposed pFedGate has superior complexity with guaranteed convergence and generalization error. Extensive experiments show that pFedGate achieves superior global accuracy, individual accuracy and efficiency simultaneously over state-of-the-art methods. We also demonstrate that pFedGate performs better than competitors in the novel clients participation and partial clients participation scenarios, and can learn meaningful sparse local models adapted to different data distributions.

ISSN: 2640-3498 abstractTranslation: 联邦学习（FL）旨在为多个客户训练机器学习模型，而无需共享他们自己的私人数据。由于客户本地数据分布的异构性，最近的研究探索了个性化 FL，它在辅助全局模型的帮助下学习和部署不同的本地模型。然而，客户端不仅在本地数据分布方面可能是异构的，而且在其计算和通信资源方面也可能是异构的。个性化模型的容量和效率受到资源最少的客户端的限制，导致个性化 FL 的性能次佳和实用性有限。为了克服这些挑战，我们提出了一种名为 pFedGate 的新方法，通过自适应且高效地学习稀疏局部模型来实现高效的个性化 FL。凭借轻量级的可训练门控层，pFedGate 能够通过生成不同的稀疏模型来充分发挥模型容量的潜力，同时考虑到异构数据分布和资源限制。同时，由于模型稀疏性和客户端资源之间的适应性，计算和通信效率都得到了提高。此外，我们从理论上表明，所提出的 pFedGate 具有卓越的复杂性，并保证收敛和泛化误差。大量实验表明，与最先进的方法相比，pFedGate 同时实现了卓越的全局精度、个体精度和效率。我们还证明，pFedGate 在新颖的客户参与和部分客户参与场景中比竞争对手表现更好，并且可以学习适应不同数据分布的有意义的稀疏本地模型。

Notes:

PUB (https://openreview.net/forum?id=ieSN7Xyo8g)

PDF (https://arxiv.org/abs/2305.02776)

CODE (https://github.com/alibaba/federatedscope)

CODE (https://github.com/yxdyc/pfedgate)

Fast Federated Machine Unlearning with Nonlinear Functional Theory

Authors: Tianshi Che; Yang Zhou; Zijie Zhang; Lingjuan Lyu; Ji Liu; Da Yan; Dejing Dou; Jun Huan

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/che23b.html

Abstract: Federated machine unlearning (FMU) aims to remove the influence of a specified subset of training data upon request from a trained federated learning model. Despite achieving remarkable performance, existing FMU techniques suffer from inefficiency due to two sequential operations of training and retraining/unlearning on large-scale datasets. Our prior study, PCMU, was proposed to improve the efficiency of centralized machine unlearning (CMU) with certified guarantees, by simultaneously executing the training and unlearning operations. This paper proposes a fast FMU algorithm, FFMU, for improving the FMU efficiency while maintaining the unlearning quality. The PCMU method is leveraged to train a local machine learning (MU) model on each edge device. We propose to employ nonlinear functional analysis techniques to refine the local MU models as output functions of a Nemytskii operator. We conduct theoretical analysis to derive that the Nemytskii operator has a global Lipschitz constant, which allows us to bound the difference between two MU models regarding the distance between their gradients. Based on the Nemytskii operator and average smooth local gradients, the global MU model on the server is guaranteed to achieve close performance to each local MU model with the certified guarantees.

ISSN: 2640-3498 abstractTranslation: 联邦机器取消学习 (FMU) 旨在根据经过训练的联邦学习模型的请求消除指定训练数据子集的影响。尽管取得了显着的性能，但现有的 FMU 技术由于在大规模数据集上进行两次连续的训练和再训练/取消学习操作而效率低下。我们之前的研究 PCMU 旨在通过同时执行训练和取消学习操作来提高具有认证保证的集中式机器取消学习（CMU）的效率。本文提出了一种快速 FMU 算法 FFMU，用于在保持取消学习质量的同时提高 FMU 效率。利用 PCMU 方法在每个边缘设备上训练本地机器学习 (MU) 模型。我们建议采用非线性函数分析技术来细化局部 MU 模型作为 Nemytskii 算子的输出函数。我们进行理论分析，得出 Nemytskii 算子具有全局 Lipschitz 常数，这使我们能够限制两个 MU 模型之间关于梯度之间距离的差异。基于 Nemytskii 算子和平均平滑局部梯度，服务器上的全局 MU 模型保证在经过认证的保证下实现与每个局部 MU 模型接近的性能。

Notes:

PUB (https://openreview.net/forum?id=6wQKmKiDHw)

LESS-VFL: Communication-Efficient Feature Selection for Vertical Federated Learning

Authors: Timothy Castiglia; Yi Zhou; Shiqiang Wang; Swanand Kadhe; Nathalie Baracaldo; Stacy Patterson

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/castiglia23a.html

Abstract: We propose LESS-VFL, a communication-efficient feature selection method for distributed systems with vertically partitioned data. We consider a system of a server and several parties with local datasets that share a sample ID space but have different feature sets. The parties wish to collaboratively train a model for a prediction task. As part of the training, the parties wish to remove unimportant features in the system to improve generalization, efficiency, and explainability. In LESS-VFL, after a short pre-training period, the server optimizes its part of the global model to determine the relevant outputs from party models. This information is shared with the parties to then allow local feature selection without communication. We analytically prove that LESS-VFL removes spurious features from model training. We provide extensive empirical evidence that LESS-VFL can achieve high accuracy and remove spurious features at a fraction of the communication cost of other feature selection approaches.

ISSN: 2640-3498 abstractTranslation: 我们提出了 LESS-VFL，一种用于具有纵向分区数据的分布式系统的通信高效特征选择方法。我们考虑一个由服务器和多个具有本地数据集的各方组成的系统，这些数据集共享样本 ID 空间但具有不同的特征集。双方希望合作训练一个用于预测任务的模型。作为培训的一部分，各方希望删除系统中不重要的特征，以提高泛化性、效率和可解释性。在 LESS-VFL 中，经过短暂的预训练期后，服务器优化其全局模型的一部分，以确定各方模型的相关输出。该信息与各方共享，然后无需通信即可进行本地特征选择。我们分析证明 LESS-VFL 消除了模型训练中的虚假特征。我们提供了广泛的经验证据，表明 LESS-VFL 可以实现高精度并以其他特征选择方法的一小部分通信成本去除虚假特征。

Notes:

PUB (https://openreview.net/forum?id=L8iWCxzwl1)

PDF (https://arxiv.org/abs/2305.02219)

Optimizing the Collaboration Structure in Cross-Silo Federated Learning

Authors: Wenxuan Bao; Haohan Wang; Jun Wu; Jingrui He

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/bao23b.html

Abstract: In federated learning (FL), multiple clients collaborate to train machine learning models together while keeping their data decentralized. Through utilizing more training data, FL suffers from the potential negative transfer problem: the global FL model may even perform worse than the models trained with local data only. In this paper, we propose FedCollab, a novel FL framework that alleviates negative transfer by clustering clients into non-overlapping coalitions based on their distribution distances and data quantities. As a result, each client only collaborates with the clients having similar data distributions, and tends to collaborate with more clients when it has less data. We evaluate our framework with a variety of datasets, models, and types of non-IIDness. Our results demonstrate that FedCollab effectively mitigates negative transfer across a wide range of FL algorithms and consistently outperforms other clustered FL algorithms.

ISSN: 2640-3498 abstractTranslation: 在联邦学习 (FL) 中，多个客户协作共同训练机器学习模型，同时保持数据分散。通过利用更多的训练数据，FL 面临着潜在的负迁移问题：全局 FL 模型甚至可能比仅使用本地数据训练的模型表现更差。在本文中，我们提出了 FedCollab，这是一种新颖的 FL 框架，它通过根据客户的分布距离和数据量将客户聚类成不重叠的联盟来减轻负转移。结果，每个客户端仅与具有相似数据分布的客户端协作，并且当其数据较少时倾向于与更多客户端协作。我们使用各种数据集、模型和非独立同分布类型来评估我们的框架。我们的结果表明，FedCollab 有效地减轻了各种 FL 算法的负迁移，并且始终优于其他集群 FL 算法。

Notes:

PUB (https://openreview.net/forum?id=rnNBSMOWvA)

PDF (https://arxiv.org/abs/2306.06508)

CODE (https://github.com/baowenxuan/fedcollab) SLIDES(https://icml.cc/media/icml-2023/Slides/23569.pdf)

Personalized Subgraph Federated Learning

Authors: Jinheon Baek; Wonyong Jeong; Jiongdao Jin; Jaehong Yoon; Sung Ju Hwang

Conference : International Conference on Machine Learning

Url: https://proceedings.mlr.press/v202/baek23a.html

Abstract: Subgraphs of a larger global graph may be distributed across multiple devices, and only locally accessible due to privacy restrictions, although there may be links between subgraphs. Recently proposed subgraph Federated Learning (FL) methods deal with those missing links across local subgraphs while distributively training Graph Neural Networks (GNNs) on them. However, they have overlooked the inevitable heterogeneity between subgraphs comprising different communities of a global graph, consequently collapsing the incompatible knowledge from local GNN models. To this end, we introduce a new subgraph FL problem, personalized subgraph FL, which focuses on the joint improvement of the interrelated local GNNs rather than learning a single global model, and propose a novel framework, FEDerated Personalized sUBgraph learning (FED-PUB), to tackle it. Since the server cannot access the subgraph in each client, FED-PUB utilizes functional embeddings of the local GNNs using random graphs as inputs to compute similarities between them, and use the similarities to perform weighted averaging for server-side aggregation. Further, it learns a personalized sparse mask at each client to select and update only the subgraph-relevant subset of the aggregated parameters. We validate our FED-PUB for its subgraph FL performance on six datasets, considering both non-overlapping and overlapping subgraphs, on which it significantly outperforms relevant baselines. Our code is available at https://github.com/JinheonBaek/FED-PUB.

ISSN: 2640-3498 abstractTranslation: 较大全局图的子图可能分布在多个设备上，并且由于隐私限制只能在本地访问，尽管子图之间可能存在链接。最近提出的子图联邦学习（FL）方法可以处理局部子图之间缺失的链接，同时在其上分布式训练图神经网络（GNN）。然而，他们忽视了由全局图的不同社区组成的子图之间不可避免的异质性，从而瓦解了局部 GNN 模型中不兼容的知识。为此，我们引入了一种新的子图 FL 问题，即个性化子图 FL，其重点是相互关联的局部 GNN 的联邦改进，而不是学习单个全局模型，并提出了一种新颖的框架：FEDerated Personalized sUBgraph Learning (FED-PUB) ，来解决它。由于服务器无法访问每个客户端中的子图，因此 FED-PUB 利用本地 GNN 的功能嵌入（使用随机图作为输入）来计算它们之间的相似度，并使用相似度对服务器端聚合执行加权平均。此外，它在每个客户端学习个性化稀疏掩码，以仅选择和更新聚合参数的子图相关子集。我们在六个数据集上验证了 FED-PUB 的子图 FL 性能，同时考虑了非重叠和重叠子图，其显着优于相关基线。我们的代码可在 https://github.com/JinheonBaek/FED-PUB 获取。

Notes:

PUB (https://openreview.net/forum?id=GXHL8ZS1GX)

PDF (https://arxiv.org/abs/2206.10206)

CODE (https://github.com/JinheonBaek/FED-PUB)

项目链接: https://zhuanlan.zhihu.com/p/648688758

作者: 白小鱼（上海交通大学计算机系博士生）

END
热文推荐