爱可可AI前沿推介(2.13)
LG - 机器学习 CV - 计算机视觉 CL - 计算与语言
1、[LG] On the Trade-Off between Actionable Explanations and the Right to be Forgotten
2、[LG] Language Models are Realistic Tabular Data Generators
3、[LG] GOGGLE: Generative Modelling for Tabular Data by Learning Relational Structure
4、[CL] Offsite-Tuning: Transfer Learning without Full Model
5、[CL] Theory of Mind May Have Spontaneously Emerged in Large Language Models
[CV] RobustNeRF: Ignoring Distractors with Robust Losses
[CL] Benchmarking Large Language Models for News Summarization
[CL] ExaRanker: Explanation-Augmented Neural Ranker
[LG] Graph Representation Learning via Aggregation Enhancement
摘要：可诉解释权和被遗忘权之间的权衡、基于语言模型的逼真表格数据生成器、基于关系结构学习的表格数据生成式建模、无需完整模型的迁移学习、大型语言模型可能会自发涌现出心智、基于鲁棒性损失的干扰物剔除、面向新闻摘要的大型语言模型基准测试、解释增强神经排序器、基于聚合增强的图表示学习
1、[LG] On the Trade-Off between Actionable Explanations and the Right to be Forgotten
M Pawelczyk, T Leemann, A Biega, G Kasneci
Harvard University & University of Tuebingen & Max Planck Institute for Security and Privacy
可诉解释权和被遗忘权之间的权衡
要点:
介绍了数据删除请求背景下的追索权无效问题，提出了关于被遗忘权背景下可诉解释权的兼容性的基本问题。 从理论上分析了当数据属于训练集的用户提交删除请求时，决定追索权不稳定的因素； 提出了一个优化框架，以确定一小部分关键的训练数据点，这些数据点一旦被删除，大部分发出的追索权就会失效，并在多个真实世界的数据集上进行了广泛的实验，以显示其发现的实际意义。
一句话总结:
分析了可诉解释权和被遗忘权之间的权衡，并提供算法找到训练数据点的一个关键子集，当删除这些数据点时，将导致资源的最大化失效。
摘要：
随着机器学习(ML)模型越来越多地被部署在高风险应用中，策略制定者提出了更严格的数据保护法规(如GDPR、CCPA)。一个关键原则是"被遗忘的权利"，它赋予用户删除其数据的权利。另一个关键原则，是获得可诉解释权，也被称为算法追索权，允许用户扭转不利的决定。到目前为止，这两个原则是否可以同时操作还不得而知。本文引入并研究了在数据删除请求的背景下追索权无效的问题。本文从理论和经验上分析了流行的最先进的算法的行为，并证明如果有少量的数据删除请求(一两个)需要更新预测模型，那么这些算法产生的追索权就有可能被废止。对于线性模型和过参数化的神经网络的设置——通过神经切线核(NTK)的角度进行研究——提出了一个框架，以确定关键训练点的最小子集，当这些训练点被删除时，无效资源的比例最大化。使用该框架，从经验上表明，从训练集中删除少至2个数据实例可以使流行的最先进的算法输出的所有资源中的95%无效。因此，本文提出关于在 "被遗忘权"背景下"可诉解释权"的兼容性的基本问题，同时也提供了关于追索权鲁棒性决定因素的建设性见解。
As machine learning (ML) models are increasingly being deployed in high-stakes applications, policymakers have suggested tighter data protection regulations (e.g., GDPR, CCPA). One key principle is the “right to be forgotten” which gives users the right to have their data deleted. Another key principle is the right to an actionable explanation, also known as algorithmic recourse, allowing users to reverse unfavorable decisions. To date it is unknown whether these two principles can be operationalized simultaneously. Therefore, we introduce and study the problem of recourse invalidation in the context of data deletion requests. More specifically, we theoretically and empirically analyze the behavior of popular state-of-the-art algorithms and demonstrate that the recourses generated by these algorithms are likely to be invalidated if a small number of data deletion requests (e.g., 1 or 2) warrant updates of the predictive model. For the setting of linear models and overparameterized neural networks – studied through the lens of neural tangent kernels (NTKs) – we suggest a framework to identify a minimal subset of critical training points which, when removed, maximize the fraction of invalidated recourses. Using our framework, we empirically show that the removal of as little as 2 data instances from the training set can invalidate up to 95 percent of all recourses output by popular state-of-the-art algorithms. Thus, our work raises fundamental questions about the compatibility of “the right to an actionable explanation” in the context of the “right to be forgotten” while also providing constructive insights on the determining factors of recourse robustness.
https://openreview.net/forum?id=HWt4BBZjVW
2、[LG] Language Models are Realistic Tabular Data Generators
V Borisov, K Sessler, T Leemann, M Pawelczyk, G Kasneci
University of Tuebingen
基于语言模型的逼真表格数据生成器
要点:
GReaT是一种新方法，用 Transformer-解码器网络架构生成现实的异质表格数据，通过文本编码方案连接表格和文本数据模态； GReaT提供了任意的调节能力，能对以任意给定的特征子集为条件的数据分布进行建模，并对剩余的特征进行采样； 实验结果表明，GReaT在各种数据集上获得了最先进的生成性能。
一句话总结:
提出一种方法，GReaT(逼真表格数据生成)，用大型语言模型生成高度逼真的合成表格数据，利用自回归生成式LLM对合成表格数据进行采样，同时在许多具有异质特征类型的现实世界数据集中保持最先进的性能。
摘要：
表格数据是最古老和最普遍的数据形式之一。然而，生成具有原始数据特征的合成样本，对于表格数据来说仍然是一个重大挑战。虽然计算机视觉领域的许多生成模型，如自编码器或生成对抗网络，已经被改编为表格数据的生成，但对最近基于 Transformer 的大型语言模型(LLM)的研究较少，这些模型在本质上也是生成的。本文提出GReaT(真实表格数据生成)，利用一个自回归生成 LLM 来对合成的但又高度现实的表格数据进行采样。此外，GReaT 可通过对任何特征子集的调节来建立表格数据分布模型；其余的特征被抽样，没有额外的开销。本文在一系列的实验中证明了所提出的方法的有效性，这些实验从多个角度量化了所产生的数据样本的有效性和质量。GReaT 在许多具有不同大小的异质特征类型的真实世界数据集中保持了最先进的性能。
Tabular data is among the oldest and most ubiquitous forms of data. However, the generation of synthetic samples with the original data's characteristics remains a significant challenge for tabular data. While many generative models from the computer vision domain, such as autoencoders or generative adversarial networks, have been adapted for tabular data generation, less research has been directed towards recent transformer-based large language models (LLMs), which are also generative in nature. To this end, we propose GReaT (Generation of Realistic Tabular data), which exploits an auto-regressive generative LLM to sample synthetic and yet highly realistic tabular data. Furthermore, GReaT can model tabular data distributions by conditioning on any subset of features; the remaining features are sampled without additional overhead. We demonstrate the effectiveness of the proposed approach in a series of experiments that quantify the validity and quality of the produced data samples from multiple angles. We find that GReaT maintains state-of-the-art performance across many real-world data sets with heterogeneous feature types coming in various sizes.
https://openreview.net/forum?id=cEygmQNOeI
3、[LG] GOGGLE: Generative Modelling for Tabular Data by Learning Relational Structure
T Liu, Z Qian, J Berrevoets, M v d Schaar
University of Cambridge
GOGGLE: 基于关系结构学习的表格数据生成式建模
要点:
GOGGLE是一种新的表格数据生成模型，学习并利用关系结构来更好地捕捉数据中的稀疏性和异质关系，同时引入先验知识和变量依赖关系的正则化； 所提出的消息传递方案，联合学习了关系结构和功能关系，使其成为第一个对表格数据做到这一点的工作； GOGGLE 在生成逼真合成数据和利用领域知识完成下游任务方面表现有效，超过了最先进的基准测试。
一句话总结:
提出一种新的表格数据生成模型GOGGLE，学习并利用关系结构来更好地模拟变量依赖性，并引入关系和先验知识的正则化。GOGGLE使用一个端到端的消息传递方案来联合学习关系结构和功能关系，作为生成合成样本的基础。
摘要：
深度生成模型学习高度复杂的非线性表征，以生成逼真的合成数据。虽然他们在计算机视觉和自然语言处理方面取得了明显的成功，但类似的进展在表格领域却不那么明显。部分原因是表格数据的生成性建模带来了一系列特殊的挑战，包括异质关系、有限的样本数量，以及纳入先验知识的困难。此外，与图像和序列领域的对应模型不同，表格数据的深度生成模型几乎只采用全连接层，它对输入之间的关系编码了弱的归纳偏差。现实世界的数据生成过程通常可以用关系结构来表示，这种结构编码变量间稀疏的异质关系。本文学习并利用表格数据背后的关系结构，来更好地模拟变量的依赖性，并作为一种自然的手段来引入关系的正则化，并包括先验知识。本文提出 GOGGLE，一个端到端的信息传递方案，联合学习关系结构和相应的功能关系，作为生成合成样本的基础。利用真实世界的数据集，提供了实证证据，证明所提出的方法在生成逼真的合成数据和利用领域知识进行下游任务方面是有效的。
Deep generative models learn highly complex and non-linear representations to generate realistic synthetic data. While they have achieved notable success in computer vision and natural language processing, similar advances have been less demonstrable in the tabular domain. This is partially because generative modelling of tabular data entails a particular set of challenges, including heterogeneous relationships, limited number of samples, and difficulties in incorporating prior knowledge. Additionally, unlike their counterparts in image and sequence domain, deep generative models for tabular data almost exclusively employ fully-connected layers, which encode weak inductive biases about relationships between inputs. Real-world data generating processes can often be represented using relational structures, which encode sparse, heterogeneous relationships between variables. In this work, we learn and exploit relational structure underlying tabular data to better model variable dependence, and as a natural means to introduce regularization on relationships and include prior knowledge. Specifically, we introduce GOGGLE, an end-to-end message passing scheme that jointly learns the relational structure and corresponding functional relationships as the basis of generating synthetic samples. Using real-world datasets, we provide empirical evidence that the proposed method is effective in generating realistic synthetic data and exploiting domain knowledge for downstream tasks.
https://openreview.net/forum?id=fPVRcJqspu
4、[CL] Offsite-Tuning: Transfer Learning without Full Model
G Xiao, J Lin, S Han
[MIT]
Offsite-Tuning: 无需完整模型的迁移学习
要点:
Offsite-Tuning 是一种保护隐私且高效的迁移学习框架，允许在不接触完整模型的情况下将大型基础模型适应于下游数据； Offsite-Tuning 涉及到用一个轻量的适配器和一个有损压缩的仿真器，来对下游数据进行微调，保留了双方的隐私，并且在计算上比现有的微调方法更高效； 该框架可以达到与完整模型微调相当的精度，同时又能保护隐私和效率，实现了高达6.5倍的速度提升和高达5.6倍的内存减少； Offsite-Tuning 可以实现以前在单个GPU上无法实现的模型微调，如OPT-6.7B和BLOOM-7.1B。
一句话总结:
提出 Offsite-Tuning，一种保护隐私且高效的迁移学习框架，允许在不接触完整模型的情况下将大型基础模型适应于下游数据。该框架包括模型所有者向数据所有者发送一个轻量级的适配器和一个有损压缩的仿真器，后者在仿真器的帮助下对下游数据的适配器进行微调。经过微调的适配器被返回给模型所有者，并插入到完整的模型中，为下游用户创建一个自适应的基础模型。
摘要：
迁移学习对于基础模型适应下游任务很重要。然而，许多基础模型是专有的，所以用户必须与模型所有者分享他们的数据以微调模型，这是很昂贵的，并引起了隐私问题。此外，微调大型基础模型是计算密集型的，对大多数下游用户来说不切实际。本文提出 Offsite-Tuning，一种保护隐私且高效的迁移学习框架，可以在不接触完整模型的情况下将十亿参数的基础模型适应于下游数据。在 Offsite-Tuning 中，模型所有者向数据所有者发送一个轻量的适配器和一个有损压缩的仿真器，在仿真器的帮助下对下游数据的适配器进行微调。微调后的适配器被返回给模型所有者，并将适配器插入完整模型中，以创建一个自适应的基础模型。Offsite-Tuning 保留了双方的隐私，并且比现有的需要访问完整模型权重的微调方法在计算上更高效。在各种大型语言和视觉基础模型上证明了 Offsite-Tuning 的有效性。Offsite-Tuning 可以达到与全模型微调相当的精度，同时又能保护隐私和效率，实现了6.5倍的速度提升和5.6倍的内存减少。
Transfer learning is important for foundation models to adapt to downstream tasks. However, many foundation models are proprietary, so users must share their data with model owners to fine-tune the models, which is costly and raise privacy concerns. Moreover, fine-tuning large foundation models is computation-intensive and impractical for most downstream users. In this paper, we propose Offsite-Tuning, a privacy-preserving and efficient transfer learning framework that can adapt billion-parameter foundation models to downstream data without access to the full model. In offsite-tuning, the model owner sends a light-weight adapter and a lossy compressed emulator to the data owner, who then fine-tunes the adapter on the downstream data with the emulator's assistance. The fine-tuned adapter is then returned to the model owner, who plugs it into the full model to create an adapted foundation model. Offsite-tuning preserves both parties' privacy and is computationally more efficient than the existing fine-tuning methods that require access to the full model weights. We demonstrate the effectiveness of offsite-tuning on various large language and vision foundation models. Offsite-tuning can achieve comparable accuracy as full model fine-tuning while being privacy-preserving and efficient, achieving 6.5x speedup and 5.6x memory reduction. Code is available at this https URL.
https://arxiv.org/abs/2302.04870
5、[CL] Theory of Mind May Have Spontaneously Emerged in Large Language Models
M Kosinski
[Stanford University]
大型语言模型可能会自发涌现出心智
要点:
最近的语言模型，特别是2022年1月版本的GPT-3和2022年11月版本的GPT-3.5，在解决心智任务方面表现出显著的性能，分别与7岁和9岁儿童的性能相当； 2022年以前发表的模型几乎没有显示出解决心智任务的能力，而最近的和最大的模型显示出很高的性能，这表明类似心智的能力可能是作为语言模型提高语言技能的副产品而自发涌现的； 类心智能力在语言模型中的出现，可能对人工智能的发展产生重大影响，改善其与人类互动和交流的能力，并发展其他依赖心智的能力，如移情、道德判断或自我意识。 该研究强调了将心理科学应用于研究复杂的人工神经网络的有用性，以及人工智能提供对人类认知的洞察力的潜力。
一句话总结:
研究了语言模型是否能自发地发展心智(ToM)，即把无法观察到的心理状态归因于他人的能力。在没有任何预训练的情况下，对几种语言模型进行了广泛用于测试人类心智的 false-belief 任务。其结果显示，最近的语言模型在心智任务中取得了很好的表现，表明类似心智的能力可能是作为语言模型提高语言技能的副产品而自发涌现的。
摘要：
心智理论(ToM)，或将无法观察到的心理状态归于他人的能力，是人类社会互动、交流、移情、自我意识和道德的核心。本文对几种语言模型进行经典的 false-belief 任务，这些任务广泛用于测试人类心智，没有任何样本或预训练。结果显示，在2022年之前发表的模型几乎没有显示出解决ToM任务的能力。然而，2022年1月版本的GPT-3(davinci-002)解决了70%的心智任务，这一性能与7岁儿童相当。此外，其2022年11月的版本（davinci-003）解决了93%的心智任务，与9岁儿童的表现相当。这些发现表明，类似心智的能力(迄今为止被认为是人类独有的)可能是作为语言模型提高语言技能的副产品而自发涌现的。
Theory of mind (ToM), or the ability to impute unobservable mental states to others, is central to human social interactions, communication, empathy, self-consciousness, and morality. We administer classic false-belief tasks, widely used to test ToM in humans, to several language models, without any examples or pre-training. Our results show that models published before 2022 show virtually no ability to solve ToM tasks. Yet, the January 2022 version of GPT-3 (davinci-002) solved 70% of ToM tasks, a performance comparable with that of seven-year-old children. Moreover, its November 2022 version (davinci-003), solved 93% of ToM tasks, a performance comparable with that of nine-year-old children. These findings suggest that ToM-like ability (thus far considered to be uniquely human) may have spontaneously emerged as a byproduct of language models' improving language skills.
https://arxiv.org/abs/2302.02083
另外几篇值得关注的论文：
[CV] RobustNeRF: Ignoring Distractors with Robust Losses
S Sabour, S Vora, D Duckworth, I Krasin, D J. Fleet, A Tagliasacchi
[Google Research]
RobustNeRF: 基于鲁棒性损失的干扰物剔除
要点:
NeRF模型在处理图像采集过程中非持久干扰物时很吃力，例如移动的物体、光照变化和阴影，导致了与视图相关的伪影； RobustNeRF 使用鲁棒性估计，将干扰物建模为优化问题中的异常值，并成功地将它们从场景中移除，从而在广泛的数据集上提高了性能；
3。 之前在 NeRF 模型中处理干扰物的方法，包括预训练的语义分割模型，将干扰物建模为每个图像的瞬时现象，以及将场景分解为静态和动态部分，但 RobustNeRF 实现起来更简单，并实现了最先进的性能。
一句话总结:
提出一种名为 RobustNeRF 的方法，使用鲁棒性估计来克服神经辐射场(NeRF)中的干扰因素问题。RobustNeRF 将训练数据中的干扰物建模为异常值，并成功将其从场景中移除，从而提高了合成和真实世界场景的性能。
Neural radiance fields (NeRF) excel at synthesizing new views given multi-view, calibrated images of a static scene. When scenes include distractors, which are not persistent during image capture (moving objects, lighting variations, shadows), artifacts appear as view-dependent effects or 'floaters'. To cope with distractors, we advocate a form of robust estimation for NeRF training, modeling distractors in training data as outliers of an optimization problem. Our method successfully removes outliers from a scene and improves upon our baselines, on synthetic and real-world scenes. Our technique is simple to incorporate in modern NeRF frameworks, with few hyper-parameters. It does not assume a priori knowledge of the types of distractors, and is instead focused on the optimization problem rather than pre-processing or modeling transient objects. More results on our page this https URL.
https://arxiv.org/abs/2302.00833
[CL] Benchmarking Large Language Models for News Summarization
T Zhang, F Ladhak, E Durmus, P Liang, K McKeown, T B. Hashimoto
[Stanford Univeristy & Columbia Univeristy]
面向新闻摘要的大型语言模型基准测试
要点:
对于大型语言模型的摘要能力来说，指令微调比模型大小更重要； 参考摘要质量对于评估大型语言模型至关重要，而现有的低质量参考摘要的基准可能提供有限的价值； LLM 生成的摘要被认为与扔撰写的摘要相当，但标注者之间仍然存在很大的差异； 本文提供了由自由撰稿人撰写的高质量摘要，作为未来改进评价工作的资源。
一句话总结:
介绍了对10个大型语言模型(LLM)进行的新闻摘要的人工评估，发现对于零样本摘要能力来说，指令微调比模型大小更重要。该研究还强调了良好的参考摘要在模型开发和评估中的关键作用，并提供了由自由撰稿人撰写的高质量摘要作为未来改进评估工作的资源。
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, and not model size, is the key to the LLM's zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LMM summaries are judged to be on par with human written summaries.
https://arxiv.org/abs/2301.13848
[CL] ExaRanker: Explanation-Augmented Neural Ranker
F Ferraretto, T Laitz, R Lotufo, R Nogueira
[UNICAMP]
ExaRanker: 解释增强神经排序器
要点:
由大型语言模型生成的解释，可以极大地提高IR模型性能，特别是在可用的标记样本较少的情况下； ExaRanker 是一种为给定的查询-文档对输出相关性标签和解释的排序模型，其表现与在没有解释的情况下对三倍多的样本进行微调的模型相当； ExaRanker 在排序过程中不产生额外的计算成本，并允许根据需要请求解释。
一句话总结:
提出 ExaRanker，一种使用大型语言模型(LLM)生成自然语言解释以增强检索数据集的神经排序器模型，提高了在信息检索(IR)任务上的性能。
Recent work has shown that inducing a large language model (LLM) to generate explanations prior to outputting an answer is an effective strategy to improve performance on a wide range of reasoning tasks. In this work, we show that neural rankers also benefit from explanations. We use LLMs such as GPT-3.5 to augment retrieval datasets with explanations and train a sequence-to-sequence ranking model to output a relevance label and an explanation for a given query-document pair. Our model, dubbed ExaRanker, finetuned on a few thousand examples with synthetic explanations performs on par with models finetuned on 3x more examples without explanations. Furthermore, the ExaRanker model incurs no additional computational cost during ranking and allows explanations to be requested on demand.
https://arxiv.org/abs/2301.10521
[LG] Graph Representation Learning via Aggregation Enhancement
M Fishman, C Baskin, E Zheltonozhskii, A David, R Banner, A Mendelson
[Technion & Habana Labs.]
基于聚合增强的图表示学习
要点:
核回归(KR)方法可以在监督和自监督的情况下强化图表示学习； KR 损失最小化导致互信息(MI)最大化，由于KR的凸性及其高效的估计，不需要额外的可学习参数，KR损失最小化可以成为 MI 估计的一个更实用的选择； 在深度图神经网络(GNN)的监督训练中纳入基于 KR 的正则化，可以改善信息聚合并缓解深度问题。 所提出的基于 KR 的自监督图表示学习算法GIRL优于之前最先进的基于对比的算法，特别是在大规模图上。
一句话总结:
提出使用核回归(KR)方法来加强监督和自监督的图表示学习。最小化 KR 损失会导致互信息(MI)最大化，在深度图神经网络(GNN)的监督训练中加入基于 KR 的正则化可以改善信息聚合。提出一种新的自监督图表示学习算法，称为图信息表示学习(GIRL)，基于KR，优于之前最先进的基于对比的算法。
Graph neural networks (GNNs) have become a powerful tool for processing graph-structured data but still face challenges in effectively aggregating and propagating information between layers, which limits their performance. We tackle this problem with the kernel regression (KR) approach, using KR loss as the primary loss in self-supervised settings or as a regularization term in supervised settings. We show substantial performance improvements compared to state-of-the-art in both scenarios on multiple transductive and inductive node classification datasets, especially for deep networks. As opposed to mutual information (MI), KR loss is convex and easy to estimate in high-dimensional cases, even though it indirectly maximizes the MI between its inputs. Our work highlights the potential of KR to advance the field of graph representation learning and enhance the performance of GNNs. The code to reproduce our experiments is available at this https URL