Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

An Optimal Transport Kernel for Feature Aggregation and its Relationship to Attention

Grégoire Mialon 1 Dexiong Chen 1 Alexandre d'Aspremont 2 Julien Mairal 1
1 Thoth - Apprentissage de modèles à partir de données massives
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann
2 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique de l'École normale supérieure, CNRS - Centre National de la Recherche Scientifique, Inria de Paris
Abstract : We introduce a kernel for sets of features based on an optimal transport distance, along with an explicit embedding function. Our approach addresses the problem of feature aggregation, or pooling, for sets that exhibit long-range dependencies between their members. More precisely, our embedding aggregates the features of a given set according to the transport plan between the set and a reference shared across the data set. Unlike traditional hand-crafted kernels, our embedding can be optimized for a specific task or data set. It also has a natural connection to attention mechanisms in neural networks, which are commonly used to deal with sets, yet requires less data. Our embedding is particularly suited for biological sequence classification tasks and shows promising results for natural language sequences. We provide an implementation of our embedding that can be used alone or as a module in larger learning models. Our code is freely available at https://github.com/claying/OTK.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

Cited literature [48 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02883436
Contributor : Grégoire Mialon <>
Submitted on : Monday, June 29, 2020 - 10:33:00 AM
Last modification on : Friday, July 10, 2020 - 7:48:03 AM

File

main_otk.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02883436, version 1

Collections

Citation

Grégoire Mialon, Dexiong Chen, Alexandre d'Aspremont, Julien Mairal. An Optimal Transport Kernel for Feature Aggregation and its Relationship to Attention. 2020. ⟨hal-02883436⟩

Share

Metrics

Record views

519

Files downloads

10