About Few-shot Learning

By LI Haoyang 2020.10.13

unfinished

Content

Datasets

MiniImageNet

Dataset: https://drive.google.com/file/d/1HkgrkAwukzEZA0TpO7010PkAOREb2Nuk/view

CIFAR 100

CUB

Approaches

MatchNet - NIPS 2016

Paper: http://papers.nips.cc/paper/6385-matching-networks-for-one-shot-learning

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, Daan Wierstra. Matching Networks for One Shot Learning. NIPS 2016.

Humans learn new concepts with very little supervision – e.g. a child can generalize the concept of “giraffe” from a single picture in a book – yet our best deep learning systems need hundreds or thousands of examples. This motivates the setting we are interested in: “one-shot” learning, which consists of learning a class from a single labeled example.

In contrast, many non-parametric models allow novel examples to be rapidly assimilated, whilst not suffering from catastrophic forgetting

We aim to incorporate the best characteristics from both parametric and non-parametric models – namely, rapid acquisition of new examples while providing excellent generalization from common examples.

Model

Given a small support set $S$, their model defines a function $c_{S}$ (or classifier) for each $S$, i.e., a mapping $S\to c_{S}(\cdot)$.

They wish to map from a small support set of $k$ examples of input-label pairs $S=\{(x_i,y_i)\}_{i=1}^k$ to a classifier $c_S(\hat{x})$. When a test sample $\hat{x}$ is given to the classifier, it defines a probability distribution over outputs $\hat{y}$:

When a new support set of examples $S^{'}$ from which to one-shot learn is given, the parametric neural network defined by $P$ can be simply used to make predictions about the appropriate label distribution $\hat{y}$ for each test example $\hat{x}$:

Perhaps, the $y_i$ here is one-hot coded ?

In which, $a$ is an attention mechanism, subsuming both KDE and kNN methods:

• $a$ is an $X\times X$ kernel $\sim$kernel density estimator
• $a$ is zero for the $b$ furthest $x_i$ from $\hat{x}$ $\sim$ 'k-b' nearest neighbor classifier.

In a more casual interpretation, $a$ computes the similarity of $\hat{x}$ and $x_i$.

The simplest form $a$ can be a cosine similarity of embedded inputs:

In which, $f$ and $g$ are embedding functions are neural networks.

In instantiation, they wish the embedding function $g$ to learn from the whole support set $S$, the exact instantiation of this model is:

Training strategy

The training process aims to obtain the parameters $\theta$ such that:

In which, a task $T$ is defined as a distribution over possible label sets $L$, i.e., a label set $L$ is sampled from task $T$; the batch $B$ and support set $S$ are sampled from the label set $L$.

The obtained $\theta$ yields a model that works well when sampling $S^{'}\sim T^{'}$ from a different distribution of novel labels.

In human words, the inner part aims to obtain a model that performs well in classifying samples from batch based on samples from support set, the outer part aims to make the inner part performs well in various subsets of the entire task set.

N-way K-shot Task

All of our experiments revolve around the same basic task: an N-way k-shot learning task. Each method is providing with a set of k labeled examples from each of N classes that have not previously been trained upon. The task is then to classify a disjoint batch of unlabeled examples into one of these N classes. Thus random performance on this task stands at 1/N.

Performance

Omniglot [14] consists of 1623 characters from 50 different alphabets. Each of these was hand drawn by 20 different people. The large number of classes (characters) with relatively few data per class (20), makes this an ideal data set for testing small-scale one-shot classification.

ImageNet is a notoriously large data set which can be quite a feat of engineering and infrastructure to run experiments upon it, requiring many resources. Thus, as well as using the full ImageNet data set, we devised a new data set – miniImageNet – consisting of 60, 000 colour images of size 84 × 84 with 100 classes, each having 600 examples

Inspiration

This paper defines a new challenging and interesting task, i.e. N-way K-shot tasks. They proposed a model to meta-learn the generic metrics between samples and use a kNN-like inference to classify test images.

Diversity Transfer Network - AAAI 2020

Code: https://github.com/Yuxin-CV/DTN

Mengting Chen, Yuxin Fang, Xinggang Wang, Heng Luo, Yifeng Geng, Xinyu Zhang, Chang Huang, Wenyu Liu, Bo Wang. Diversity Transfer Network for Few-Shot Learning. AAAI 2020. arXiv:1912.13182

The idea is to use the diversity of the larger training dataset to augment the few-shot dataset by generating new samples with more diversity.

N-way K-shot task

Given two datasets with disjoint label spaces $\cal{D}_{train}$ and $\cal{D}_{test}$, the problem is to take $N$ classes, each with $K$ images, of samples to form a support set $\{(X_{s}^{n,k},Y_{s}^{n,k})\}$ from the training/testing set:

A query sample $(X_q,Y_q)$ is then sampled from the rest images of the $N$ classes:

The goal is to classify the query into one of the $N$ classes correctly based only on the support set and the prior meta-knowledge learned from the training set $\cal{D}_{train}$.

The problem seems to be defined differently from that in MatchNet.

Method

The structure consists of a meta-learning branch and an auxiliary branch.

All of these images are firstly mapped into normalized feature vectors. The diversity between $H$ paired reference features $\bold{z}_{r}^{h,j}\in\Bbb{R}^C,j\in[1,2],h\in[1,2,\dots,H]$ , measured by transformed subtraction, are used to augment support features $\bold{z}_s^{n}$:

For $K$ support images, the augmented dataset will have $K\times (H+1)$ generated samples.

The meta-classifier is an $N$-way classifier, represented by a matrix $\bold{W}\in\Bbb{R}^{N\times C}$, in which each row $\bold{w}_n\in\Bbb{R}^C$ can be viewed as a proxy of the $n$-th category:

It's the normalized center of the cluster of $n$-th category.

The prediction of the fed extracted query feature $\bold{z}_q$ is:

And during training, the loss of meta-task parameterized with $\alpha$ is calculated as:

By minimizing this loss, it increases the cosine similarity between the query feature and corresponding query proxy.

As the proxy is determined by the augmented datasets, does the optimizer solely update $\alpha$ in the loss function? It seems to be ditched in inference.

The auxiliary is a meta-classifier trained on mini batches (i.e. $\{(X_i,Y_i)\},i\in[1,2,\dots,B]$) sampled from $\cal{D}_{train}$, optimized according to the same form of loss function:

In which, $z_i=F(X_i)$ and $\bold{w}_n^{'}\in\Bbb{R}^C$ is the $n$-th row of $\bold{W}^{'}$.

The auxiliary branch is designed to accelerate the optimization of feature extractor.

A Organized Auxiliary task co-Training (OAT) strategy is proposed to organize auxiliary branch and meta-learning branch.

They select $T$ training epochs to form one training unit $\cal{U}$, the $i$-th training unit $\cal{U}_i$ has $\gamma_i$ meta-training epochs ($\cal{M}$) and $(T-\gamma_i)$ auxiliary training epochs ($\cal{A}$). With $L$ training units, the whole training sequence $\cal{S}$ is:

In human words, there are $L$ steps of training, in each step, there are $(T-\gamma_i)$ auxiliary epochs and $\gamma_i$ meta-learning epochs. The whole process is intertwined.

Experiment settings

Three datasets, miniImageNet, CIFAR100 and CUB, are used for evaluation.

The feature extractor for DTN is a CNN with 4 convolutional modules, each of which contains a $3\times 3$ convolutional layer with 64 channells followed by a batch normalization layer, a ReLU non-linearity layer and a $2\times 2$ max-pooling layer. The output is a feature vector of 1024 dimensions.

The mapping function $\phi_i$ is a fully-connected layer with 2048 units followed by a leaky ReLU activation (max(x,0.2x)) layer and a dropout layer with 30% dropout rate.

The mapping function $\phi_2$ has the same settings with $\phi_1$ except that the number of units of the FC layer is 1024.

Performance

Visualization

As shown above, the generated samples are roughly clustered in the cloud of real samples of a larger dataset, which indicates the effectiveness of the proposed augmentation.

Ablation study

The results gradually become better as the number of generated features increases. No improvement was observed when the number of generated features exceeds 64. We attribute this to the fact that 64 generated features have been well fitted to the real sample distribution.