Joint image filters transfer structural details from a guidance picture used as a prior to a target image, particularly for enhancing spatial resolution and suppressing noise. Previous methods based on convolutional neural networks (CNNs) combine nonlinear activations of spatially-invariant kernels to estimate structural details and regress the filtering result. In this paper, we learn instead sparse and spatially-variant kernels explicitly. We propose a CNN architecture, called a de-formable kernel network (DKN), that outputs sets of neighbors and their corresponding weights adaptively for each pixel. The filtering result is then computed as a weighted average. We also propose an efficient implementation that runs about 3000× faster than a brute-force one for an image of size 640 × 480. We demonstrate the effectiveness and flexibility of our model on the tasks of depth map upsampling, saliency map upsampling, cross-modality image restoration, and texture removal. In particular, we show that the weighted averaging process with sparsely sampled 3 × 3 kernels outperforms the state of the art by a significant margin. Our code and models are available online: https://github.com/jun0kim/DeformableKernelNetwork