Seizure events feature temporary abnormalities in muscle control or movements. They are usually caused by excessive neuronal activities in the brain, and are called epileptic seizures (ES). Nevertheless, not all seizures are epileptic in origin. Some are caused by psychological reasons, and such type of seizures are called psychogenic non-epileptic seizures (PNES). We propose a method to classify ES and PNES based on clinical signs in the seizure videos. In particular, inspired by BERT, we propose a Transformer-based framework that pre-trains on large unlabeled clinical videos, and then we fine-tune the pre-trained model for seizure classification with a minimum modification. We conduct a leave-one-subjectout (LOSO) validation on our dataset. The F1-score and accuracy are 0.82 and 0.75, respectively. To our knowledge, the proposed approach is the first attempt to use large unannotated data and learn useful representations for downstream tasks in the field of video based seizure analysis.