This paper is devoted to the study of the Hausdorff dimension of the singular set of the minimum time function $T$ under controllability conditions which do not imply the Lipschitz continuity of $T$. We consider first the case of normal linear control systems with constant coefficients in $\mathbb{R}^N$. We characterize points around which $T$ is not Lipschitz as those which can be reached from the origin by an optimal trajectory (of the reversed dynamics) with vanishing minimized Hamiltonian. Linearity permits an explicit representation of such set, that we call $\mathcal{S}$. Furthermore, we show that $\mathcal{S}$ is $\mathcal{H}^{N-1}$-rectifiable with positive $\mathcal{H}^{N-1}$-measure. Second, we consider a class of control-affine \textit{planar} nonlinear systems satisfying a second order controllability condition: we characterize the set $\mathcal{S}$ in a neighborhood of the origin in a similar way and prove the $\mathcal{H}^1$-rectifiability of $\mathcal{S}$ and that $\mathcal{H}^1(\mathcal{S})>0$. In both cases, $T$ is known to have epigraph with positive reach, hence to be a locally $BV$ function (see \cite{CMW,GK}). Since the Cantor part of $DT$ must be concentrated in $\mathcal{S}$, our analysis yields that $T$ is $SBV$, i.e., the Cantor part of $DT$ vanishes. Our results imply also that $T$ is locally of class $\mathcal{C}^{1,1}$ outside a $\mathcal{H}^{N-1}$-rectifiable set. With small changes, our results are valid also in the case of multiple control input.