In this paper the problem of adaptive observer design in the presence of disturbances is studied, and an augmented adaptive observer is proposed. First, the $H_{\infty}$ gain of a conventional adaptive observer is estimated, which characterizes the effect of disturbances on output errors. Next, it is shown that if the disturbance is ''matched'' in the plant equations, then it is possible to introduce additional sliding-mode feedback, dependent on the plant output, improving the accuracy of observation. Simulation results confirm the improvement.