In this paper, an improvement of a 2D stochastic model based handwritten entity recognition system is described. To model the handwriting considered as being a two dimensional signal, a context based, segmentation-free Hidden Markov Model (HMM) recognition system was used. The baseline approach combines a Markov Random Field (MRF) and a HMM so-called Non-Symmetric Half Plane Hidden Markov Model (NSHP-HMM). To improve the results performed by this baseline system operating just on low-level pixel information an extension of the NSHP-HMM is proposed. The mechanism allows to extend the observations of the NSHP-HMM by implanting structural information in the system. At present, the accuracy of the system on the SRTP (Service de Recherche Technique de la Poste) French postal check database is 87.52% while for the handwritten Bangla city names is 86.80%. The gain using this structural information for the SRTP dataset is 1.57%.