Situated face-to-face interactions are dynamic processes, building our social role and our relationships. In the present study, the bond resulting from these communicational processes is called the socio-affective glue, which is expected to be the element to recognise in our ASR prototype. This system is motivated to follow the relation changing instead of being accurate to distinguish the lexical cues and their semantics, to adapt the possible feedbacks giving the impression of a socio-affective intelligence in the perspective of a global incremental and iterative dialogue system (SASI) that can ensure the communication continuity. This approach uses the observations of the EEE corpus, composed by spontaneous and ecological interactions between socio-isolated elderly and a Smart Home's butler robot, which is activating automation through vocal commands.