Computational co-creativity is an emerging area of research in co-creativity, that connects to user-centered research in design, human-computer interaction and artificial intelligence. A common vocabulary among researchers is critical for advancing research in such interdisciplinary work, but it is often difficult to achieve. The concept of agency is one of the topics of interest for the development of computational co-creative systems. While agency can be broadly defined as a sense of control over the creative outcomes of a co-creative process, there is no shared understanding of how to design systems for and with agency. In this paper, we reflect on the works we have developed over the last five years as a starting point for discussing the definition of agency. Further research will allow us to investigate how agency manifests itself in human-computer creative collaboration, also known as co-creativity. We believe that having a common ground allows for a more efficient process of analyzing and designing human-computer collaborative systems.