In this paper we present a self-organization technique to manage interference in femtocell networks. We model femtocells as a multiagent system implementing a form of Reinforcement Learning (RL) known as Q-Learning (QL) to solve the aggregated interference problem originated due to the macro-femtocell systems coexistence. We discuss this approach and propose a modification in order to solve some of the potential implementation limits of the QL algorithm. In order to achieve a more accurate environment representation and therefore a more appropriate femtocell system behavior, we combine Fuzzy logic with QL, in the form of Fuzzy Q-Learning (FQL), which allows agents to represent continuously the state and action spaces. Results are presented for the proposed QL and FQL algorithms as well as for two other FQL approaches with less complex perceptions of the surrounding environment and targets and a smart power control proposed by the 3GPP standardization body.