Inter-Cell Interference Coordination (ICIC) is commonly identified as a key radio resource management mechanism to enhance system performance of 4G networks. This paper addresses the problem of ICIC in the downlink of cellular OFDMA (LTE and WiMAX) systems in the context of Self-Organizing Networks (SON). The problem is posed as a cooperative Multi-Agent control problem. Each base station is an agent that dynamically changes power masks on a subset of its bandwidth to control interference it produces to its neighbouring cells. The agent learns the optimal coordinated power allocation strategy using information from its own and its neighbouring cells. A Fuzzy Inference System (FIS) is used to handle continuous state space defined by the input quality indicators to the controller performing the ICIC. The FIS is optimized using Reinforcement Learning (RL) with a Fuzzy Q-Learning (FQL) implementation. Simulation results illustrate the important performance gain brought about by the proposed ICIC scheme.