This paper emphasizes the link between parallel asynchronous distributed computations (PADC) and Markov Decision Processes (MDPs), which are a powerful generic model for computing optimal control. We review some results arguing that reasonably small state space MDPs can be solved with PADC. We then propose a solution for extending these results when the state space is large. This shows that difficult optimal control problems have natural neural network-like solutions and suggests a general methodology for constructing neural networks.