A neural network based optimal control synthesis approach is presented for systems modeled by partial differential equations. The problem is formulated via discrete dynamic programming and the necessary conditions of optimality are derived. For synthesis of the controller. we propose two sets of neural networks: the set of action networks captures the mapping between the state and control, while the set of critic networks captures the mapping between the state and costate. We illustrate the solution process with a parabolic equation involving a nonlinear term. For comparison, we consider the linear quadratic regulator problem for the diffusion equation, for which the Ricatti-operator based solution is known. Results show that this adaptive-critic based systematic approach holds promise for obtaining the optimal control design of both linear and nonlinear distributed parameter systems.