Hydrogen can be produced by solar energy via photocatalytic water splitting. In the process, both hydrogen and oxygen are produced in the same environment. An inert gas (e.g. nitrogen) can be used in the reactor in order to eliminate the explosion hazards. In this case study, membrane cascade design was implemented to recover the hydrogen outside the flammability range using a commercial polyimide membrane from UB (R) Industries (Japan). Two different ternary (H-2, O-2, N-2) mixtures were investigated to recover and purify hydrogen outside the flammability range. Mixture (1) contains 4 mol-% hydrogen while Mixture (2) contains 6 mol-%. The process optimization is defined as a nonlinear programming (NLP). The aim of the objective function is to minimize the present value (PV) of all outgoing cash flows (no income) for the separation process with both flammability and recovery constraints to be satisfied. For a 100 kmol/h hydrogen feed basis, the results showed high investment and operating costs for Mixture (1) with a specific cost of 8.20 $/kg. The maximum recovery and purity that can be achieved, without further increase in cost, are 92% and 92.5%, respectively. Switching to Mixture (2) will result in a specific cost reduction of around 25% due to lower feed flow rates and higher molar concentrations of hydrogen. The maximum recovery is 90% with 95% purity. Beyond these values, the specific cost increase dramatically. This is the first fundamental process optimization considering safety issues as well as product recovery yield simultaneously from a tertiary gas mixture. (C) 2016 Hydrogen Energy Publications LLC. Published by Elsevier Ltd. All rights reserved.