The topological phase and optical properties of LuNiBi bulk and nano-layer are studied. The self-consistent calculations are carried out in presence of spin-orbit coupling, based on density functional theory using generalized gradient approximation, local density approximation, Engle-Vosco generalized gradient approximation and modified Becke and Johnson potential. The effect of hydrostatic and uniaxial pressures on band order, energy band gap and optical properties of LuNiBi compound is investigated within different approaches. The topological phase of surface states and the optical properties of LuNiBi nano-layer are studied. The results show that the topological phase and energy gap of LuNiBi bulk and nano-layer are sensitive to pressure and type of exchange-correlation potential. There is an anisotropic response of LuNiBi nano-layer to the incident electromagnetic waves polarizations, parallel and perpendicular to the nano-layer surface. (C) 2017 Elsevier B.V. All rights reserved.