With Brewster angle autocorrelation spectroscopy involving a combination of Brewster angle microscopy and autocorrelation technique we present quantitative measurements of the tilt angle in octadecanol carried out at the triple point of the next nearest neighbor tilted L’(2), the nontilted destorted hexagonal LS(Rot I), and the hexagonal LS(Rot II) phases. We show that the transition from the tilted phase to the nontilted phases, which changes from first order (L’(2)/LS(Rst I)) to second order (L’(2)/LS(Rot II)), is associated with strong hysteresis effects in the L’(2) phase, leading to an ambiguity of the tilt angle in the vicinty of the triple point. The behavior gives indications for a hindered first-order phase transition within the L’(2) phase.