The color discrimination capacity of color vision normal observers can be assessed using color aptitude tests (CATs), although data on the test/retest reliability and inter-test agreement of such tests using appropriate statistical measures is limited. In this study, 32 color vision normal observers twice performed each of five CATs, being the Farnsworth-Munsell 100 hue test, HVC Color Vision Skills Test, inter-Society Color Council Color Aptitude Test, the Nagel Anomaloscope (Nagel), and a custom designed two-color discrimination test. Two methods of determining the Nagel matching range were used based on the initially accepted matching range or on repeated measures. The test/retest performance of each test was determined using the method outlined by Bland and Altman[Lancet 1986;1:307-310] and the inter-test agreement was analyzed using the same method, after converting test scores into z score units. No test was found to have a significant systematic alteration in test performance between test and retest. Test/retest performance was generally poor, indicating that tests can only reliably classify observers into very broad performance bands. All tests failed to show inter-test agreement, and our data indicate a primary reason for this: as test/retest performance is generally poor, inter-test agreement must also be poor as no test can agree more with another test than it agrees with itself. (c) 2014 Wiley Periodicals, Inc. Col Res Appl, 40, 224-231, 2015