Over the past two decades, colloidal particles with a variety of shapes, sizes, and compositions have been synthesized and characterized successfully. One of the most important applications for colloidal building blocks is to engineer functional structures as mechanical, electrical, and optical metamaterials. However, complex interaction dynamics between the building blocks as well as sophisticated structure-property relationships make it challenging to design structures with predictable target properties. In this paper, we implement an inverse material design framework using Genetic Algorithm (GA)-based techniques to streamline the design of colloidal structures based on target properties. We investigate spherical particles as well as colloidal molecules of different sizes and shapes and evaluate a Geometric Landscape Accessibility parameter that identifies the size of feasible domains within the geometric phase space of each structure. Considering target photonic properties, our GA-assisted framework is further utilized to identify sets of building blocks and structures that lead to various target values for the size of the photonic band gaps. The proposed framework in this study will provide new insight for predictive computational material design approaches and help establish more efficient ways of understanding structure-property relations in sub-micrometer-scale materials.