We consider the coupled process of phase separation and dewetting of metal alloys of nanoscale thickness deposited on solid substrates. The experiments involve applying nanosecond laser pulses that melt the Ag40Ni60 alloy films in two setups: either on thin supporting membranes or on bulk substrates. These two setups allow for extracting both temporal and spatial scales on which the considered processes occur. The theoretical model involves a longwave version of the Cahn-Hilliard formulation used to describe spinodal decomposition, coupled with an asymptotically consistent longwave-based description of dewetting that occurs due to destabilizing interactions between the alloy and the substrate, modeled using the disjoining pressure approach. Careful modeling, combined with linear stability analysis and fully nonlinear simulations, leads to results consistent with the experiments. In particular, we find that the two instability mechanisms occur concurrently, with the phase separation occurring faster and on shorter temporal scales. The modeling results show a strong influence of the temperature dependence of relevant material properties, implying that such a dependence is crucial for the understanding of the experimental findings. The agreement between theory and experiment suggests the utility of the proposed theoretical approach in helping to develop further experiments directed toward formation of metallic alloy nanoparticles of desired properties.