Recognition of single-stranded DNA (ssDNA) or single-stranded RNA (ssRNA) is important for many fundamental cellular functions. A variety of single-stranded DNA-binding proteins (ssDBPs) and single-stranded RNA-binding proteins (ssRBPs) have evolved that bind ssDNA and ssRNA, respectively, with varying degree of affinities and specificities to form complexes. Structural studies of these complexes provide key insights into their recognition mechanism. However, computational modeling of the specific recognition process and to predict the structure of the complex is challenging, primarily due to the heterogeneity of their binding energy landscape and the greater flexibility of ssDNA or ssRNA compared with double-stranded nucleic acids. Consequently, considerably fewer computational studies have explored interactions between proteins and single-stranded nucleic acids compared with protein interactions with double-stranded nucleic acids. Here, we report a newly developed energy-based coarse-grained model to predict the structure of ssDNA-ssDBP and ssRNA-ssRBP complexes and to assess their sequence-specific interactions and stabilities. We tuned two factors that can modulate specific recognition: base-aromatic stacking strength and the flexibility of the single-stranded nucleic acid. The model was successfully applied to predict the binding conformations of 12 distinct ssDBP and ssRBP structures with their cognate ssDNA and ssRNA partners having various sequences. Estimated binding energies agreed well with the corresponding experimental binding affinities. Bound conformations from the simulation showed a funnel-shaped binding energy distribution where the native-like conformations corresponded to the energy minima. The various ssDNA-protein and ssRNA-protein complexes differed in the balance of electrostatic and aromatic energies. The lower affinity of the ssRNA-ssRBP complexes compared with the ssDNA-ssDBP complexes stems from lower flexibility of ssRNA compared to ssDNA, which results in higher rate constants for the dissociation of the complex (koff) for complexes involving the former.