We present a comprehensive, ground-state density functional theory study of the size dependence of the optical and electronic properties and the stability of spherical silicon nanocrystals (NCs) with different impurities on the surface. We vary the size of the NCs from 1.0 to 3.5 nm, considering single-bonded $({\mathrm{CH}}_{3}$, F, Cl, OH) and double-bonded (O, S) impurities and bridged oxygen. We show that the density of states (DOS) and absorption indices of the NCs with single-bonded impurities are very similar to each other and the fully hydrogenated NCs, except for the 1.0-nm NCs, where a slight difference is present. In the case of the NCs with double-bonded impurities, the DOS and absorption indices exhibit a significant difference, compared to the fully hydrogenated NCs, for sizes up to 2.5 nm. We argue that this difference arises from the difference in the contribution from the impurity to the states around the gap, which can considerably change the character of the states. We demonstrate that the double-bonded impurities contribute significantly to the states around the gap, compared to the single-bonded impurities, causing changes in the symmetry of these states. This observation was further supported by analyzing the changes of the Fourier transform of the charge densities of the highest occupied and lowest unoccupied eigenstate. We also show that the formation energies of NCs with bridged oxygen and fluorine are the lowest, regardless of the size. Furthermore, we show that high hydrogen concentration can be used to suppress the addition of oxygen and fluorine on the surface of the Si NCs.