The aggregation properties of conjugated polymers can play a crucial role in their thin film structures and performance of electronic devices. Control of these aggregated structures is particularly important in producing efficient all-polymer solar cells (all-PSCs), considering that strong demixing of the polymer donor and polymer acceptor typically occurs during film formation because of the low entropic contribution to the thermodynamics of the system. Here, three naphthalenediimide (NDI)-based polymer acceptors with different backbone chlorination patterns are developed to investigate the effect of the chlorination patterns on the aggregation tendencies of the polymer acceptors, which greatly influence their crystalline structures, electrical properties, and device performances of the resultant all-PSCs and organic field-effect transistors (OFETs). The counterparts of NDI units, dichlorinated bithiophene (Cl2T2), monochlorinated bithiophene (ClT2), and dichlorinated thienylene-vinylene-thienylene (Cl2TVT), are employed to synthesize a series of P(NDIOD-Cl2T2), P(NDIOD-ClT2), and P(NDIOD-Cl2TVT) polymers. The P(NDIOD-Cl2T2) polymer takes advantage of strong noncovalent bonding induced by its chlorine substituents, resulting in the formation of optimal face-on oriented crystalline structures which are suitable for efficient all-PSC devices. In comparison, the P(NDIOD-Cl2TVT) polymer forms bimodal crystalline structures in thin films to yield optimal performances in the resultant OFETs. When the three chlorinated polymers are applied to all-PSCs with the PBDTTTPD polymer donor, P(NDIOD-Cl2T2) achieves a maximum power conversion efficiency (PCE) of 7.22% with an appropriate blend morphology and high fill factor, outperforming P(NDIOD-ClT2) (PCE = 4.80%) and P(NDIOD-Cl2TVT) (PCE = 5.78%). Our observations highlight the effectiveness of the chlorination strategy for developing efficient polymer acceptors and demonstrate the important role of polymer aggregation in modulating the blend morphology and all-PSC performance.