Natural selection at one site shapes patterns of genetic variation at linked sites. Quantifying the effects of “linked selection” on levels of genetic diversity is key to making reliable inference about demography, building a null model in scans for targets of adaptation, and learning about the dynamics of natural selection. Here, we introduce the first method that jointly infers parameters of distinct modes of linked selection, notably background selection and selective sweeps, from genome-wide diversity data, functional annotations and genetic maps. The central idea is to calculate the probability that a neutral site is polymorphic given local annotations, substitution patterns, and recombination rates. Information is then combined across sites and samples using composite likelihood in order to estimate genome-wide parameters of distinct modes of selection. In addition to parameter estimation, this approach yields a map of the expected neutral diversity levels along the genome. To illustrate the utility of our approach, we apply it to genome-wide resequencing data from 125 lines in Drosophila melanogaster and reliably predict diversity levels at the 1Mb scale. Our results corroborate estimates of a high fraction of beneficial substitutions in proteins and untranslated regions (UTR). They allow us to distinguish between the contribution of sweeps and other modes of selection around amino acid substitutions and to uncover evidence for pervasive sweeps in untranslated regions (UTRs). Our inference further suggests a substantial effect of other modes of linked selection and of adaptation in particular. More generally, we demonstrate that linked selection has had a larger effect in reducing diversity levels and increasing their variance in D. melanogaster than previously appreciated., Author Summary One of the major discoveries in modern population genetics is the profound effect that natural selection on one locus can have on genetic variation patterns at linked loci. Since the first evidence for linked selection was uncovered in Drosophila melanogaster over two decades ago, substantial effort has focused on quantifying the effects and on distinguishing the relative contributions of purifying and positive selection. We introduce an approach to jointly model the effects of positive and negative selection along the genome and infer selection parameters. To this end, we consider how closely linked each neutral site is to different types of annotations and substitutions. When we apply the inference method to genome-wide data from 125 D. melanogaster lines, our model explains most of the variance in diversity levels at the megabase scale and allows us to distinguish among the contribution of different modes of selection on proteins and UTRs. More generally, we provide a map of the effects of natural selection along the genome, and show that selection at linked sites has had an even more drastic effect on diversity patterns than previously appreciated. We also make a tool available to apply this approach in other species.