Most head and neck squamous cell carcinomas (HNSCCs) are caused by lifestyle, such as cigarette smoking, or by viruses, such as human papillomavirus (HPV) and Epstein–Barr virus (EBV). HNSCC remains a clinical challenge, notwithstanding the improvements observed in the past years, involving surgery, radiotherapy, and chemotherapy. Recurrent/metastatic (R/M) disease represents an unmet clinical need. Immunotherapy has improved the prognosis of a small proportion of these patients, but most still do not benefit. In the last decade, several preclinical and clinical studies have explored the HNSCC tumor immune microenvironment (TIME), identifying important differences between smoking-associated and virus-associated HNSCCs. This review aims to present how different etiologies affect the HNSCC TIME, affecting immune escape mechanisms and sensitivity to immunotherapy.