(1) Background: Merkel cell carcinoma (MCC) is caused by the Merkel cell polyomavirus and UV radiation. Understanding of the underlying biology is limited, but identification of prognostic markers may lead to better prognostic stratification for the patients. (2) Methods: Ninety patients diagnosed with MCC (1996–2012) were included. Virus status was estimated by polymerase chain reaction (qPCR) and immunohistochemistry (IHC). Ulceration status, PD-L1, cd66b neutrophils, cd8 lymphocytes and biomarkers of vascularization (cd34 endothelial cells) and migration (e-cadherin) were estimated by IHC and analyzed with digital pathology. (3) Results: Virus was present in 47% of patient samples and correlated with lower E-cadherin expression (p = 0.0005), lower neutrophil-to-CD8 lymphocyte ratio (N:CD8 ratio) (p = 0.02) and increased PD-L1 expression (p = 0.03). Ulceration was associated with absence of virus (p = 0.03), increased neutrophil infiltration (p < 0.0001) and reduced CD8 lymphocyte infiltration (p = 0.04). In multivariate analysis, presence of virus (p = 0.01), ulceration (p = 0.05) and increased CD8 lymphocyte infiltration (p = 0.001) showed independent prognostic impacts on MCC-specific survival. (3) Conclusions: In this study, we found that a high N:CD8 ratio, ulceration, virus-negative status and absence of CD8 lymphocytes are negative prognostic markers. Accurate prognostic stratification of the patients may be important in the clinical setting for determination of adjuvant treatment.