Background Grade group 5 (GG5) prostate cancer (PCa) carries a less favorable prognosis after standard-of-care (SOC) therapy, necessitating novel therapeutic approaches. High-dose rate brachytherapy (HDRBT) and androgen deprivation therapy (ADT) may modulate immune response in GG5 PCa, particularly in tumors with increased immune content. This study evaluated whether the addition of nivolumab to SOC was associated with improved disease control in patients with high-volume GG5 PCa, including those with oligometastatic disease. Methods In this non-randomized phase II trial, 31 patients with localized or oligometastatic GG5 PCa and >30% positive biopsy cores were evaluated between September 2018 and April 2021. Patients received four doses of nivolumab (240 mg every 2 weeks) beginning 4 weeks prior to HDRBT, alongside ADT, HDRBT, and external beam radiation. The primary endpoint was to evaluate whether the 2-year freedom from biochemical recurrence (FFBR) rate would exceed a prespecified historical control rate of 75%. Results Among the 31 patients, the median follow-up was 38.8 months (IQR 31.0–46.5 months). The addition of nivolumab to SOC RT with ADT was associated with a 2-year FFBR rate of 90.3% (95% CI 74.3% to 98.0%) (median FFBR not reached), exceeding the prespecified historical control rate of 75% (one-sided p value from binomial test=0.024). Definitive and probable nivolumab-related toxicity included 6.3% acute grade 2 and 6.3% acute grade 3 adverse events (AEs), with no grade 4+ AEs observed. A higher Decipher immunosuppression score at diagnosis correlated with early pathologic response (p=0.005) and was independently associated with time to metastatic failure (p=0.044). Conclusions Nivolumab combined with SOC was associated with encouraging FFBR in this high-risk GG5 PCa population and may represent a promising therapeutic intensification strategy. The Decipher immunosuppression score may serve as a predictive biomarker for response. These findings warrant further investigation in randomized trials.