Immunotherapy in the form of immune checkpoint inhibitors (ICIs) has transformed the treatment landscape in numerous types of advanced cancer. However, the majority of patients do not benefit from this treatment modality. Although data are scarce, in general, patients with low-grade neuroendocrine tumours (NETs) do not benefit from treatment with ICIs in contrast to patients with neuroendocrine carcinoma, in which a small subgroup of patients may benefit. Low- and intermediate-grade NETs predominantly lack factors associated with response to ICIs treatment, like immune cell infiltration, and have an immunosuppressive tumour metabolism and microenvironment. In addition, because of its potential influence on the response to ICIs, major interest has been shown in the tryptophan-degrading enzymes indoleamine 2,3-dioxygenase (IDO) and tryptophan 2,3-dioxygenase (TDO). These enzymes work along the kynurenine pathway that deplete tryptophan in the tumour microenvironment. IDO and TDO are especially of interest in NETs since some tumours produce serotonin but the majority do not, which potentially deplete the precursor tryptophan. In this review, we summarize the current knowledge on the immune tumour microenvironment of neuroendocrine tumours and implications for treatment with immune checkpoint inhibitors. We also discuss (targetable) factors in the NET tumour microenvironment that potentially modulate the anti-cancer immune response.