The use of androgen deprivation therapy and second-line anti-androgens in prostate cancer has led to the emergence of tumors employing multiple androgen receptor (AR)-dependent and AR-independent mechanisms to resist AR-targeted therapies in castration-resistant prostate cancer (CRPC). While the AR signaling axis remains the cornerstone for therapeutic development in CRPC, a clearer understanding of the heterogeneous biology of CRPC tumors is needed for innovative treatment strategies. In this review, we discuss the characteristics of CRPC tumors that lack AR activity and the temporal and spatial considerations for the conversion of an AR-dependent to an AR-independent tumor type. We describe the more prevalent treatment-emergent phenotypes arising in the CRPC disease continuum, including amphicrine, AR-low, double-negative, neuroendocrine and small cell phenotypes. We discuss the association between the loss of AR activity and tumor plasticity with a focus on the roles of transcription factors like SOX2, DNA methylation, alternative splicing, and the activity of epigenetic modifiers like EZH2, BRD4, LSD1, and the nBAF complex in conversion to a neuroendocrine or small cell phenotype in CRPC. We hypothesize that only a subset of CRPC tumors have the propensity for tumor plasticity and conversion to the neuroendocrine phenotype and outline how we might target these plastic and emergent phenotypes in CRPC. In conclusion, we assess the current and future avenues for treatment and determine that the heterogeneity of CRPCs lacking AR activity will require diverse treatment approaches.