An individual’s fear of hate crime victimization might be partially explained by direct experiences that influence their assessment of victimization risk. In some cases, though, fear of hate crime victimization is driven not by direct, personal experiences, but by historical and contemporary trauma suffered by those holding the targeted status. Using data from the 2019 nationally representative Experiences with Religious Discrimination Study (ERDS) survey, we show that part of Jewish and Muslim adults’ greater fears of victimization is explained by their past personal victimization experiences, their knowledge of close friends and family who have been victimized, and their greater religious visibility. Still, even after accounting for these factors, Jewish and Muslim adults report greater fears of religious hate crime victimization compared to Christian adults. We attribute this residual fear to the culture of anti-Semitism and Islamophobia within the United States and violence attributable to that culture, as well as the collective memory of historical religion-based victimization of Muslim and Jewish communities. These findings suggest the collective memory and knowledge of contemporary religious victimization may continue to affect Jewish and Muslim adults via a mechanism of fear, which has implications for scholarly and policy efforts to decrease religious victimization and its impact.