Our heliosphere is formed by the interaction between the dynamic solar wind (SW) and partially ionized, local interstellar medium. “Pickup ions” (PUIs) are produced by charge exchange between SW ions and interstellar neutral atoms and propagate with the SW into the outer heliosphere. Neutralization of these PUIs in the heliosheath creates energetic neutral atoms (ENAs) that are observed near Earth by the Interstellar Boundary Explorer (IBEX). In this study, we analyze the evolution of ENAs observed by IBEX between ~0.5 and 6 keV from 2009 to 2018, presenting the evolution of ENA spectral indices as a function of latitude in different regions of the sky. Our results show that IBEX ENA spectra reflect unique differences in the northern and southern polar coronal holes (PCHs) as a function of latitude and time. ENA spectra reflect a delayed closure of the southern PCH compared to the north by ~1–2 yr in solar cycle 24, but then a rapid opening of the southern PCH beginning ~1–2 yr later, in agreement with evolving SW speeds emanating from the PCHs. While ENA spectra from the front and flanks of the heliosphere exhibit similar behavior, ENA spectra from the back behave differently, suggesting that the heliotail and ENA source structures are significantly different. The mean ENA spectral index observed by IBEX evolves gradually and quasi-periodically over the ~11 yr solar cycle, reflecting large-scale changes in the PCHs and SW speed. The standard deviation of the spectral index exhibits bumps at times when the SW dynamic pressure changes abruptly.