Tumor immunotherapy has resulted in dramatic effects in some cancer types, including curing of previously untreatable patients. However, the response rates are typically very heterogenous, with some patients showing dramatic responses whereas others do not or only barely respond. Consequently, there has been an ever-increasing research effort to better understand the factors that govern immunotherapy responsiveness and efficiency in order to identify predictive biomarkers and novel therapeutic targets. Clearly, traits of the tumor cells as well as aspects of the tumor microenvironment (TME) play an important role in this regard. However, a growing tumor not only interacts with cells in its immediate vicinity, but also reciprocally communicates with the entire host organism (and its microbiota). Thus, systemic influences on tumor growth and progression are likely to be similarly important as the microenvironment. In this review, we focus on various aspects of the “tumor organismal environment” (TOE), namely the lymphatic, the hematopoietic, the microbial, the neurogenic and the metabolic environment, and discuss their impact on tumor growth and immunotherapy.