A key issue when designing clinical trials is the estimation of the number of subjects required. Assuming for multicenter trials or biomarker-stratified designs that the effect size between treatment arms is the same among the whole study population might be inappropriate. Limited work is available for properly determining the sample size for such trials. However, we need to account for both, the heterogeneity of the baseline hazards over clusters or strata but also the heterogeneity of the treatment effects, otherwise sample size estimates might be biased. Most existing methods account for either heterogeneous baseline hazards or treatment effects but they dot not allow to simultaneously account for both sources of variations. This article proposes an approach to calculate sample size formula for clustered or stratified survival data relying on frailty models. Both theoretical derivations and simulation results show the proposed approach can guarantee the desired power in worst case scenarios and is often much more efficient than existing approaches. Application to a real clinical trial designs is also illustrated.