The long-term effect of the timing of fertility decline on population size