Might be worth noting that this is a huge swing from a bygone era of high infant and child death, such that women were expected to have children early and often in hopes that they could outperform the mortality rate. Population rates in Japan had been low and relatively flat for centuries. Then the industrial revolution and modern medicine dramatically reduced mortality rates, causing populations to climb rapidly for around a century.
Now we're settling into a new normal of sub-replacement rate births (not no births by any stretch, just births slower than the post-40s boom years) and everyone's freaking out like Japan won't exist in another generation.
The Japanese people could likely support a higher population via socialist public policy. But they could also just have a smaller population going into the 21st century. It's not like 123M is a magic number the nation needs to persist. If Japan's population fell into the 80M mark, what's the horrible thing that could happen? Koreans and Philippinos and Italians and Egyptians might be legally allowed to immigrate at last? Oh no!!!! Death of a nation!!!