Verhaeren had hoped to make a triumphant departure after leading Australia's campaign in Japan, but decided against staying Down Under for an extra year to attend the rescheduled Games.
"I tried to look for ways to extend, but you can't compromise in a high performance environment, nor did I want to compromise my family," the highly respected coach said in a statement released by Swimming Australia.
Verhaeren, 51, was appointed in 2013 after Australia's poor results at the London Olympics a year earlier, a campaign marred by ill-discipline, drug use and drunkenness.
At the time, he was best known best known for guiding Dutch greats Pieter van den Hoogenband and Inge de Bruijn to Olympic success in Sydney and Athens.
He helped restore Australia as a swimming superpower, with the squad finishing second only to the United States at the 2015 and 2019 world titles.
"(He) has overseen significant change and development in a complex system, laying a strong foundation for future success," Swimming Australia chief executive Leigh Russell said.
She said Verhaeren was leaving with Swimming Australia's best wishes and had decided "it is the right time to put family needs above anything else".
He will be replaced by Australian Rohan Taylor, who is currently state coach for Victoria and Tasmania.
The pair will work together during a transition period before Verhaeren departs in September.