You can exercise the duty and the power that Congress has to remove the President any time you want. All you need to do is to get enough of the country to agree with your opinion of him.
The fact that you can't isn't a flaw in the Constitution. It's a flaw in your opinion.