I never thought making the back button go back with one tap would be so difficult, but for some reason, the navigation drawer insists that if it didn't get time to finish its closing animation, then the back press is for it.
I may have fixed it, but I'm not exactly sure...