I can try do the terminfo way of setting keybindings my config. I eventually went with this route but it really doesn't make a pretty home-manager config. I wonder why I haven't seen more people with this issue. Perhaps people don't use vim mode in zsh, which did seem to aggravate the issue