If you aren’t going to play with gloves, it mainly is increasing risk for Hep C and bacterial infections like shigella (among others).
You should be using an appropriately antiseptic hand wash before washing and between sessions, with as hot as water as you can find and tolerate.