Yeah, immature males and adult females can look alike. I can't tell the difference. During migration, I just assume they're female.
Key features are 2 white bars on the wing (adult male has 1), and yellow tail. Head and back can be speckled or not.
(Photo from Cornell Ornith, All About Birds.)