Time is the most precious, finite thing you have. True friends will respect the ways you use it even when that means you don't get to interact as much as they'd hope.
Time shared is a gift, never an obligation. So be careful - anyone telling you that you owe them your time likely isn't a friend.