First, the words "majority government" didn't appear on anyone's ballot.
Second, voters do elect individual MPs (whether they think about it like that or not).
Third, imagine an inverse scenario. Say the Liberals won 174 seats. And then three left to join the Conservatives. Would that be wrong?