In my experience, no one pushes past your boundaries.
Swingers is all about consent so if you don’t want to do anything, no one can force you to.
If someone does try to manoeuvre around your boundaries, you can always report to the event security as well.