Idea: compose safe skills instead of combining safety rewards. First learn preference-aligned skills that capture different user preferences from crowd data. Then compose these skills to solve new downstream tasks. The high-level policy solves the task. The low-level skills provide safety.
🧵6/7
