Worker roles aren't meant to be deleted, even if they were added to a worker by mistake. That's because once a worker had that role even briefly, they had the permissions that came with it. We need to keep a record to show they had those permissions in case they made any changes during that time. Keeping roles added in error helps maintain an accurate audit trail.
So, it's intentional that users can't delete worker roles. Just setting an end date is enough, it removes the permissions without breaking the audit history.