My organisation is currently working on an SSO implementation, though from my understanding if you want to manage viewer access the security group using SAML would mean that at minimum you’re managing access for users that are intended to be viewers from 365 and the rest are ‘default access’ which translates to ‘managed within monday’ to make things less complex you woud likely create a security group for each permission set and start managing permisions for Monday within 365 itself.
IMO