Can I block every number except those in my contacts list?

No.

There is a limit to how many numbers can be blocked, but that limit is not documented anywhere. Simple Call Blocker’s Status screen will display whether or not the blocker loaded successfully.

On iOS 13 and later, you can turn on “Silence Unknown Callers” to achieve this.