If you want to remove the action for everyone you can remove it from the action models where it is used. OOTB the Reassign action is used in the following action models:
Reassign:

Reassign Tasks:

If you want to remove it for certain users then you will need to create a custom validator to determine when the action should be enabled. OOTB the action is enabled by the validators highlighted above.
This article provides steps to remove a different action from its associated action models. The general process should be similar for removing these actions, if desired:
https://support.ptc.com/appserver/cs/view/solution.jsp?n=CS156342