I don't think Creo relations allow expressions using matrix or vector objects.
You will have to expand the matrix expression shown in your screenshot into its components and enter these these into the relation editor...
Also, keep in mind that in creo, for trigonometric functions like sin(theta), theta is in degrees!
