Solve , maximum of area of triangle ?

This is actually a very simple problem that can be solved straightforward in a variety of ways.

Let O be the center of the circle and, without loss of generality, set the radius of the circle to 1 (AB=2).

Then you could for example very easily express the area as a function of the distance OD.

Or you could express it as a function of the angle DOC, which is also very easy to do.

Both require just the application of the Pythagorean theorem.

Once you have an area function in one variable, you have several options in Mathcad for determining the maximum area.
The classic method would be to set the first derivative to zero to determine the corresponding independent variable and then plug it into the area function.

The equation could be solved either symbolically using "solve" or numerically by using either the "root" function or a solve block with "find".  Or you could do (numerically) without the derivative and use the "maximize" function.

Example of using the distance OD as parameter

Example of using the angle phi = DOC as parameter

If you still experience problems solving this task I would suggest that you show what you had tried and attach your worksheet.