What is the definition and operation of [row,col]

What exactly do you mean? Which "operation"? In what context?

Mathcad and Prime offer functions named "rows()" and "cols()" which return the number of rows resp. number of columns of a matrix.

There are no "operations" in Mathcad that would use square brackets around their arguments as in "[row,col]" which you cited.