cancel
Showing results for
Did you mean:
cancel
Showing results for
Did you mean:

Community Tip - Did you get an answer that solved your problem? Please mark it as an Accepted Solution so others with the same problem can find the answer easily. X

# How can we make a 3D Widget on HoloLens visible in front of me every time ?

The goal here is to get a some UI containing 3D Widgets (3dButtons , 3dImages , 3d Labels and 3d Models) in front of current HoloLens view. For example after saying "show UI" the UI elements should appear in front view e.g. distance of 1,5 meters. Example:

In the Tech Tip article "How to create a custom button pressable on HoloLens device?" is shown how to create a custom UI button. Now  based on the example of this project   the functionality will be extended by adding the possibility for dynamic display of the UI elements in front of the current view.

To be able to move the elements in front of us we need to calculated the current view matrix and the inverse matrix  of if.  For this goal we need to prepare some mathemtics function like , matrix inverse calculation, marix multiplication , matrix with vector multiplicaiton, vector product und dot vector product ... etc. When we sear in Internet there a lot of sources where we can find the informormation how to implement such mathematical tools.

• The first steps is to calculate the view matrix - e.g. some code:

``````...
function get_mat4x4_camera(eyepos,eyedir,eyeup)
{
// printVector(eyepos, "eyepos")
// printVector(eyedir, "eyedir")
// printVector(eyeup, "eyeup")
var mat4x4 =[[1.0,0.0,0.0,0.0],[0.0,1.0,0.0,0.0],[0.0,0.0,1.0,0.0],[0.0,0.0,0.0,1.0]];

var i=0;
var  eyeZaxis    	= this.vec_normilize(eyedir);
var  eyeYaxis  	= this.vec_normilize(eyeup);
var  eyeXaxis    	= this.vec_normilize(this.vec_product(eyeYaxis,eyeZaxis))

for (i=0;i<3;i++) mat4x4[i][0] = eyeXaxis[i];
for (i=0;i<3;i++) mat4x4[i][1] = eyeYaxis[i];
for (i=0;i<3;i++) mat4x4[i][2] = eyeZaxis[i];
for (i=0;i<3;i++) mat4x4[i][3] =   eyepos[i];
//for (i=0;i<3;i++) mat4x4[i][3] =  0.0-eyepos[i];
return mat4x4;
//return this.transpondMat(mat4x4);
}
////////////////////// vector with length =1.0
function vec_normilize(ary1) {
ret=[];
var len=this.vec_len(ary1);
for (var i = 0; i < ary1.length; i++)
ret[i]=ary1[i]/len;
return ret;
};
//vector product range =3
function vec_product(a, b) {

var vecprod =[0.0,0.0,0.0];
vecprod[0]=a[1]*b[2]-  a[2]*b[1];
vecprod[1]=a[2]*b[0]-  a[0]*b[2];
vecprod[2]=a[0]*b[1]-  a[1]*b[0];

return vecprod;
};``````

When we find programming code for mathematical operation we need to verify it by some examples where we know the results, because often there are some math sources which does not implement the correct solution.

I think a good resource are e.g.:

https://www.learnopencv.com/rotation-matrix-to-euler-angles/

https://en.wikipedia.org/wiki/Conversion_between_quaternions_and_Euler_angles

https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/Matrix_math_for_the_web

So that we do not need to re-invent the wheel, but we need to be careful to find the correct mathematical relations and to test and verify if they really work.

• Next step is to calculate the invert matrix of the view matrix.  A good implementation I found is this one:

``````function matrix_invert(M){
// I use Guassian Elimination to calculate the inverse:
// (1) 'augment' the matrix (left) by the identity (on the right)
// (2) Turn the matrix on the left into the identity by elemetry row ops
// (3) The matrix on the right is the inverse (was the identity matrix)
// There are 3 elemtary row ops: (I combine b and c in my code)
// (a) Swap 2 rows
// (b) Multiply a row by a scalar

//if the matrix isn't square: exit (error)
if(M.length !== M[0].length){return;}

//create the identity matrix (I), and a copy (C) of the original
var i=0, ii=0, j=0, dim=M.length, e=0, t=0;
var I = [], C = [];
for(i=0; i<dim; i+=1){
// Create the row
I[I.length]=[];
C[C.length]=[];
for(j=0; j<dim; j+=1){

//if we're on the diagonal, put a 1 (for identity)
if(i==j){ I[i][j] = 1; }
else{ I[i][j] = 0; }

// Also, make the copy of the original
C[i][j] = M[i][j];
}
}

// Perform elementary row operations
for(i=0; i<dim; i+=1){
// get the element e on the diagonal
e = C[i][i];

// if we have a 0 on the diagonal (we'll need to swap with a lower row)
if(e==0){
//look through every row below the i'th row
for(ii=i+1; ii<dim; ii+=1){
//if the ii'th row has a non-0 in the i'th col
if(C[ii][i] != 0){
//it would make the diagonal have a non-0 so swap it
for(j=0; j<dim; j++){
e = C[i][j];       //temp store i'th row
C[i][j] = C[ii][j];//replace i'th row by ii'th
C[ii][j] = e;      //repace ii'th by temp
e = I[i][j];       //temp store i'th row
I[i][j] = I[ii][j];//replace i'th row by ii'th
I[ii][j] = e;      //repace ii'th by temp
}
//don't bother checking other rows since we've swapped
break;
}
}
//get the new diagonal
e = C[i][i];
//if it's still 0, not invertable (error)
if(e==0){return}
}

// Scale this row down by e (so we have a 1 on the diagonal)
for(j=0; j<dim; j++){
C[i][j] = C[i][j]/e; //apply to original matrix
I[i][j] = I[i][j]/e; //apply to identity
}

// Subtract this row (scaled appropriately for each row) from ALL of
// the other rows so that there will be 0's in this column in the
// rows above and below this one
for(ii=0; ii<dim; ii++){
// Only apply to other rows (we want a 1 on the diagonal)
if(ii==i){continue;}

// We want to change this element to 0
e = C[ii][i];

// Subtract (the row above(or below) scaled by e) from (the
// current row) but start at the i'th column and assume all the
// stuff left of diagonal is 0 (which it should be if we made this
// algorithm correctly)
for(j=0; j<dim; j++){
C[ii][j] -= e*C[i][j]; //apply to original matrix
I[ii][j] -= e*I[i][j]; //apply to identity
}
}
}

//we've done all operations, C should be the identity
//matrix I should be the inverse:
return I;
}``````

• Further we need to extract from the inverse matrix the Euler angels for rx , ry, and rz:
``````//==============================
function lcs2Euler(	X1x, X1y, X1z,
Y1x, Y1y, Y1z,
Z1x, Z1y, Z1z) {

var x=0;
var y=0;
var z=0;

var sy = Math.sqrt(X1x * X1x + Y1x * Y1x);
if (!sy < DBL_EPSILON) {
x = Math.atan2( Z1y, Z1z);
y = Math.atan2(-Z1x,  sy);
z = Math.atan2( Y1x, X1x);
}
else {

x =  Math.atan2(-Y1z, Y1y);
y =  Math.atan2(-Z1x,  sy);
z =  0;
}
//printVector	([pre,nut,rot],"lcs2Euler");
return [x,y,z];
}``````
• We already calucalted the rotation and now we need to calculate the x,y, z postion. This will be a point which is in front of use - along the gaze vector with particular distance and then x, and z corrinates of the view X and View Y vector:
``````\$scope.setWidgetProp( wdgName,  'rx', rad2deg(EulerAnglesInv[0]) )

var eye_dir_norm   = vec_normilize(\$scope.eyedir)
//projected to normal
var  eyeZaxis    	= eye_dir_norm;
var  eyeYaxis  	= vec_normilize(\$scope.eyeup);
var  eyeXaxis    	= neg_dir(vec_normilize(vec_product(eyeYaxis,eyeZaxis)))
var  newCalc       = [];
// here it will translate along the eye_dir and move in x and y axis
for (var i=0; i<3; i++)
newCalc[i]= \$scope.eyepos[i] + disScale*eye_dir_norm[i]
+ Xcomp*eyeXaxis[i] + Ycomp*eyeYaxis[i];

//correction of the translation to the view plane in distnace of disScale
\$scope.setWidgetProp( wdgName,  'x', newCalc[0] )
\$scope.setWidgetProp( wdgName,  'y', newCalc[1] )
\$scope.setWidgetProp( wdgName,  'z', newCalc[2] )``````

So that now we can display the UI widgets with some calls like this:

``````     \$scope.setWdgetPos('ShopingCard3DImage', 1.8, -0.25,   0.35)
\$scope.setWdgetPos('3DImage-1'         , 1.6,  0.28,  -0.1)
\$scope.setWdgetPos('3DImage-2'         , 1.6,  0.28,  0.05)
\$scope.setWdgetPos('3DImage-3'         , 1.6,  0.28,   0.3)
\$scope.setWdgetPos('3DButton-1'        , 1.6, -0.25,  -0.1)
\$scope.setWdgetPos('3DButton-2'        , 1.6, -0.25,   0.0)
\$scope.setWdgetPos('3DButton-3'        , 1.6, -0.25,   0.1)
\$scope.setWdgetPos('model-2'           , 1.7,     0,   0.1 , 1)
\$scope.setWdgetPos('3DLabel-1'         , 1.6, 	0.25,   0.14)``````

In this article I provided a demo project which is attached to this article as zip file. I attached also the javascript math function which I used.

Where we have the widget name, distance along the gaze vector from the device and the X and Y position /X and Y view vectors /screen

• To be able to get the device  gaze , up(y) and the x vector we can use the 'tracking' event:

``````//====================
\$scope.eyeMat=[];
\$scope.eyeInvMat=[];
//====================
\$rootScope.\$on('tracking', function( tracker,fargs  ) {

for(var i=0; i<fargs.position.length; i++)
{\$scope.eyepos[i]	=fargs.position[i];
\$scope.eyedir[i]	=fargs.gaze[i];
\$scope.eyeup [i]	=fargs.up [i];
}

\$scope.eyeMat	=get_mat4x4_camera(fargs.position,neg_dir(fargs.gaze),[0,1,0]);
\$scope.eyeInvMat	= matrix_invert(\$scope.eyeMat)

var EulerAngles		=transfMat2Euler(\$scope.eyeMat);
var EulerAnglesInv	    =transfMat2Euler(\$scope.eyeInvMat);

\$scope.\$applyAsync();
//////////////////////// finished tml3dRenderer.setupTrackingEventsCommand
})``````

The tracking callback function will save the current view vectors (gaze, up, postion) to a global variables which could be used later from the other functions.

I created a demo project which is one possible example showing how we can implement such functionality. The sample project is for the HoloLens - I tested with on preview mode and on the  HoloLens 1 device and it was working fine. (- zipped project with password "PTC" - please, unzip it and import it to Studio)

A note about the attached file:

- the file project-"HoloLens3dFlexibleUI-article_ExampleProject.zip" should be unzipped to HoloLens3dFlexibleUI-article.zip - which is a project file for Vuforia Studio - demo project and could be imported in the Vuforia Studio UI. To unzip the project-example.zip you need a password simple: 'PTC ' - in capital characters

-the file: myMathFunc.zip contains the used mathematical function definitions (zipped js file) . The javascript file is also contained by  the Studio project/ upload folder.

When we test the project:

Where we see in front in the middle a cursor where we can tab or we can say show UI to display the UI:

I attached a new version of the  myMathFunc.js (newVersion2-myMathFunc.zip)where rotation function and the verification was improved

Now the library could be verified by call in Node.js

``..\node>node  myMathFunc.js``

``````>>\node>node  myMathFunc.js
HERE SOME VERIFICATIONS: for the math functions is OK
sin(45) = 0.8509035245341184
======================================
TEST according http://statistik.wu-wien.ac.at/~leydold/MOK/HTML/node17.html
Matrix ::a
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=1.0000;
Row[1]=:: m[1][0]=0.0000; m[1][1]=1.0000;
=======================
Matrix ::b
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=0.0000;
Row[1]=:: m[1][0]=1.0000; m[1][1]=1.0000;
=======================
Matrix ::multMatTest
=======================
Row[0]=:: m[0][0]=2.0000; m[0][1]=1.0000;
Row[1]=:: m[1][0]=1.0000; m[1][1]=1.0000;
=======================
Matrix ::a
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=0.0000;
Row[1]=:: m[1][0]=1.0000; m[1][1]=1.0000;
=======================
Matrix ::b
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=1.0000;
Row[1]=:: m[1][0]=0.0000; m[1][1]=1.0000;
=======================
Matrix ::multMatTest
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=1.0000;
Row[1]=:: m[1][0]=1.0000; m[1][1]=2.0000;
=======================
matRx=getRotMatrixX(20)
Matrix ::matRx
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=0.0000; m[0][2]=0.0000; m[0][3]=0.0000;
Row[1]=:: m[1][0]=0.0000; m[1][1]=0.9397; m[1][2]=0.3420; m[1][3]=0.0000;
Row[2]=:: m[2][0]=0.0000; m[2][1]=-0.3420; m[2][2]=0.9397; m[2][3]=0.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
Vector[length=6] :: TESt @ matRx
=======================
vec[0]= 20
vec[1]= 0
vec[2]= 0
vec[3]= 0
vec[4]= 0
vec[5]= 0
=======================
matRy=getRotMatrixY(45)
Matrix ::matRy
=======================
Row[0]=:: m[0][0]=0.7071; m[0][1]=0.0000; m[0][2]=-0.7071; m[0][3]=0.0000;
Row[1]=:: m[1][0]=0.0000; m[1][1]=1.0000; m[1][2]=0.0000; m[1][3]=0.0000;
Row[2]=:: m[2][0]=0.7071; m[2][1]=0.0000; m[2][2]=0.7071; m[2][3]=0.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
Vector[length=6] :: TESt @ matRy
=======================
vec[0]= 0
vec[1]= 45
vec[2]= 0
vec[3]= 0
vec[4]= 0
vec[5]= 0
=======================
matRz=getRotMatrixZ(60)
Matrix ::matRz
=======================
Row[0]=:: m[0][0]=0.5000; m[0][1]=0.8660; m[0][2]=0.0000; m[0][3]=0.0000;
Row[1]=:: m[1][0]=-0.8660; m[1][1]=0.5000; m[1][2]=0.0000; m[1][3]=0.0000;
Row[2]=:: m[2][0]=0.0000; m[2][1]=0.0000; m[2][2]=1.0000; m[2][3]=0.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
Vector[length=6] :: TESt @ matRz
=======================
vec[0]= 0
vec[1]= 0
vec[2]= 59.99999999999999
vec[3]= 0
vec[4]= 0
vec[5]= 0
=======================
matTr=ggetTransMatrix(1,2,3)
Matrix ::matTr
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=0.0000; m[0][2]=0.0000; m[0][3]=1.0000;
Row[1]=:: m[1][0]=0.0000; m[1][1]=1.0000; m[1][2]=0.0000; m[1][3]=2.0000;
Row[2]=:: m[2][0]=0.0000; m[2][1]=0.0000; m[2][2]=1.0000; m[2][3]=3.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
Vector[length=6] :: TESt @ matTr
=======================
vec[0]= 0
vec[1]= 0
vec[2]= 0
vec[3]= 1
vec[4]= 2
vec[5]= 3
=======================
trfTestMat=transpondMat(mult(mult(transpondMat(matRz),mult(transpondMat(matRy),transpondMat(matRx))),transpondMat(matTr)))
Matrix ::calculated:: trfTestMat
=======================
Row[0]=:: m[0][0]=0.3536; m[0][1]=0.6124; m[0][2]=-0.7071; m[0][3]=1.0000;
Row[1]=:: m[1][0]=-0.6929; m[1][1]=0.6793; m[1][2]=0.2418; m[1][3]=2.0000;
Row[2]=:: m[2][0]=0.6284; m[2][1]=0.4044; m[2][2]=0.6645; m[2][3]=3.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
eulerOutput=transfMat2Euler(trfTestMat)
eulerOutputDEGREE=tconvEuler2Dec(eulerOutput)
Vector[length=6] :: TEST @ EulerVecDeg at LOAD
=======================
vec[0]= 0.3490658503988659
vec[1]= 0.7853981633974483
vec[2]= 1.0471975511965976
vec[3]= 1
vec[4]= 2
vec[5]= 3
=======================
Vector[length=6] :: TEST @ convEuler2Dec:: eulerOutputDeg at LOAD
=======================
vec[0]= 20
vec[1]= 45
vec[2]= 59.99999999999999
vec[3]= 1
vec[4]= 2
vec[5]= 3
=======================
trfTestMat1=getTrfMat(20,45,60,1,2,3)
Matrix ::calculated:: trfTestMat1
=======================
Row[0]=:: m[0][0]=0.3536; m[0][1]=0.6124; m[0][2]=-0.7071; m[0][3]=1.0000;
Row[1]=:: m[1][0]=-0.6929; m[1][1]=0.6793; m[1][2]=0.2418; m[1][3]=2.0000;
Row[2]=:: m[2][0]=0.6284; m[2][1]=0.4044; m[2][2]=0.6645; m[2][3]=3.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
eulerOutput1=transfMat2Euler(trfTestMat1)
eulerOutputDEGREE1=tconvEuler2Dec(eulerOutput1)
Vector[length=6] :: TEST @ EulerVecDeg1 at LOAD
=======================
vec[0]= 0.3490658503988659
vec[1]= 0.7853981633974483
vec[2]= 1.0471975511965976
vec[3]= 1
vec[4]= 2
vec[5]= 3
=======================
Vector[length=6] :: TEST @ convEuler2Dec:: eulerOutputDeg1 at LOAD
=======================
vec[0]= 20
vec[1]= 45
vec[2]= 59.99999999999999
vec[3]= 1
vec[4]= 2
vec[5]= 3
=======================
Time:: 02:16:56 PM
finished TESTLOAD Version ::version 1.003 08.April 2020``````

This is a great tutorial!
I tested it on the HoloLens 2. It works in the browser preview, but unfortunately not on HoloLens 2. What am I doing wrong?
I need a UI solution for the HoloLens 2.

Here is a picture of how it is displayed at the moment.

Hello  @Lucas ,

So far I see the picture (it is HoloLens 2? device right? ) and it is invertet - means that  you will see the backside  of the UI (assembly which implements the 3d buttons panel - is inverted and other things are working - on the correct plane but also moved.

Could you please, clarify. I did created it some time ago and need to check more detailed to understand what is going wrong on the HoloLens 2. My first guest is that we need to rotate the 3D objects about 180 degree. I remember that after some update there was a problem and the project was not working (also in Preview mode) - so I did recreated it and maybe there are some mistakes. I test that it worked generally on preview without further to check if everything is on the correct place also on the eyewear device

How do you want to use this – mean what kind of target (space target, model target or thingmark)

Additionally I want to point that the project was created at time where we did have this functionality on the HoloLens yet. In the meantime, we have new functionality 3D panel which is container for other 3D widgets and it will follow you display and could be pin on a particular location. Maybe it could be convenient to use it

Hello @RolandRaytchev,

I have worked on the weekend further and dealt with the 3D panel. It would work, but the 3D panel is too far away once it is bound to the field of view. How can I adjust the distance?
I would like to implement some kind of menu with it. The user should be able to open and close it with a gesture. In the menu are then different buttons and displays that the user constantly needs.

Hi @Lucas , to adjust the distance of a 3D panel from your point of view adjust the offset-z value of the panel. Try setting it to 60cm and it should be good enough.

I'd be interested into understanding what you've actually replaced with the 3D panel.

@RolandRaytchev as always great tutorial. I'll save some time to test this out on the HoloLens2!

I confirm that in preview mode it works as intended.

Anyway I have a question similar to Lucas', which is how to call gestures by clicking something. I tried to understand it from your example project but I'm not that good. Also I understand it's counter productive to say "hey you have gestures but I make you click something" it's just that in some cases it won't track the gesture or, or example, when I define a new voice command but the environment is so loud that the hololens won't hear the command.

Hello Roland, many thanks for your great ideas and shared tests, they are very useful.

I have tested on my Hololens, and it works!

Anyway, I would like to ask you if you can give me clarifications on this function:

``````...
\$scope.setWdgetPos('3DImage-1', 1.6,  0.28,  -0.1)
...``````

Can you please tell me how can I determine the correct numeric values when I use my 3D elements? I have seen the function definition but I can't understand how to use these parameters.

In particular, I would like to replicate the tethered/untethered functionality and so understand why you chose "1.6" "0.28" "-0.1" as values based on that 3DImage-1.

Hi @aletenti ,

this techique in the article is working still but it  was originaly  intented as workaround before there was a 3DPanel widget. So the quesiton was to allign the elements norm to eyevector with distance to the device position.

OK the quesiton is what are your requirements. e.g. There is alternative to use the 3DPanel which is more easy to work for the same task. OK there is 3D assembly used as 3D buttons assembly , the concept here is  also from the time before 3D buttons are availible in the HoloLens , We have all this functionality in the mean time so that I think this techniques is not neccessarly and relevant for the current usage. Remeber that here we use only tap click. Means you gaze the element and then do Air tap. on the current 3D button we have really a finger click on them. OK Air tab is further supported and work. The finger click/tab is possibly only ith the 3D button widgets

But OK let consider how this function work \$scope.setWdgetPos()

• it receive arguments widget name , the disScale Xcomp, Ycomp - coordiante

In the example

``\$scope.setWdgetPos('3DImage-1', 1.6,  0.28,  -0.1)``

•  to understand the difference I will use also the example
``\$scope.setWdgetPos('3DImage-1', 1.6,  0.28,  0.0)​``

distScale is argument for the distance from your possition (device ) along the Eyevector, so larger values will increasd the distance to the plane where the UI will be located.

Xcomp

Ycomp

are x,and y coordinate of the component.

You have to see that the center of your device screen has a normal vector which is the eyevector and this is the x=0,y =0 zerro point if the view plane. The view plane is  flat plane paralell to you device - if iPad then paralell to the iPad screen. And here you need the flat  coordinates in x,y of the component in meter so far I remember , So here is the postion of the 3DImage-1 shift to the right =0.28 and shifted down (minus) -0.1.

So we can see that when we call the showUI it will move all elements paralel to the view plane (through the eyes - if  the device is a tablet then this is  the plane of the screen) . The plane where the element should be displayed is shift paralell to the screen plane in distance of 1.6 m. the center is 0, 0. And 2.8, 0.0 is the element moved to the center , right

Crystal clear. AMAZING work!

Thank you so much, @RolandRaytchev !

Version history
Last update:
‎Apr 08, 2020 08:53 AM
Updated by:
Labels (1)

Contributors