Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- Community
- Augmented Reality
- Vuforia Studio and Chalk Tech Tips
- How can we make a 3D Widget on HoloLens visible i...

Options

- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Notify Moderator

100% helpful
(1/1)

The goal here is to get a some UI containing 3D Widgets (3dButtons , 3dImages , 3d Labels and 3d Models) in front of current HoloLens view. For example after saying "show UI" the UI elements should appear in front view e.g. distance of 1,5 meters. Example:

In the Tech Tip article "How to create a custom button pressable on HoloLens device?" is shown how to create a custom UI button. Now based on the example of this project the functionality will be extended by adding the possibility for dynamic display of the UI elements in front of the current view.

To be able to move the elements in front of us we need to calculated the current view matrix and the inverse matrix of if. For this goal we need to prepare some mathemtics function like , matrix inverse calculation, marix multiplication , matrix with vector multiplicaiton, vector product und dot vector product ... etc. When we sear in Internet there a lot of sources where we can find the informormation how to implement such mathematical tools.

- The first steps is to calculate the view matrix - e.g. some code:

```
...
function get_mat4x4_camera(eyepos,eyedir,eyeup)
{
// printVector(eyepos, "eyepos")
// printVector(eyedir, "eyedir")
// printVector(eyeup, "eyeup")
var mat4x4 =[[1.0,0.0,0.0,0.0],[0.0,1.0,0.0,0.0],[0.0,0.0,1.0,0.0],[0.0,0.0,0.0,1.0]];
var i=0;
var eyeZaxis = this.vec_normilize(eyedir);
var eyeYaxis = this.vec_normilize(eyeup);
var eyeXaxis = this.vec_normilize(this.vec_product(eyeYaxis,eyeZaxis))
for (i=0;i<3;i++) mat4x4[i][0] = eyeXaxis[i];
for (i=0;i<3;i++) mat4x4[i][1] = eyeYaxis[i];
for (i=0;i<3;i++) mat4x4[i][2] = eyeZaxis[i];
for (i=0;i<3;i++) mat4x4[i][3] = eyepos[i];
//for (i=0;i<3;i++) mat4x4[i][3] = 0.0-eyepos[i];
return mat4x4;
//return this.transpondMat(mat4x4);
}
////////////////////// vector with length =1.0
function vec_normilize(ary1) {
ret=[];
var len=this.vec_len(ary1);
for (var i = 0; i < ary1.length; i++)
ret[i]=ary1[i]/len;
return ret;
};
//vector product range =3
function vec_product(a, b) {
var vecprod =[0.0,0.0,0.0];
vecprod[0]=a[1]*b[2]- a[2]*b[1];
vecprod[1]=a[2]*b[0]- a[0]*b[2];
vecprod[2]=a[0]*b[1]- a[1]*b[0];
return vecprod;
};
```

When we find programming code for mathematical operation we need to verify it by some examples where we know the results, because often there are some math sources which does not implement the correct solution.

I think a good resource are e.g.:

https://www.learnopencv.com/rotation-matrix-to-euler-angles/

https://en.wikipedia.org/wiki/Conversion_between_quaternions_and_Euler_angles

https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/Matrix_math_for_the_web

So that we do not need to re-invent the wheel, but we need to be careful to find the correct mathematical relations and to test and verify if they really work.

- Next step is to calculate the invert matrix of the view matrix. A good implementation I found is this one:

```
function matrix_invert(M){
// I use Guassian Elimination to calculate the inverse:
// (1) 'augment' the matrix (left) by the identity (on the right)
// (2) Turn the matrix on the left into the identity by elemetry row ops
// (3) The matrix on the right is the inverse (was the identity matrix)
// There are 3 elemtary row ops: (I combine b and c in my code)
// (a) Swap 2 rows
// (b) Multiply a row by a scalar
// (c) Add 2 rows
//if the matrix isn't square: exit (error)
if(M.length !== M[0].length){return;}
//create the identity matrix (I), and a copy (C) of the original
var i=0, ii=0, j=0, dim=M.length, e=0, t=0;
var I = [], C = [];
for(i=0; i<dim; i+=1){
// Create the row
I[I.length]=[];
C[C.length]=[];
for(j=0; j<dim; j+=1){
//if we're on the diagonal, put a 1 (for identity)
if(i==j){ I[i][j] = 1; }
else{ I[i][j] = 0; }
// Also, make the copy of the original
C[i][j] = M[i][j];
}
}
// Perform elementary row operations
for(i=0; i<dim; i+=1){
// get the element e on the diagonal
e = C[i][i];
// if we have a 0 on the diagonal (we'll need to swap with a lower row)
if(e==0){
//look through every row below the i'th row
for(ii=i+1; ii<dim; ii+=1){
//if the ii'th row has a non-0 in the i'th col
if(C[ii][i] != 0){
//it would make the diagonal have a non-0 so swap it
for(j=0; j<dim; j++){
e = C[i][j]; //temp store i'th row
C[i][j] = C[ii][j];//replace i'th row by ii'th
C[ii][j] = e; //repace ii'th by temp
e = I[i][j]; //temp store i'th row
I[i][j] = I[ii][j];//replace i'th row by ii'th
I[ii][j] = e; //repace ii'th by temp
}
//don't bother checking other rows since we've swapped
break;
}
}
//get the new diagonal
e = C[i][i];
//if it's still 0, not invertable (error)
if(e==0){return}
}
// Scale this row down by e (so we have a 1 on the diagonal)
for(j=0; j<dim; j++){
C[i][j] = C[i][j]/e; //apply to original matrix
I[i][j] = I[i][j]/e; //apply to identity
}
// Subtract this row (scaled appropriately for each row) from ALL of
// the other rows so that there will be 0's in this column in the
// rows above and below this one
for(ii=0; ii<dim; ii++){
// Only apply to other rows (we want a 1 on the diagonal)
if(ii==i){continue;}
// We want to change this element to 0
e = C[ii][i];
// Subtract (the row above(or below) scaled by e) from (the
// current row) but start at the i'th column and assume all the
// stuff left of diagonal is 0 (which it should be if we made this
// algorithm correctly)
for(j=0; j<dim; j++){
C[ii][j] -= e*C[i][j]; //apply to original matrix
I[ii][j] -= e*I[i][j]; //apply to identity
}
}
}
//we've done all operations, C should be the identity
//matrix I should be the inverse:
return I;
}
```

- Further we need to extract from the inverse matrix the Euler angels for rx , ry, and rz:

```
//==============================
function lcs2Euler( X1x, X1y, X1z,
Y1x, Y1y, Y1z,
Z1x, Z1y, Z1z) {
var x=0;
var y=0;
var z=0;
var sy = Math.sqrt(X1x * X1x + Y1x * Y1x);
if (!sy < DBL_EPSILON) {
x = Math.atan2( Z1y, Z1z);
y = Math.atan2(-Z1x, sy);
z = Math.atan2( Y1x, X1x);
}
else {
x = Math.atan2(-Y1z, Y1y);
y = Math.atan2(-Z1x, sy);
z = 0;
}
//printVector ([pre,nut,rot],"lcs2Euler");
return [x,y,z];
}
```

- We already calucalted the rotation and now we need to calculate the x,y, z postion. This will be a point which is in front of use - along the gaze vector with particular distance and then x, and z corrinates of the view X and View Y vector:

```
$scope.setWidgetProp( wdgName, 'rx', rad2deg(EulerAnglesInv[0]) )
$scope.setWidgetProp( wdgName, 'ry', rad2deg(EulerAnglesInv[1]) )
$scope.setWidgetProp( wdgName, 'rz', rad2deg(EulerAnglesInv[2]) )
var eye_dir_norm = vec_normilize($scope.eyedir)
//projected to normal
var eyeZaxis = eye_dir_norm;
var eyeYaxis = vec_normilize($scope.eyeup);
var eyeXaxis = neg_dir(vec_normilize(vec_product(eyeYaxis,eyeZaxis)))
var newCalc = [];
// here it will translate along the eye_dir and move in x and y axis
for (var i=0; i<3; i++)
newCalc[i]= $scope.eyepos[i] + disScale*eye_dir_norm[i]
+ Xcomp*eyeXaxis[i] + Ycomp*eyeYaxis[i];
//correction of the translation to the view plane in distnace of disScale
$scope.setWidgetProp( wdgName, 'x', newCalc[0] )
$scope.setWidgetProp( wdgName, 'y', newCalc[1] )
$scope.setWidgetProp( wdgName, 'z', newCalc[2] )
```

So that now we can display the UI widgets with some calls like this:

```
$scope.setWdgetPos('ShopingCard3DImage', 1.8, -0.25, 0.35)
$scope.setWdgetPos('3DImage-1' , 1.6, 0.28, -0.1)
$scope.setWdgetPos('3DImage-2' , 1.6, 0.28, 0.05)
$scope.setWdgetPos('3DImage-3' , 1.6, 0.28, 0.3)
$scope.setWdgetPos('3DButton-1' , 1.6, -0.25, -0.1)
$scope.setWdgetPos('3DButton-2' , 1.6, -0.25, 0.0)
$scope.setWdgetPos('3DButton-3' , 1.6, -0.25, 0.1)
$scope.setWdgetPos('model-2' , 1.7, 0, 0.1 , 1)
$scope.setWdgetPos('3DLabel-1' , 1.6, 0.25, 0.14)
```

In this article I provided a demo project which is attached to this article as zip file. I attached also the javascript math function which I used.

Where we have the widget name, distance along the gaze vector from the device and the X and Y position /X and Y view vectors /screen

- To be able to get the device gaze , up(y) and the x vector we can use the 'tracking' event:

```
//====================
$scope.eyeMat=[];
$scope.eyeInvMat=[];
//====================
$rootScope.$on('tracking', function( tracker,fargs ) {
for(var i=0; i<fargs.position.length; i++)
{$scope.eyepos[i] =fargs.position[i];
$scope.eyedir[i] =fargs.gaze[i];
$scope.eyeup [i] =fargs.up [i];
}
$scope.eyeMat =get_mat4x4_camera(fargs.position,neg_dir(fargs.gaze),[0,1,0]);
$scope.eyeInvMat = matrix_invert($scope.eyeMat)
var EulerAngles =transfMat2Euler($scope.eyeMat);
var EulerAnglesInv =transfMat2Euler($scope.eyeInvMat);
$scope.$applyAsync();
//////////////////////// finished tml3dRenderer.setupTrackingEventsCommand
})
```

The tracking callback function will save the current view vectors (gaze, up, postion) to a global variables which could be used later from the other functions.

I created a demo project which is one possible example showing how we can implement such functionality. The sample project is for the HoloLens - I tested with on preview mode and on the HoloLens 1 device and it was working fine. (- zipped project with password "PTC" - please, unzip it and import it to Studio)

A note about the attached file:

- the file project-"HoloLens3dFlexibleUI-article_ExampleProject.zip" should be unzipped to HoloLens3dFlexibleUI-article.zip - which is a project file for Vuforia Studio - demo project and could be imported in the Vuforia Studio UI. To unzip the project-example.zip you need a password simple: '**PTC** ' - in capital characters

-the file: myMathFunc.zip contains the used mathematical function definitions (zipped js file) . The javascript file is also contained by the Studio project/ upload folder.

When we test the project:

Where we see in front in the middle a cursor where we can tab or we can say show UI to display the UI:

Comments

Apr 08, 2020
06:34 AM

- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Notify Moderator

Apr 08, 2020
06:34 AM

I attached a new version of the myMathFunc.js (newVersion2-myMathFunc.zip)where rotation function and the verification was improved

Now the library could be verified by call in Node.js

`..\node>node myMathFunc.js`

```
>>\node>node myMathFunc.js
TESTLOAD for myMathFunc.js OK!
HERE SOME VERIFICATIONS: for the math functions is OK
sin(45) = 0.8509035245341184
======================================
TEST according http://statistik.wu-wien.ac.at/~leydold/MOK/HTML/node17.html
Matrix ::a
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=1.0000;
Row[1]=:: m[1][0]=0.0000; m[1][1]=1.0000;
=======================
Matrix ::b
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=0.0000;
Row[1]=:: m[1][0]=1.0000; m[1][1]=1.0000;
=======================
Matrix ::multMatTest
=======================
Row[0]=:: m[0][0]=2.0000; m[0][1]=1.0000;
Row[1]=:: m[1][0]=1.0000; m[1][1]=1.0000;
=======================
Matrix ::a
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=0.0000;
Row[1]=:: m[1][0]=1.0000; m[1][1]=1.0000;
=======================
Matrix ::b
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=1.0000;
Row[1]=:: m[1][0]=0.0000; m[1][1]=1.0000;
=======================
Matrix ::multMatTest
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=1.0000;
Row[1]=:: m[1][0]=1.0000; m[1][1]=2.0000;
=======================
matRx=getRotMatrixX(20)
Matrix ::matRx
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=0.0000; m[0][2]=0.0000; m[0][3]=0.0000;
Row[1]=:: m[1][0]=0.0000; m[1][1]=0.9397; m[1][2]=0.3420; m[1][3]=0.0000;
Row[2]=:: m[2][0]=0.0000; m[2][1]=-0.3420; m[2][2]=0.9397; m[2][3]=0.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
Vector[length=6] :: TESt @ matRx
=======================
vec[0]= 20
vec[1]= 0
vec[2]= 0
vec[3]= 0
vec[4]= 0
vec[5]= 0
=======================
matRy=getRotMatrixY(45)
Matrix ::matRy
=======================
Row[0]=:: m[0][0]=0.7071; m[0][1]=0.0000; m[0][2]=-0.7071; m[0][3]=0.0000;
Row[1]=:: m[1][0]=0.0000; m[1][1]=1.0000; m[1][2]=0.0000; m[1][3]=0.0000;
Row[2]=:: m[2][0]=0.7071; m[2][1]=0.0000; m[2][2]=0.7071; m[2][3]=0.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
Vector[length=6] :: TESt @ matRy
=======================
vec[0]= 0
vec[1]= 45
vec[2]= 0
vec[3]= 0
vec[4]= 0
vec[5]= 0
=======================
matRz=getRotMatrixZ(60)
Matrix ::matRz
=======================
Row[0]=:: m[0][0]=0.5000; m[0][1]=0.8660; m[0][2]=0.0000; m[0][3]=0.0000;
Row[1]=:: m[1][0]=-0.8660; m[1][1]=0.5000; m[1][2]=0.0000; m[1][3]=0.0000;
Row[2]=:: m[2][0]=0.0000; m[2][1]=0.0000; m[2][2]=1.0000; m[2][3]=0.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
Vector[length=6] :: TESt @ matRz
=======================
vec[0]= 0
vec[1]= 0
vec[2]= 59.99999999999999
vec[3]= 0
vec[4]= 0
vec[5]= 0
=======================
matTr=ggetTransMatrix(1,2,3)
Matrix ::matTr
=======================
Row[0]=:: m[0][0]=1.0000; m[0][1]=0.0000; m[0][2]=0.0000; m[0][3]=1.0000;
Row[1]=:: m[1][0]=0.0000; m[1][1]=1.0000; m[1][2]=0.0000; m[1][3]=2.0000;
Row[2]=:: m[2][0]=0.0000; m[2][1]=0.0000; m[2][2]=1.0000; m[2][3]=3.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
Vector[length=6] :: TESt @ matTr
=======================
vec[0]= 0
vec[1]= 0
vec[2]= 0
vec[3]= 1
vec[4]= 2
vec[5]= 3
=======================
trfTestMat=transpondMat(mult(mult(transpondMat(matRz),mult(transpondMat(matRy),transpondMat(matRx))),transpondMat(matTr)))
Matrix ::calculated:: trfTestMat
=======================
Row[0]=:: m[0][0]=0.3536; m[0][1]=0.6124; m[0][2]=-0.7071; m[0][3]=1.0000;
Row[1]=:: m[1][0]=-0.6929; m[1][1]=0.6793; m[1][2]=0.2418; m[1][3]=2.0000;
Row[2]=:: m[2][0]=0.6284; m[2][1]=0.4044; m[2][2]=0.6645; m[2][3]=3.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
eulerOutput=transfMat2Euler(trfTestMat)
eulerOutputDEGREE=tconvEuler2Dec(eulerOutput)
Vector[length=6] :: TEST @ EulerVecDeg at LOAD
=======================
vec[0]= 0.3490658503988659
vec[1]= 0.7853981633974483
vec[2]= 1.0471975511965976
vec[3]= 1
vec[4]= 2
vec[5]= 3
=======================
Vector[length=6] :: TEST @ convEuler2Dec:: eulerOutputDeg at LOAD
=======================
vec[0]= 20
vec[1]= 45
vec[2]= 59.99999999999999
vec[3]= 1
vec[4]= 2
vec[5]= 3
=======================
trfTestMat1=getTrfMat(20,45,60,1,2,3)
Matrix ::calculated:: trfTestMat1
=======================
Row[0]=:: m[0][0]=0.3536; m[0][1]=0.6124; m[0][2]=-0.7071; m[0][3]=1.0000;
Row[1]=:: m[1][0]=-0.6929; m[1][1]=0.6793; m[1][2]=0.2418; m[1][3]=2.0000;
Row[2]=:: m[2][0]=0.6284; m[2][1]=0.4044; m[2][2]=0.6645; m[2][3]=3.0000;
Row[3]=:: m[3][0]=0.0000; m[3][1]=0.0000; m[3][2]=0.0000; m[3][3]=1.0000;
=======================
eulerOutput1=transfMat2Euler(trfTestMat1)
eulerOutputDEGREE1=tconvEuler2Dec(eulerOutput1)
Vector[length=6] :: TEST @ EulerVecDeg1 at LOAD
=======================
vec[0]= 0.3490658503988659
vec[1]= 0.7853981633974483
vec[2]= 1.0471975511965976
vec[3]= 1
vec[4]= 2
vec[5]= 3
=======================
Vector[length=6] :: TEST @ convEuler2Dec:: eulerOutputDeg1 at LOAD
=======================
vec[0]= 20
vec[1]= 45
vec[2]= 59.99999999999999
vec[3]= 1
vec[4]= 2
vec[5]= 3
=======================
Library test on LOAD finished!
Time:: 02:16:56 PM
finished TESTLOAD Version ::version 1.003 08.April 2020
```

Apr 15, 2021
04:14 PM

- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Notify Moderator

Apr 15, 2021
04:14 PM

I tested it on the HoloLens 2. It works in the browser preview, but unfortunately not on HoloLens 2. What am I doing wrong?

I need a UI solution for the HoloLens 2.

Apr 15, 2021
04:19 PM

- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Notify Moderator

Apr 15, 2021
04:19 PM

Here is a picture of how it is displayed at the moment.

Apr 16, 2021
04:37 AM

- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Notify Moderator

Apr 16, 2021
04:37 AM

Hello @Lucas ,

thanks for your feedback!

So far I see the picture (it is HoloLens 2? device right? ) and it is invertet - means that you will see the backside of the UI (assembly which implements the 3d buttons panel - is inverted and other things are working - on the correct plane but also moved.

Could you please, clarify. I did created it some time ago and need to check more detailed to understand what is going wrong on the HoloLens 2. My first guest is that we need to rotate the 3D objects about 180 degree. I remember that after some update there was a problem and the project was not working (also in Preview mode) - so I did recreated it and maybe there are some mistakes. I test that it worked generally on preview without further to check if everything is on the correct place also on the eyewear device

How do you want to use this – mean what kind of target (space target, model target or thingmark)

Additionally I want to point that the project was created at time where we did have this functionality on the HoloLens yet. In the meantime, we have new functionality 3D panel which is container for other 3D widgets and it will follow you display and could be pin on a particular location. Maybe it could be convenient to use it

Apr 19, 2021
01:26 AM

- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Notify Moderator

Apr 19, 2021
01:26 AM

Hello @RolandRaytchev,

thank you for your quick reply.

I have worked on the weekend further and dealt with the 3D panel. It would work, but the 3D panel is too far away once it is bound to the field of view. How can I adjust the distance?

I have worked on the weekend further and dealt with the 3D panel. It would work, but the 3D panel is too far away once it is bound to the field of view. How can I adjust the distance?

I would like to implement some kind of menu with it. The user should be able to open and close it with a gesture. In the menu are then different buttons and displays that the user constantly needs.

Nov 01, 2022
10:47 AM

- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Notify Moderator

Nov 01, 2022
10:47 AM

Hi @Lucas , to adjust the distance of a 3D panel from your point of view adjust the offset-z value of the panel. Try setting it to 60cm and it should be good enough.

I'd be interested into understanding what you've actually replaced with the 3D panel.

@RolandRaytchev as always great tutorial. I'll save some time to test this out on the HoloLens2!

I confirm that in preview mode it works as intended.

Anyway I have a question similar to Lucas', which is how to call gestures by clicking something. I tried to understand it from your example project but I'm not that good. Also I understand it's counter productive to say "hey you have gestures but I make you click something" it's just that in some cases it won't track the gesture or, or example, when I define a new voice command but the environment is so loud that the hololens won't hear the command.