Tag-Along 3D Labels in HoloLens experiences

Hello @IstvanPolacsek ,

found that your question was staying for long time period  without answer. 

So, for me at first time was not quite clear what was your question , because I  did not know the term "Microsoft calls tag-along " but I checked the Microsoft  side like:

https://docs.microsoft.com/en-us/windows/mixed-reality/billboarding-and-tag-along

where this is more detailed explained.

So that I think , you want to have  some kind of dynamically movement of elements to stay in same position  in relation to the gaze vector and eye postion.

The answer is ... it could be possible 

Let explain.

-currently in the HoloLens our AR world is the Vuforia View application. This means you need to use one of the possible tracking methods to have a position for all object in the space. So when you scan for example a thingmrk - in this case this is our world coordinate system and all 3d object are located in regards to model coordinate system. (actually the model coordinate system is an offset to the thin mark because it is seldom to the point 0.0,0) IF you move the location of the thingmark and scan it again , then the position all 3d objects  will update  regarding the new location of the thingmark.

But we have a functionality what may be helpful here. We can track the eye /camera position by having position vector, eye_direction vector and eye_up direction. In this case theoretical you can calculate  every time when you eye /HoloLens device is moved or rotated  - so that you will have the effect like some kind of billboard to your HoloLens.

The following code :

/////////////////////////////////////
$rootScope.$on('modelLoaded', function() { 
 $scope.setEYEtrack()
// ...
})
//////
////////////////////////////////////////////////////////////
$scope.setEYEtrack= function() {

tml3dRenderer.setupTrackingEventsCommand (function(target,eyepos,eyedir,eyeup) {
 
 
 $scope.app.params['eyepos'] ="eyepos=("+eyepos[0].toFixed(2)+","+eyepos[1].toFixed(2)+","+eyepos[2].toFixed(2)+")";
 $scope.app.params['eyedir'] ="eyedir=("+eyedir[0].toFixed(2)+","+eyedir[1].toFixed(2)+","+eyedir[2].toFixed(2)+")";
 $scope.app.params['eyeup' ] ="eyeup =("+ eyeup[0].toFixed(2)+","+ eyeup[1].toFixed(2)+","+ eyeup[2].toFixed(2)+")";
 
 var cam_mat = $scope.Get_mat4x4_camera(eyepos,eyedir,eyeup) 
.... 
 }) 
//....

will stetup a tracking event and will provide to every change in the eye position the vectors for eyedir,eyeup and eyepos (eye position in respect to the model coordinates)

Regarding to the calculation of the correct values of the 3d element coordinates -  it could be a little bit complicated but is still possible . Also, we need to have such calculation for each object we want to move.

The most trivial way is simple to set  the billboard property of the element and to move only the position 

var scale=5.0;
 $scope.setWidgetProp('3DImage-1', 'x', ( eyepos[0]+eyedir[0]*scale));
 $scope.setWidgetProp('3DImage-1', 'y', ( eyepos[1]+eyedir[1]*scale));
 $scope.setWidgetProp('3DImage-1', 'z', ( eyepos[2]+eyedir[2]*scale));

So it does not matter where we move the model ( move because I tested it here in preview) the  3dimage goes always to the screen center and with a distance offset to the screen plane  in eye direction

Of course we could do a more precise positioning but need some additional mathematical consideration

So I tested it also on android device but the movement seem to be not updated on my first attempt fluently so may be  will be better to update it  on request - so means in some situation I need to update the position of my elements by command or by object click etc.  

I

think my project have a lot of different thing I tested there,  so that the result are ,may be,  not so relevant according the performance and this should be further tested if better results could be achieved.