Tag-Along 3D Labels in HoloLens experiences

[IstvanPolacsek]

Hello,

I am currently working on a HoloLens experience which will be used in large physical space and the users need access to some of the information no matter whether they are near the loaded experience or not.

The HoloLens Start menu and the Vuforia Studio Experience selector menu uses the concept, which Microsoft calls tag-along.

Is it possible to include/edit this property for a studio 3D Label?

Thanks in advance!

[RolandRaytchev · Accepted solution]

Hello @IstvanPolacsek ,

found that your question was staying for long time period  without answer. 

So, for me at first time was not quite clear what was your question , because I  did not know the term "Microsoft calls tag-along " but I checked the Microsoft  side like:

https://docs.microsoft.com/en-us/windows/mixed-reality/billboarding-and-tag-along

where this is more detailed explained.

So that I think , you want to have  some kind of dynamically movement of elements to stay in same position  in relation to the gaze vector and eye postion.

The answer is ... it could be possible 

Let explain.

-currently in the HoloLens our AR world is the Vuforia View application. This means you need to use one of the possible tracking methods to have a position for all object in the space. So when you scan for example a thingmrk - in this case this is our world coordinate system and all 3d object are located in regards to model coordinate system. (actually the model coordinate system is an offset to the thin mark because it is seldom to the point 0.0,0) IF you move the location of the thingmark and scan it again , then the position all 3d objects  will update  regarding the new location of the thingmark.

But we have a functionality what may be helpful here. We can track the eye /camera position by having position vector, eye_direction vector and eye_up direction. In this case theoretical you can calculate  every time when you eye /HoloLens device is moved or rotated  - so that you will have the effect like some kind of billboard to your HoloLens.

The following code :

/////////////////////////////////////
$rootScope.$on('modelLoaded', function() { 
 $scope.setEYEtrack()
//   ...
})
//////
////////////////////////////////////////////////////////////
$scope.setEYEtrack= function() {

tml3dRenderer.setupTrackingEventsCommand (function(target,eyepos,eyedir,eyeup) {
 
  
   $scope.app.params['eyepos'] ="eyepos=("+eyepos[0].toFixed(2)+","+eyepos[1].toFixed(2)+","+eyepos[2].toFixed(2)+")";
   $scope.app.params['eyedir'] ="eyedir=("+eyedir[0].toFixed(2)+","+eyedir[1].toFixed(2)+","+eyedir[2].toFixed(2)+")";
   $scope.app.params['eyeup' ] ="eyeup =("+ eyeup[0].toFixed(2)+","+ eyeup[1].toFixed(2)+","+ eyeup[2].toFixed(2)+")";
  
  var cam_mat = $scope.Get_mat4x4_camera(eyepos,eyedir,eyeup) 
....  
  })   
//....

will stetup a tracking event and will provide to every change in the eye position the vectors for eyedir,eyeup and eyepos (eye position in respect to the model coordinates)

Regarding to the calculation of the correct values of the 3d element coordinates -  it could be a little bit complicated but is still possible . Also, we need to have such calculation for each object we want to move.

The most trivial way is simple to set  the billboard property of the element and to move only the position 

var scale=5.0;
 $scope.setWidgetProp('3DImage-1',  'x', ( eyepos[0]+eyedir[0]*scale));
 $scope.setWidgetProp('3DImage-1',  'y', ( eyepos[1]+eyedir[1]*scale));
 $scope.setWidgetProp('3DImage-1',  'z', ( eyepos[2]+eyedir[2]*scale));

So it does not matter where we move the model ( move because I tested it here in preview) the  3dimage goes always to the screen center and with a distance offset to the screen plane  in eye direction

Of course we could do a more precise positioning but need some additional mathematical consideration

So I tested it also on android device but the movement seem to be not updated on my first attempt fluently so may be  will be better to update it  on request - so means in some situation I need to update the position of my elements by command or by object click etc.  

I

think my project have a lot of different thing I tested there,  so that the result are ,may be,  not so relevant according the performance and this should be further tested if better results could be achieved.

[IstvanPolacsek]

Hello @RolandRaytchev ,

Many thanks for the solution.

I was able to implement it using the following code:

$scope.setTagalong= function() {
	tml3dRenderer.setupTrackingEventsCommand (function(target,eyepos,eyedir,eyeup) {
		$scope.app.params['eyepos'] = [eyepos[0].toFixed(2),eyepos[1].toFixed(2),eyepos[2].toFixed(2)];
		$scope.app.params['eyedir'] = [eyedir[0].toFixed(2),eyedir[1].toFixed(2),eyedir[2].toFixed(2)];
		$scope.app.params['eyeup' ] = [eyeup[0].toFixed(2), eyeup[1].toFixed(2), eyeup[2].toFixed(2)];
	})
};

$scope.showTagalongLabel = function() {
	$scope.setWidgetProp('tagalongLabel','visible',true);
  	$scope.setWidgetProp('tagalongLabel','billboard',true);
	$scope.setTagalong();
	$scope.setWidgetProp('tagalongLabel','x',(Number($scope.app.params['eyepos'][0])+Number($scope.app.params['eyedir'][0])));
	$scope.setWidgetProp('tagalongLabel','y',(Number($scope.app.params['eyepos'][1])+Number($scope.app.params['eyedir'][1])));
	$scope.setWidgetProp('tagalongLabel','z',(Number($scope.app.params['eyepos'][2])+Number($scope.app.params['eyedir'][2])));
};

I made a binding to a 2D button and now, each time I press the button, the position of the 3D label updates.

That's great!

What I am missing here is an event, which I can bind it to have the values updated constantly.

Any ideas?

Thanks,

Istvan

[IstvanPolacsek]

I was able to make the label follow the camera using setInterval method.

I made a test on an iPhone, will try to achieve the same thing using HoloLens.

@RolandRaytchev , Thank you very much again for the help!

[IstvanPolacsek]

Unfortunately I had no success reproducing it in HoloLens.

Seems like the tml3dRenderer function is not working on HoloLens.

Or maybe I am missing something, any ideas?

Thanks!

[RolandRaytchev]

Hi @IstvanPolacsek ,

I tested it today on the HoloLens but it was not working with the current release. I am not 100% sure , but I believe it was working earler. I  tested with 8.4.2 both preivew on studio and the vuforia view on the Hololens

In preview of HoloLense project work but not on the enddevice - HoloLens 1

The following code tested :

$rootScope.$on('modelLoaded', function() { 

$scope.setEYEtrack()
console.warn(tml3dRenderer);

});



////////////////////////////////////////////////////////////
$scope.setEYEtrack= function() {
// OUTSIDE a Function when loading
// but better to call in modelload

//// define tracingEvent only on end device
tml3dRenderer.setupTrackingEventsCommand (function(target,eyepos,eyedir,eyeup) {


//3DImage-1
var scale=5.0;
$scope.setWidgetProp('3DImage-1', 'x', ( eyepos[0]+eyedir[0]*scale));
console.log("3DImage-1 -> x ="+ ( eyepos[0]+eyedir[0]*scale));
$scope.$applyAsync();
$scope.setWidgetProp('3DImage-1', 'y', ( eyepos[1]+eyedir[1]*scale));
console.log("3DImage-1 -> y ="+ ( eyepos[1]+eyedir[1]*scale));
$scope.$applyAsync();
$scope.setWidgetProp('3DImage-1', 'z', ( eyepos[2]+eyedir[2]*scale));
console.log("3DImage-1 -> z ="+ ( eyepos[2]+eyedir[2]*scale));
$scope.$applyAsync();
$scope.app.params['eyepos'] ="eyepos=("+eyepos[0].toFixed(2)+","+eyepos[1].toFixed(2)+","+eyepos[2].toFixed(2)+")";
$scope.app.params['eyedir'] ="eyedir=("+eyedir[0].toFixed(2)+","+eyedir[1].toFixed(2)+","+eyedir[2].toFixed(2)+")";
$scope.app.params['eyeup' ] ="eyeup =("+ eyeup[0].toFixed(2)+","+ eyeup[1].toFixed(2)+","+ eyeup[2].toFixed(2)+")";
$scope.$applyAsync();
}) 
////////////////////////////////////////////////////
}
////// finish setEYEtrack

But this did not work on the HoloLens - so I also observed that this event seem not to fire

[RolandRaytchev]

Hi @IstvanPolacsek,

so now I got it after  a hint from  PTC R&D and it works now on the HoloLens.

So you need to add/set in Vuforia Studio Installation   the property "showTrackingEvents": true in the builder-settings.json.

Then when your restart studio and open the project and then you can see the property "Enable Tracking Events"  for 3d Container Widget. You need to activate it.

Afterword’s you can track the event by calling  :

$scope.$root.$on('tracking', function() { ...} 

I

If you need the vectors for the camera you need to call the tml3dRenderer.setupTrackingEventsCommand  - so then you do not need the definition of the $scope.$root.$on('tracking',... event

Example:

scope.setMyEYEtrack= function() {
 if(tml3dRenderer)  
{ try {
tml3dRenderer.setupTrackingEventsCommand (function(target,eyepos,eyedir,eyeup) {
  var scale=5.0;
 $scope.setWidgetProp('3DImage-1',  'x', ( eyepos[0]+eyedir[0]*scale));
 $scope.setWidgetProp('3DImage-1',  'y', ( eyepos[1]+eyedir[1]*scale));
 $scope.setWidgetProp('3DImage-1',  'z', ( eyepos[2]+eyedir[2]*scale));
 $scope.$applyAsync();

  },undefined) } catch (e) { $scope.setWidgetProp('3DLabel-2',  'text', "exception=");}
      
}else 
  $scope.setWidgetProp('3DLabel-2',  'text', "null tml3dRenderer object on HoloLens");

} ////// finish setEYEtrack

This will move a 3d Image widget (billboard = true) always in the middle of your view field with distance = scale - is here 5 m