How to Click on a Model Located at a Distant Position in HoloLens Experience

[usami]

In the Hololens project, I would like to perform a click on a model that is located at a distant position.

Is it possible to designate distant models using gaze manipulation or other gestures?

I would like to achieve the ability to display multiple models in AR and, when clicking on a specific model among them, display the attribute information of the selected model.

[RolandRaytchev]

Hi @usami ,

I am not sure if I understand correctly your requiriert , but if you need to click on model and then to receive further Information about this coponent e.g. form metadata or other source (Json file containing information ) then this is possible. You can e.g.  depending on the component selection display further documents , images or models .

Here is an project which demonstrate the principle for mobile platform. But it could be done the same for HoloLens :

here I attached the demo project. So you can test it to see if this is something which possibly could be helpful to achieve your requirements

[usami]

I would like to achieve exactly the same thing on the HoloLens.

Is the "userpick" event available on the HoloLens? Additionally, do I need to physically touch the target model with my finger to trigger that event?

I want to click on a model located far away in a situation where I can't move.

Is it possible to remotely click on a model using gaze or gestures?

[RolandRaytchev]

Hi @usami ,

yes this should work on HL but need to be adapted to this device project typ. The problem is that some widgets (2d widget are not available on HoloLens device) therefore I just used a 3DImage file where I created a svg where the text was displayed and then set it to the image - not perfect but should demonstrate  that this works on HL device. OK I tested it only in preview , because today I do not have access to a  HL device . I attached the project for HL. Please, let me know if this is not working for you - so I will check it on HL device

[usami]

Hi @RolandRaytchev, 
Thank you very much. The project you provided is very close to what I want to achieve.

In the case of a userpick trigger, it is my understanding that direct clicking on the model is required. However, is there a way to select the model from a distance without directly clicking on it?

I am trying to overlay an AR model of pipes onto a factory pipeline, as shown in the attached image. I would like to be able to select the pipe model, but sometimes the pipes are located beyond a fence or in a high-temperature area, making it impossible to approach them.

Is it possible to trigger the selection using gaze input or hand ray with the Hololens?

[RolandRaytchev]

Hi @usami ,

today I was able to test this on a HoloLens 2 device and it seems that the project attached in the previous post is working.

Regarding your question - so the userpick on Hololens device is a click with the mouse only  something which is available on Preview mode - chrome which should provide approximately display for testing.

On the HoloLens device the userpick is actually a old style click- means you gaze to the component so that the gaze symbol change from point to ring and then you tab with the finger (inside the camera field) . This should work also for object which are far from the position of the camera (HL)

Please, test it on HoloLens device and let  me know if this is working for you. Thanks

[usami]

Hi @RolandRaytchev, 

When I tested it with HoloLens, I noticed that the gaze symbol appeared and allowed me to select a remote object.
However, when using ThingMark in the project, I did not see the gaze symbol. Are there any specific requirements or limitations, such as the necessity of using Spatial Target?

[RolandRaytchev]

Hi @usami ,

I am not aware that there should be an difference of the gaze and userpick behavior with Thingmark comparing to spatial target. Possibly is an issue with tracking, so that when you do not see the ThingMark and the extended tracking (traking where the target is not directly in the device camera area). The HoloLens device have always extended tracking but it means that you have also to scan the enviroment so that HoloLens will create a mesh of the enviroment around the Thingmark so that it will hold the tracking when the thingmark is not visible. I will change the project target to thingmark and will test , but currently I do not have to  HL , Possibly tomorrow or in the next few days I could test this and will provide a feedback,

[MarkConig]

Hello, while I do not have a solution, I've been dealing with this issue for years now and would like to provide some extra information. Hopefully we can find a solution.

I work with model and area targets, but I imagine it would be the same issue with Thingmarks. As @usami said, the gaze symbol appears when using spatial targets. When using model and area (and presumably Thingmark) targets, there is no gaze symbol. The strange thing about this is the gaze symbol does exist, and it DOES work, it's just invisible. If you take a guess on what your looking at and do the air tap gesture, the code will run exactly like you were using a spatial target. In my instance, it changes the part's color and displays the name in the metadata, that works, but the user cannot see what they're clicking. Imagine your computer mouse cursor is invisible, but still works if you get lucky and happen to double click your icons.

I've been looking for solutions to either turn on the gaze symbol or make my own dot in the center of my vision, but no luck. I do not know if this is intended or a glitch, but a solution would be greatly appreciated, Thank you!

[RolandRaytchev · Accepted solution]

@MarkConig thanks for the feedback! Your right and your feedback is helpful! So testing with thingmark I can reproduce the behavior as your described (also as @usami mentioned) Did not test it further with MT and AT but I think that I could rely that what you mentioned in your post is the current situation.

Also further check in PTC database I found the following article:

https://www.ptc.com/en/support/article/cs335925

https://www.ptc.com/en/support/article/cs387848

Especially the last one saying:

he Eye gaze cursor in Vuforia View for Hololens is not displayed after tracking a ThingMark or a Model Target
>>>
Description
==============
The Eye gaze cursor in Vuforia View for Hololens is not displayed after tracking a ThingMark or a Model Target
The Eye gaze cursor is shown in an Experience with Spatial Target

Resolution
=============
The Eye gaze cursor only appears when there is something that requires gaze+air tap to interact with. The cases where this is true are:
All experiences with a Spatial target, as the spatial manipulator buttons require gaze+air tap
Any experience that contains a widget when the Click event is bound or executes JS, except if the device is a Hololens2 and the only such Widget(s) are 3D buttons.

So following the logic on this article I simple defined e.g. the following dummy function on the same Project but with thingmark:

//----------------------------------------------------------------------
app.myClick=function(){
console.log("app.myClick();");
};

And added in the UI js box for the click event on the both model widgets (previous project attached on this topic)

Afterword's I tested it and it worked with thingmark as target - it shows again the gaze symbol

[MarkConig]

Thank you very much! That worked perfectly for me, I can now see the gaze cursor when using model targets.

[usami]

Thank you very much! I tested it on Hololens and I was able to completely achieve what I wanted.