Moving multiple 3D models using hand gesture

Aditya1702 · ‎Jan 05, 2023

Hi Everyone,

I want to pan, rotate & scale each model separately using hand gesture in a project having multiple models.

Can anyone help me out with this.

Thanks in Advance.

RolandRaytchev · ‎May 02, 2023

I found this older post where was no answer , so I will try to answer here.

I do not know what mobile platform you want to use (HoloLens or Mobile device) but I think there is regarding this issue no big difference . So far I know the answer is that this is currently not possible- we have no real way to track the touch coordinates for the specific rotation , pan or move of components- only one exception - I think there is only possible way of such effect as touch for special target but it does not work for specific components. In this case you will move the position of the model world coordinate. Here related posts:

https://community.ptc.com/t5/Vuforia-Studio/support-touch-screen-thingworx-studio/m-p/549493

https://community.ptc.com/t5/Vuforia-Studio/Translate-or-rotate-object-by-touch-only-in-Vuforia-Studio/m-p/621743

https://community.ptc.com/t5/Vuforia-Studio/How-to-zoom-and-rotate-an-3D-object-by-touch-in-AR-developed-in/m-p/505899

Regarding to a way how to implement some handling of components - the only way what you can use is the tracking event and the coordinate of device location and the eye vector which are passed as arguments to the tracking event callback. I found also similar suggestion in one of the R&D post:

Trick is to use the pos,gaze info to give you an xyz location that is relative to the device.
The navigator widget is an example. Because gaze is a vector, you can add this to pos to give you a point in front of the camera. So for example you can use some pointer (object e.g. a modewidget which is updated dynamicaly)
e.g.  techniques 
1.	tracking the user gaze on hl2
2.	attaching a ‘pointer’ a fixed distance in front
3.	when user ‘doupletap’ (double airtap) the pointer is left in a fixed position in space
4.	 ‘tapping’ to record that location.
a marker could be placed to show this

I think via similar techniques you can find some points in the current augmentation and then interpret them as modification of selected component e.g. 2 points in space could be interpreted as vector for movement. Or they could define a rotation axis and one additional point should define the rotation along this axis. But all this requires that you will calculate the transformation matrix of the specific geometrical change ,of the component position and then translate it new values of the model/modelItem properties x,y,z, rx,ry,ry and this w is not trivial.

Here some post which described related techniques , which could be helpful:

https://community.ptc.com/t5/Vuforia-Studio/Translating-a-model-when-clicked-on-it/m-p/636185#M7112

https://community.ptc.com/t5/Vuforia-Studio/Rotate-model-with-button-click-along-Z-rotation-Axis/m-p/633018

https://community.ptc.com/t5/Vuforia-Studio/Simultaniously-rotate-model-and-image-Vuforia-Studio/m-p/653234#M7709

https://community.ptc.com/t5/Vuforia-Studio/Possibility-to-change-set-the-spincenter-csys-of-models-and/m-p/796282#M10843

https://community.ptc.com/t5/Vuforia-Studio/I-want-to-change-position-of-modelitem-to-move-when-i-click-on/m-p/637163#M7161

https://community.ptc.com/t5/Vuforia-Studio/Model-rotation-please-share-sample-project/m-p/854323#M11534

and these handling a tracking of the eye vector and position

https://community.ptc.com/t5/Vuforia-Studio-and-Chalk-Tech/How-can-we-make-a-3D-Widget-on-HoloLens-visible-in-front-of-me/ta-p/658541

https://community.ptc.com/t5/Vuforia-Studio-and-Chalk-Tech/How-to-create-a-custom-button-pressable-on-HoloLens-device/ta-p/654984

ClayHelberg · ‎May 10, 2023

I implemented a very simple version of this (using vertical swipes to "toss" a virtual object), by leveraging HTML5 touch events. The process goes something like this:

Add a Card widget to the center 2D panel, and use CSS to make it fixed and fill the screen

Attach event listeners for touch events to that panel ("panel-6" in the code snippet), like so:

$scope.init = function() {
  try {
    var container = document.querySelector("*[widget-id='panel-6'] div");
      container.addEventListener("touchstart",$scope.graphicTouchStart);
      container.addEventListener("touchmove",$scope.graphicTouchMove);
      container.addEventListener("touchend",$scope.graphicTouchEnd);
    $scope.originZ = $scope.view.wdg['model-1'].z;
    $scope.originY = $scope.view.wdg['model-1'].y;
  }
  catch (ex) {
    $scope.setWidgetProp("label-1","text","Couldn't set touch listeners: " + ex.message);
  }
}

angular.element(document).ready($scope.init);

Define the graphic event handlers. For your use case, "touchmove" is probably the most important one. You can determine the movement by storing the start location on "touchstart", and calculating the difference. Once you know the direction, you can apply the appropriate rotation to the desired model.
Keep in mind that this approach means that your "touchscreen" panel will intercept most touch events, so Vuforia View may not see certain things like tapping on a button or a 3D model any more. You can manage the visibility of the panel to control when the user is in "normal Vuforia touch mode" and your custom "rotate my model" mode.

For details on how HTML5 touch event handling works, see this excellent article from Mozilla: https://developer.mozilla.org/en-US/docs/Web/API/Touch_events/Using_Touch_Events

This will be a bit of work for you, but hopefully the result will be worth the effort!

RolandRaytchev · ‎May 11, 2023

Hi @ClayHelberg ,

thanks for pointing to this possible solution. So I did test the touch event in the past but seems that it is not supported by each widget- so with card work but e.g. was not working with the panel- at least the event did not fire when I tested it , but possibly was something wrong in my project.

Anyways with card widget seems to work fine. But the calculation of the correct movement -required to be considered more detailed - , how to interpret the measure in windows coordinate to model coordinate is something what need a concept- e.g. in CAD systems e.g. Creo there are some transformations (by transformation matrix - or euler x,y,z,rx,ry,rz) which did the transformation between model, screen and windows coordinates and also component coordinates - e.g. when we want to move a component. So means also it is important that the movement should be not zoom sensitive

Based on your suggestion I want to provide small example which do only movements of an model with simple delta calc, no real transformations -

Also want to mention that this approach works only for mobile project but not for Eyewear -because there is no 2D element - possibly could work in tmlText

here also a good link for touch event : https://w3c.github.io/touch-events/#dom-touchevent-touches