Shaders created in Vuforia Studio are not working on Hololens, but on mobile (both IOS and Android).

[rennerd]

I've developed an shader, which displays flow movement in a pipe.

The shader works fine on Android and IO's, but on the Hololens it looks like the shader is not applied to the model.

Sadly i cannot provide the project, but i am able to share snippets of the shader:

fragment:

    float diff = mod(-vUV + t, .024);
    float smallerDiff = mod(-vUV + t, .024);
    if(diff < 0.008)
    {
        // ...
        vec4 normalInViewSpace = //removed
        if (/* removed */)
        {
            vec3 color = vec3(1.,1.,0.);
            gl_FragColor = vec4(color, .5);
        }
        else
        {
            discard;
        }
    }
    else 
    {
        discard;
    }

vertex:

    vNormal = vertexNormal;
    vec4 vp = vec4(vertexPosition, 1.0);
    gl_Position = modelViewProjectionMatrix * vp;

The shader is updated at runtime with an interval:

$scope.view.wdg['model-1']['shader'] = "reflect;time f " + z;

Thanks in advance! 

[AllanThompson]

From my testing shaders don't work at all on the HoloLens.  I've put shaders in mobile experiences that work fine, but when I try them on the HoloLens, nothing happens.

There are things that are available for mobile that aren't on HoloLens e.g. the new option for casting shadows.  A while ago, HoloLens experiences couldn't display textures, but that did get fixed. 

[RolandRaytchev]

There are 2 specific  things where we have to pay attention on the HoloLens. 

1.) is to call all functions calls  when the models are alreday loaded. This means it is recommended to call all shaders from a modelload event - example:

$rootScope.$on('modelLoaded', function() { 
$scope.view.wdg['modelItem-1']['shader'] = "xray;";
$scope.view.wdg['modelItem-1']['shader'] = "";

  $scope.render_status=0;
setInterval(function() {

  $scope.$apply(function () {
    if($scope.render_status==0)
    {$scope.render_status=1
     $scope.setWidgetProp('modelItem-1', 'shader', 'xray;');
     $scope.setWidgetProp('modelItem-2', 'shader', 'Default;');
     
   // $scope.view.wdg['modelItem-1']['shader'] = "xray;";
   // $scope.view.wdg['modelItem-2']['shader'] = "";

    }
    else
    {$scope.render_status=0
     $scope.setWidgetProp('modelItem-1', 'shader', 'Default;');
     $scope.setWidgetProp('modelItem-2', 'shader', 'xray;');   
    }         
  }
               );
}
            , 1500);

2.) the other point to use setWidgetProp method - instead of the statement:

$scope.view.wdg['modelItem-1']['shader'] = "xray;";
 //use this 
$scope.setWidgetProp('modelItem-1', 'shader', 'xray;');

 For static shaders this seem to work fine on the HoloLens

Here is  an example for the xray shader where I tested it successfully . The example code will change ever 1.5 seconds in an interval function the shader display of modelitems (from shader xray to default and vice versa)  To use it we have to create a TML text widget and paste the shader devinition on it.

Here the shader definition I used for the test (example from dev team)

<script name="xray" type="x-shader/x-fragment">
  precision mediump float;

  varying vec3 I;
  varying vec3 N;

  const float edgeFalloff = 1.0;
  const float intensity   = 1.0;
  const float ambient     = 0.2;

  void main() {
    vec4 color     = vec4(1.0);
    float opacity  = abs(dot(normalize(-N), normalize(-I)));
    opacity        = ambient + intensity*(1.0-pow(opacity,edgeFalloff));
    gl_FragColor   = opacity * color;
    gl_FragColor.a = opacity;
  }
</script>

<script name="xray" type="x-shader/x-vertex">
  attribute vec3 vertexPosition;
  attribute vec3 vertexNormal;

  varying vec3 I;
  varying vec3 N;

  uniform mat4 modelViewProjectionMatrix;
  uniform mat4 modelViewMatrix;
  uniform mat4 normalMatrix;

  void main() {
    vec4 vp4    = vec4(vertexPosition,1.0);
    vec4 P      = modelViewMatrix * vp4;
    I           = P.xyz - vec3(0);
    N           = vec3(normalMatrix * vec4(vertexNormal,0.0));
    gl_Position = modelViewProjectionMatrix * vp4;
  }
</script>

So it seems that shaders will work on the HoloLens in generally and in your case we have some specific problem which needs first to be indetified.

[RolandRaytchev]

After some further checks I also think that the shader functionality is not working / or not working correctly on a  HoloLens device. It seems to work fine on IOS and Android devices.

In the test what I mention in my previous post , setting of the shader property did  change the appearance of the model Item on HoloLens and I had the impression that it looks ok – color and transparency are similar to the expected,   but when I tested it more detailed I  found  that it was not correct( or not 100% correct – compared to the display on Android or IOS), so means that it did not display the functionality what  the shader does implement.

[RolandRaytchev]

So reported issue to PTC Development Team. 

Answer:

OpenGL   is not supported on the HoloLens.

=================================

[rennerd]

What's then supported?

How about DirectX(HLSL)? If it's supported, can you provide the available attributes?

Or isn't there any alternative at the moment and shaders won't work at all?

[RolandRaytchev]

The problem here is that studio is on windows and the preview will work on WebGl - I think for the graphic is three framework environment. Of course it understand WebGl and shared could be passed to the envoronment

On the enddevice the code is running on Vuforia View. The Vuforia View is implented for the different plattforms with differnt tools. When the vufora view does not use some functionality you could not access it anywhere.

Because the HoloLens has tools for JS openGl - I did ask already the develpment:

=========================

Do you know if we  could write a shader using other API - based on Microsoft .- for example - HoloJs - is possible to use it instead?
https://github.com/Microsoft/HoloJS

Answer from development:

No support to recognize that would need to be added into View

=====================================

So as see - it is not possible yet. If you need this functionality you have to report a product Idea or discuss the requirement of this functionality with your sales presentative.

[RolandRaytchev]

Hi @rennerd ,

I saw this comment here because there was a request where this point was mentioned.

So far at the current time:

- for Android and IOS devices we can use GLSL shader - means it is now official supported as mentioned in the PTC help:

http://support.ptc.com/help/vuforia/studio/en/#page/Studio_Help_Center%2FWidgetTMLText.html

http://support.ptc.com/help/vuforia/studio/en/#page/Studio_Help_Center%2Fmetadata%2FMetadata_201_Highlight_Parts.html

https://github.com/ptc-ar-sharing/vuforiastudio/wiki/TML-Text

https://github.com/ptc-ar-sharing/vuforiastudio/wiki/TML-101---Shaders

Ok , some of the mentioned links do not contain much information but it means that the TML + Shaders should be official supported.

Also there is mentioned that for the HoloLens we can create shaders but in this case we need to use the syntax of the HLSL where the principle of the shader implementation is the same as GLSL /calculations but some parts are different especially the variable definitions

here example for vertex shader with GLSL for mobile:

<script name="test" type="x-shader/x-vertex"> 
attribute vec4 vertexPosition; 
attribute vec2 vertexTexCoord; 
varying vec2 texcoord; 
uniform mat4 modelViewProjectionMatrix; 
void main() { 
gl_Position=vec4(modelViewProjectionMatrix * vertexPosition);
 texcoord = vertexTexCoord - 0.5; } 
</script> 

and here below the same shader as HLSL for the HoloLens (in TML text area):

<script name="pingerhl" type="x-shader/x-vertex"> 
cbuffer ModelConstantBuffer : register(b0) {  float4x4 model;  float4x4 inverse; }; 
cbuffer MaterialConstantBuffer : register(b1) {  float4 diffuseColor;  bool twoSided; }; 
cbuffer ViewProjectionConstantBuffer : register(b2) {  float4x4 viewProjection[2];  
float4x4 viewInverse;  
float4x4 viewRotationInverse; }; 
struct VertexShaderInput {  half4 pos : POSITION;  half4 normal : NORMAL;  half2 texcoord: TEXCOORD;  uint instId : SV_InstanceID; }; 
struct VertexShaderOutput {  half4 pos : SV_POSITION;  half2 tcoord : TEXCOORD0;  
uint rtvId : SV_RenderTargetArrayIndex; }; 
VertexShaderOutput main(VertexShaderInput input) {  
VertexShaderOutput output;  
half4 pos = half4(input.pos);  
int idx = input.instId % 2;  
pos = mul(pos, model);  
output.tcoord = input.texcoord - 0.5;  
pos = mul(pos, viewProjection[idx]);  
output.pos = (half4)pos;  output.rtvId = idx;  
return output; } 
</script>';

So we see that based on  HLSL there is another syntax required ->refer to:

 https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-writing-shaders-9  ,    

https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-pguide,

 https://docs.microsoft.com/de-de/visualstudio/debugger/graphics/hlsl-shader-debugger?view=vs-2019

https://codepen.io/alaingalvan/post/glsl-vs-hlsl

https://github.com/aras-p/hlsl2glslfork

and a converter from GLSL to HLSL  https://pema99.github.io/glsl2hlsl/  

Unfortunately, the last link does not convert to HLSL but some Unity shader format.  In case that you have GLSL code already you can try to search for some GLSL2HLSL shader or try to convert simple using the HLSL manuals ... https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-reference

https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-language-syntax

https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl

[RolandRaytchev]

Here is an example /sample project  for ping shader for HoloLens. The project requires 2 definition of the same shader - one for GLSL and one for HLSL. So here we can check the current mode and if it is preview then use the pingergl shader and otherwise we can use the pingerhl shader. In the demo project the both shaders are defined in separate TMLText widgets. To check the mode called window.twx.app.isPreview()

[RolandRaytchev]

new version of the Eyewear project attached here - it contains also implementation of the reflect shader for both  preview and HoloLens device:

[Wes_Tomer]

Could you please upload a brief video clip (screen recording) showing this in action? I'm not sure I understand what it looks like, but it sounds clever. I too am trying to show oil flow through tubes and cavities within a pump.

[RolandRaytchev]

Hi @Wes_Tomer , 

the best impression of it - hwo it looks like will be if you load the demo project form the post https://community.ptc.com/sejnu66972/attachments/sejnu66972/studio/10291/1/propSetComponetsHoloLensExamleCommuntyShaderUsageReflect.zip  and test it in the Studio environment. You need simple import it as project in the env. 

The main goal in the previous post is to provide an example and to demonstrate  how to create a shader which will work in preview mode (GLSL also it will work on android and on IOS) but should work also on eyewear device (HoloLens 2 as HLSL) The shader definitions sample are for reflect , xray and pinger shaders.

Regarding to pipe flow shader  could check this post : https://community.ptc.com/t5/Vuforia-Studio/Simulate-Fluid-Flow/m-p/670190#M8349

This post is working currently only for GLSL –(Preview, android and IOS) if you need it for HL2 device , then ,please, let me know so I will try to create a HLSL example of it. Thanks!

sample project for fluid flow