Create parquet file from Thingworx Extension

[gfontana]

Hello everyone,

 I've created an extension that convert a file from csv to parquet and send it to Event Hub. 

The creation of parquet is based on spark-3.3.0-bin-hadoop2  libraries installed on file system and referred by an enviroment variable. 

This scenario is working well but i wondering if is possible perform the same action loading the spark-3.3.0-bin-hadoop2  as a Thingworx extension. In this case how  can I referred the library in an environment variables ?

Answering question  could solve the problem when Thingworx is installed in a cloud environment. 

Thank you

[VladimirRosu · Accepted solution]

Is the code you're running with that library referencing any existing Hadoop installation? In other words, does that library needs access to an existing Apache Hadoop installation?

[gfontana]

Hello Vladimir,

I've discovered that, in reality, if you include the right library in your project you can avoid to referred the Hadoop installation.

It's a forcing because the Hadoop installation is always searched, when you generate the parquet file, but if it's missing the writer use those jar referred locally; so the extension works well.

Thank you for your answer and have a nice day 

Giorgio Fontana

[VladimirRosu]

Very good information. So, did you manage to use that library without having any conflict with the built-in ThingWorx JAR libraries? Usually external JAR libraries also have some dependencies, which in turn can conflict with the ThingWorx libraries (but not necessarily). I looked at the library you mentioned and it had 240 JAR libraries in there as dependencies. If it works as an extension this means that all those libraries are already included in ThingWorx, which is a very lucky situation.

[gfontana]

In my experience, developing the extension with eclipse, every time you include some external jars and run the extension is the system that notify those conflict.

So it's enough remove them from your project and the game is solved. This is happened also in this case, is a boring works but at the end it worth.

Thank you (also for the kudos)

[VladimirRosu]

(your welcome)

Still, you're in a lucky situation, because if there is a conflict, and the library you would try to use requires a newer version of the JAR library then the one you have in ThingWorx, you could possible have a blocker (eg: if a method signature changed). Meaning that if there's significant differences you could reach a blocker.

For future users, if I would be in such a situation, I would package the extension as a JAVA Edge SDK implementation (effectively very small difference to an extension SDK, but you need to deploy it somewhere and make sure it runs )

[gfontana]

your absolute right