How to use Splunk services in Thingworx

[rr-31]

I Installed Thingworx7 and imported Splunk extension and configured SplunkThing with Port, UserName, Password.

I Installed Splunk4.4 in my local host.

Now how can I use Splunk to Add/Push Thingworx data.

I checked PersistanceProvider unable to identify newly created SplunkThing.

I am thinking to use Splunk for at least store and retrieve data.

I cant see any of Splunk services for adding of data through Splunk extension.

Please let me know how these operations can be done and also your thoughts for best practices.

Not relevant to above question - I posted same question through chrome and was unable to write anything in issue description, so posting same one again through IE. Not sure if it duplicated.

[Tmetcalf]

Hi Rajesh,

I am moving this to the Developer Community to get the right eyes on it.

Sorry for the posting trouble you experienced; I am working with Jive to resolve the issue - thank you for your patience.

Best,

Toby

[PaiChung]

This should come with a Template that has services.

I believe you post data to Splunk and retrieve analytical results from Splunk through web services

If need be I can find some time to look into this a bit more in depth.

The data isn't 'persisted' in Thingworx for Splunk to analyze from what I remember, but just posted as desired to Splunk.

[Rajesh_A]

Thanks for responding Chung.

1. I referred to Splunk Partnership Ties Together Big Data & IoT Services - ThingWorx : ThingWorx

2.  Installed Splunk

3. Imported Splunk Extension from Thingworx Market and created ThingTemplate with Splunk configuration(Url, Port) in it.

Now Splunk 3 services shown as ; GetUsers, Search &  SendEvent.

I would like to understand how the data Push & Retrieve operations can be done.

Is this can only be done through Web services or any extension provided services can be used?

Looking at alternate storage -> Can I take the Extension advantage of Cassandra DB? e.g easy data push & retrieval operation like same as H2PersistanceProvider (Ignoring H2PersistanceProvider due to limit in size)

If any advise on alternate Thingworx & Big Data uses.

Thanks.

[PaiChung]

So it looks like the SendEvent is used to submit incremental data to Splunk, that means something has to be defined for Splunk to recognize the event and you have to create a payload to submit with it. I pulled an example

// Create an InfoTable

var params = {

  infoTableName : "InfoTable",

  dataShapeName : "StadiumEventData"

};

// CreateInfoTableFromDataShape(infoTableName:STRING("InfoTable"), dataShapeName:STRING):INFOTABLE(StadiumEventData)

var result = Resources["InfoTableFunctions"].CreateInfoTableFromDataShape(params);

// Add a record to the InfoTable

  result.AddRow({

  timestamp : timestamp,

  sourceType : "Thing",

  source : source,

  location : location,

  Temp : Temp,  

  kWh : kWh, 

  Sold : Sold,         

  Venue : Venue,         

  MachineType : MachineType,         

  LocalTimeStamp : LocalTimeStamp,

  Capacity : Capacity

});

// send the record to Splunk

var params = {

  timestampField: "LocalTimeStamp" /* STRING */,

  event: result,

  type: "StadiumVending"

};

Things["LocalSplunkServer"].SendEvent(params);

So it looks like type is something that Splunk recognizes to then take the event and apply it to some analysis model.

Then the Search command is used to pull information from Splunk, this looks to be using Splunk Query code, here is an example I pulled.

var params = {

  searchExpression: "search sourcetype=StadiumVending earliest=-12h | stats avg(Sold) AS UnitsSold, avg(kWh) as kWhUsed by twsource",

  dataShape: "TrendByMachine"

};

// result: INFOTABLE

var result = me.Search(params);

var params = {

  searchExpression: "search sourcetype=StadiumVending twsource = SVM-01* earliest=-12h | stats avg(Sold) AS UnitsSold, avg(kWh) as kWhUsed by twsource",

  dataShape: "TrendByMachine"

};

// result: INFOTABLE

var result2 = me.Search(params);

var tableLength = result2.getRowCount();

for (var x = 0; x < tableLength; x++) {

  var row = result2.getRow(x);

    // Add the row to result

  result.AddRow({

  twsource : row.twsource,

  UnitsSold : row.UnitsSold,

    kWhUsed : row.kWhUsed

});

}

[Rajesh_A]

Hi Chung,

Thanks for your detailed info.

Your assumption about Search operation (Splunk Service: "Search") and Users (Splunk Service: "GetUser") is working and I can able to fetch data from Splunk DB. Thank you again.

But the important feature "Add Data" service is failing through "SendEvent" Service.

I followed your SendEvent note: Things["LocalSplunkServer"].SendEvent(params) , but this did not work. It always thrown exception: " Wrapped java.lang.NullPointerException Cause: null"

I started reading Splunk related docs to find the appropriate function for "Add Data" service and that is HTTPEventCollector.

But I could not match HTTPEventCollector with SendEvent Service, did not get any idea how to implement in SendEvent.

Then I went for REST Api option through Resources["ContentLoaderFunctions"].PostJSON(params)

Below is code I used, but it always thrown SSL related exception: Wrapped javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? Cause: Unrecognized SSL message, plaintext connection?

Script:

var params = {

  proxyScheme: undefined /* STRING */,

headers: { 'Content-Type': 'application/json', 'appKey': 'C7B4DF75-C03C-4CCC-9B5C-2D7037A4663F' } /* JSON */,

    ignoreSSLErrors: undefined /* BOOLEAN */,

  useNTLM: undefined /* BOOLEAN */,

  workstation: undefined /* STRING */,

  useProxy: undefined /* BOOLEAN */,

  withCookies: undefined /* BOOLEAN */,

  proxyHost: undefined /* STRING */,

url: "https://localhost:8088/services/collector/" /* STRING */,

  content: "{event:My first Splunk Data}" /* STRING */,

  timeout: undefined /* NUMBER */,

  proxyPort: undefined /* INTEGER */,

  password: undefined /* STRING */,

  domain: undefined /* STRING */,

  username: undefined /* STRING */

};

// result: JSON

var result = Resources["ContentLoaderFunctions"].PostJSON(params);

Got appKey from Splunk -> HTTPEventCollector

So could you please here.

Thanks.

[PaiChung]

In regards to Alternate storage.

We support H2, PostgreSQL and Cassandra.

H2 and PostGreSQL can store both Model and Data, Cassandra is only for Data

You can create additional Persistence Providers as desired as a way to expand the ability to store data.

Postgres can store a lot of data, Cassandra doesn't just store a lot of data, it also provides a much higher rate of ingress.

Cassandra is a paid service that you will need to sign up with separate from licensing Thingworx.