cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - Need to share some code when posting a question or reply? Make sure to use the "Insert code sample" menu option. Learn more! X

Massive data exportation in csv

LuisCabello
12-Amethyst

Massive data exportation in csv

Hello community,

 

I have a requirement for one customer. The customer would like to export in csv 1 millon of rows.

To clarify the use case: Take the data in csv from one year  from 100 devices.

I'm using the CSV Parser extension (https://support.ptc.com/help/thingworx_hc/thingworx_8_hc/en/index.html#page/ThingWorx/Help/Extensibility/CSVParser.html)  & the Data Exportation Widget (https://support.ptc.com/help/thingworx_hc/thingworx_8_hc/en/index.html#page/ThingWorx/Help/Mashup_Builder/Widgets/DataExportWidget.html)

but the both using infotables to process the data and im not sure if a infotable is a good element 

to do this...

1. Exist any other method or solution for massive data exportation?

2. Exist a way to put  in "second plain" the generation of the csv? 

 

Thank you in advance,

Luis.

 

1 ACCEPTED SOLUTION

Accepted Solutions
PaiChung
22-Sapphire I
(To:LuisCabello)

How many rows is a bit dependent on how much memory you have available so I can't say for sure. May need a little experimentation to find out what is the most efficient way.

To create the link, the FileRepo has a service GetFileListingsWithLink I believe.

View solution in original post

3 REPLIES 3
PaiChung
22-Sapphire I
(To:LuisCabello)

I would not use the data export widget, this would require all the rows to be loaded into the mashup.

I would export the data server side to a File Repository and then make a download link available to the client.

This will also allow you to retrieve/write the data in batches if a single batch proves to big. File Repository will have Write and Append services available.

Hello @PaiChung,

 

Thank you for your answer, Oks, so for example, the solution would be create different CSV Files with the services of CSVParser, and later append together in one zip with FileRepository services that must be accesible by one link. My questions now are:

1. How many rows I can have in each file more or less?

2. How I can create the link to the user? 

 

Thank you in advance,

 

Luis.

PaiChung
22-Sapphire I
(To:LuisCabello)

How many rows is a bit dependent on how much memory you have available so I can't say for sure. May need a little experimentation to find out what is the most efficient way.

To create the link, the FileRepo has a service GetFileListingsWithLink I believe.

Top Tags