Massive data exportation in csv

[LuisCabello]

Hello community,

I have a requirement for one customer. The customer would like to export in csv 1 millon of rows.

To clarify the use case: Take the data in csv from one year  from 100 devices.

I'm using the CSV Parser extension (https://support.ptc.com/help/thingworx_hc/thingworx_8_hc/en/index.html#page/ThingWorx/Help/Extensibility/CSVParser.html)  & the Data Exportation Widget (https://support.ptc.com/help/thingworx_hc/thingworx_8_hc/en/index.html#page/ThingWorx/Help/Mashup_Builder/Widgets/DataExportWidget.html)

but the both using infotables to process the data and im not sure if a infotable is a good element 

to do this...

1. Exist any other method or solution for massive data exportation?

2. Exist a way to put  in "second plain" the generation of the csv? 

Thank you in advance,

Luis.

[PaiChung]

I would not use the data export widget, this would require all the rows to be loaded into the mashup.

I would export the data server side to a File Repository and then make a download link available to the client.

This will also allow you to retrieve/write the data in batches if a single batch proves to big. File Repository will have Write and Append services available.

[LuisCabello]

Hello @PaiChung,

Thank you for your answer, Oks, so for example, the solution would be create different CSV Files with the services of CSVParser, and later append together in one zip with FileRepository services that must be accesible by one link. My questions now are:

1. How many rows I can have in each file more or less?

2. How I can create the link to the user? 

Thank you in advance,

Luis.

[PaiChung · Accepted solution]

How many rows is a bit dependent on how much memory you have available so I can't say for sure. May need a little experimentation to find out what is the most efficient way.

To create the link, the FileRepo has a service GetFileListingsWithLink I believe.