Skip to main content
L
LuisCabello13-Aquamarine
13-Aquamarine
April 11, 2023
Solved

Massive data exportation in csv

  • April 11, 2023
  • 1 reply
  • 1433 views

Hello community,

 

I have a requirement for one customer. The customer would like to export in csv 1 millon of rows.

To clarify the use case: Take the data in csv from one year  from 100 devices.

I'm using the CSV Parser extension (https://support.ptc.com/help/thingworx_hc/thingworx_8_hc/en/index.html#page/ThingWorx/Help/Extensibility/CSVParser.html)  & the Data Exportation Widget (https://support.ptc.com/help/thingworx_hc/thingworx_8_hc/en/index.html#page/ThingWorx/Help/Mashup_Builder/Widgets/DataExportWidget.html)

but the both using infotables to process the data and im not sure if a infotable is a good element 

to do this...

1. Exist any other method or solution for massive data exportation?

2. Exist a way to put  in "second plain" the generation of the csv? 

 

Thank you in advance,

Luis.

 

Best answer by PaiChung

How many rows is a bit dependent on how much memory you have available so I can't say for sure. May need a little experimentation to find out what is the most efficient way.

To create the link, the FileRepo has a service GetFileListingsWithLink I believe.

1 reply

P
PaiChung22-Sapphire I
22-Sapphire I
April 13, 2023

I would not use the data export widget, this would require all the rows to be loaded into the mashup.

I would export the data server side to a File Repository and then make a download link available to the client.

This will also allow you to retrieve/write the data in batches if a single batch proves to big. File Repository will have Write and Append services available.

    L
    LuisCabello13-AquamarineAuthor
    13-Aquamarine
    April 18, 2023

    Hello @PaiChung,

     

    Thank you for your answer, Oks, so for example, the solution would be create different CSV Files with the services of CSVParser, and later append together in one zip with FileRepository services that must be accesible by one link. My questions now are:

    1. How many rows I can have in each file more or less?

    2. How I can create the link to the user? 

     

    Thank you in advance,

     

    Luis.

    P
    PaiChung22-Sapphire IAnswer
    22-Sapphire I
    April 18, 2023

    How many rows is a bit dependent on how much memory you have available so I can't say for sure. May need a little experimentation to find out what is the most efficient way.

    To create the link, the FileRepo has a service GetFileListingsWithLink I believe.

      Terms & ConditionsCookie settingsAccessibility statement