Import Data Table data multiple times

[avillanueva]

I could not find any article or post that discussed this specifically so asking here. Should be an easy one. In the process of migrating apps to a new install of Thingworx, this should be a common concern. So, we have moved over our Entities but now need to move over the data from the data tables. I have 5 that I need to move over but currently, I am in a rehearsal/test mode.

According to CS393743  I should be able to select each, export the data and import to the new server. This fails to address anything about subsequent updates. Since I am in a test mode,  I want to validate things are working so I need the data but data is still being actively updated in production. During a Go live event, I will take the final extract and wish to re-import the data to this new instance. What will occur?

I can see three possible outcomes:

1. Import succeeds and all data exists in new instance (ideal)
2. Import fails as unique entries already exist (bad)
3. Data is duplicated or merged with previous import leading to duplication issues (very bad)

If anyone has been down this road, please reply with your experiences and best path forward.

[GuillaumeL]

Hello,

If I remember well, answer 3 !!

In my case I scripted the import by using following steps. So that at the end of this, we are in answer 1 !!

1- deleting all values

2- Importing values from a CSV file (which was previously exported in CSV)

Here is a piece of code : (name corresponds to the DataTable thing)

	Things[name].PurgeDataTableEntries();

	Things[name].AddOrUpdateDataTableEntries({
		values: csvContent
	});

Regards,

Guillaume

[avillanueva]

So just to be clear, if I had a DataTable like this and did the export/import once, they should be in sync:

Index	Fruit	Count
1	Apple	13
2	Orange	10
3	Banana	8
4	Peach	20

And later in Prod we add Mango and change Apples to 15, we could end up with something like this on next import?

Index	Fruit	Count
1	Apple	13
2	Orange	10
3	Banana	8
4	Peach	20
5	Apple	15
6	Orange	10
7	Banana	8
8	Peach	20
9	Mango	5

That is what I feared. I have to rethink my strategy and see what options I have. Glad I asked first.

[GuillaumeL]

In fact it depends on what is the key of your table ! If Fruit is the key, you should have the 5 lines expected.

There is as well the "deletion" aspect to consider.

[avillanueva]

Fortunately, I do not have to worry about deletion. Thanks, this helps.

[avillanueva]

So I inspected my datatables I need to transfer over. I think the main one has a key that would make that field unique so a subsequent import should just add the additional entries like in my example above correct? The other two are larger and their key is generated using generateGUID() function. Still, I think its the same deal as I do not expect any deletions just new entries. I think this answers my concern that I would not have data duplication given my current setup.

[avillanueva · Accepted solution]

I was able to find a sample data table from an abandoned project to use as a test. I exported it and imported it to my new server. From there, I returned to my source system, added a new entry and made a small change to a field in another. The ID or key of that entry did not change so that's a good thing. Repeated the process of export and and import. No duplication but I did see the updated entry and the new entry appear. Check and Check. Now, if I had made new entries in and mods in my target system, I would expect that there would be some merging of data. Same for deletions in the source. Goes without saying but this nails it. Thanks for the confidence @GuillaumeL .