Include DevOps Capability within the ThingWorx composer when writing services

[eribbeck]

I know that you can write code in other IDE's and then import entities back into thingworx, but this process is very lengthy with many steps between writing scripts and testing functionality within TWX. The other option is to develop directly within TWX, but then you run into source control/overwriting issues if multiple people are working on the same entity at the same time.

To add speed to development and resiliency to version control, something like github built into the composer would be extremely helpful.

[bbeuckSIG]

For version control the GitBackup Extension is available. But it is not integrated in Composer and also does not prevent accidental overwriting. Upvote.

[Constantine]

Just to share some experience -- we develop in individual ThingWorx sandboxes, use mainstream Git clients (IntelliJ, SourceTree, etc.) and merge code in Git via pull requests, reviewing the code at the same time (gitflow). 8 concurrent developers, ~50 commits and ~5 deployments per day, no issues in regards to source control (with the only exception of merging Mashups). You can do reasonably efficient concurrent development with standard ThingWorx and a couple of tiny shell scripts.

[JonathanRigsby]

@Constantine I would love to learn more about your team's solution. We've been struggling to get an efficient source control solution up and running for a while now.

How do you handle getting changes in and out of Thingworx?

How do you handle keeping your dev instances in sync with the latest changes in prod?

How do you handle moving changes from dev to QA to prod?

[Constantine]

Hello @JonathanRigsby,

> How do you handle getting changes in and out of Thingworx?

Getting changes out: A shell script runs curl, which executes SourceControlFunctions.ExportSourceControlledEntities and removes some garbage from XML files (modification date, history, platform version, etc.) to minimize the diff. The essential trick is to specify the modification date during export, which should be after the entities were impoted into the platform in the first place -- that allows to export only changes that you actually did, and not some entities that ThingWorx decided to modify for whatever reason (e.g. change the order of localization tokens). That makes the diff really manageable.

Getting changes in: Git pull, then docker-compose down / up. On the first boot the container loads extensions and XMLs from mounted Git folder. Such "full restart" takes about 5 minutes (ThingWorx 8.5.4 on Postgres + LDAP + eMessage Connector, ~20 extensions, ~500 XML files). As a result for each story or bugfix the developer has a clean environment, which is 100% aligned with Git content. We even created some system-wide hotkeys, so that when you're ready to start working on a new story, you just press WIN+R and go make a coffee -- when you return, your Composer is opened with the clean application, corresponding to the latest state of your sprint branch. You make your changes, press WIN+E to export, open your favorite Git client and commit / push the changes to your story branch. Then you open a pull request and start working on a next story.

> How do you handle keeping your dev instances in sync with the latest changes in prod?

If you're talking about the data, then we don't. We clone PROD database to get PREPROD, when necessary (for release dry runs, for example). For INTEGRATION environment we use data generated by our unit / integration tests, and in QA our test team creates their own data. So we don't really sync PROD data, at least not regularly.

If your question was about the changes in the application code, we deploy complete Git repo every time and ThingWorx ensures that the process is incremental. This application is hosted in PTC Cloud, so we don't have full control to the infrastructure, therefore we have to live with incremental deployments, and so it's hard to retire stuff. On the previous project we hosted it ourselves and followed immutable infrastructure approach (where ThingWorx app was considered part of the infrastructure) -- creating new VMs for each new release and migrating data. Needless to say, this approach makes practical sense only if you can automate it, but once you do -- it allows doing crazy things like swapping the underlying Linux distro, application re-design and re-architecture, etc. as routine operations, so we extensively refactored and improved that ThingWorx application with each new release every 1.5 month.

> How do you handle moving changes from dev to QA to prod?

Our (trivial) CI pipeline handles all deployments, we never import any files on any of our cloud environments by hand. Developers don't even have admin access to QA, PREPROD and (obviously) PROD environments. INT is slightly more relaxed, but we don't really care because this environment is deployed 5 -- 10 times per day, so even if we make changes or break it, nobody is going to notice. This pipeline is configured to deploy Sprint-N branch to INT for every merged pull request (it also runs unit tests), the RC branch is deployed to QA at the end of the sprint, and the Master is deployed to PROD once the release candidate gets approved by QA team three weeks later.

By the way, the same pipeline is used to deploy on local Docker containers, when we start them (that 5 minute startup I mentioned earlier), so there's a good potential for reuse.

Regards,
Constantine

[bbeuckSIG]

I want to give Kudos for this but cannot on the idea platform here. I suggest to copy this as best practice description to the Thingworx Developers forum, here it is not visible to everyone. Then I also can give Kudos

[Constantine]

Thanks @bbeuckSIG,

I'll reformulate it as a practice in a dedicated post sometime later.

/ Constantine

[JonathanRigsby]

Hi, @Constantine. Thank you so much for sharing all that information! Could you give some insight into your testing set-up? How do you tell the Docker container to import the Git folder on load?

If it's out of the scope of this thread, feel free to DM me.

[olivierlp]

A design time environment is under consideration for IDE integration