Simplified ThingWorx Deployment Setup – How to handle bug fixes efficiently?

[MA8731174]

Hi everyone,

I am currently designing a simplified deployment setup for our ThingWorx environment based on the standard multi-stage pipeline.

Our planned flow looks like this:

Local Instances (Developers) → DEV Environment → Git (Build Pipeline) → Artifact Repository → TEST Server → Production Servers (multiple locations like China, Italy, etc.)

This structure seems efficient for our team and helps streamline deployments, but I’m unsure how to handle bug fixes in this setup.

For example, if a bug is found in production, what’s the best practice for fixing and validating it —
should it go directly from DEV → TEST → PROD again, or is a temporary branch/environment recommended? We are deploying our projects as extensions.

Would appreciate any insights or examples from those who’ve implemented a similar lightweight pipeline.

Thanks in advance!

— Jamal

[VladimirRosu]

Hi @MA8731174 ,

It all stems from your release cadence, acceptance criteria and bug severity labels (note that I don't see STAGING there, which is worrying , since you won't have the guarantee that you'll have a problem-free installation on PROD).

Assuming you have a release cadence of 3 weeks (based on a strong CI/CD chain) you probably will have situations when a breaking Bug manifests during this cycle which makes that release unusable. 

Q1: do you want to manage this situation?

if yes:

Q2: is it acceptable by business to wait until next release cycle?

if not:

Q3: since you need to fix that bug ASAP in the current release, which means you'll need either a temporary environment OR an environment like TEST or QA where you always keep the last version deployed in Prod for such quick fixes (as editable entities since you might need to modify them) . The decision stems from how easy it is to spin up a temporary environment and access rights (we observe in many situations that local dev environments can't be used for such activity because they don't have access to the external systems we're interacting with, and in other cases spinning up resources without approval is not possible because the company does not allow that). Since Dev is unstable, this also means that once fixing that bug in TEST/QA, you'll need to fix it again in Dev (again, assuming Dev is not stable). 

[MA8731174]

Hi, @VladimirRosu  thank you.

Q1: Yes
Q2: No

Regarding Q3 — you’re right, and this is exactly where our main challenge lies. Our project is actively used by production teams inside the company, so when a bug appears, it often needs to be resolved very quickly. Your point about having an environment that mirrors Production, including the same editable entities, makes complete sense. At the moment, our Dev environment is not stable and may not fully reflect the Production environment, which indeed makes it unsuitable for urgent fixes. This is why I’ve been trying to understand the most appropriate place and process for handling hotfixes.

I also had not fully considered the need for a STAGING environment or another stable environment that always carries the last Production build. Your suggestion highlights an important gap in our setup. Having a temporary environment or a dedicated Test/QA environment that always reflects Production seems like the most practical approach for handling fast bug fixes without blocking the release cycle.

Right now, we’re trying to keep the setup simple — mainly ensuring that multiple developers can work in parallel using local instances, pushing changes to Git, deploying to Dev for initial checks, and then moving changes to Test and Prod. But it’s clear that for production-critical teams, we need a more stable environment strategy specifically for hotfixes.

What would you recommend in this case?

Would it be sufficient to introduce a Staging environment and deploy the production-ready version there as editable entities (not as an extension) after every release?

I’m curious if this would align with best practices or if you would suggest a different approach.

[Constantine]

---

Local Instances (Developers) → DEV Environment → Git (Build Pipeline) → Artifact Repository → TEST Server → Production Servers (multiple locations like China, Italy, etc.)

---

I must say that pipeline looks a bit weird. What is the purpose of your DEV Environment, how changes get there and why don't you commit your code to Git from your local instance instead? How do changes get from Test to Prod?..

On my projects I do it differently:

1. Developer implements a change in a personal sandbox, exports it and pushes to Git on a feature branch
2. They open a pull request for the release branch, which acts as a quality gate. The pipeline deploys the PR content on a temporary PR environment, where it can be (re)tested by developer and code reviewer before the PR is merged.
3. Once the PR is merged, the pipeline deploys release branch to the Sprint testing environment, where the rest of the team can test it.
4. At the end of the sprint, if the Sprint Review is successful, the release branch is merged into master, and the pipeline deploys master to preprod, where it will be tested for one more sprint.
5. Once the testing confirms all is good, we tag the master branch, and the pipeline then deploys it to prod (it triggers on tagging).

There are the following environments in this setup:

1. Individual Dev sandbox -- recycled every sprint, deployed approx. once per day
2. Temporary PR environment (optional) -- deployed once and recycled when the PR is merged
3. Sprint Review env -- deployed a few times per day, once per each merged PR
4. Preprod env -- deployed once per sprint (with 1 sprint delay)
5. Prod -- deployed once per sprint (with 2 sprint delays)

The crucial notes:

1. All those environments, including Dev and Prod are deployed using exactly the same shell script (DEV uses Import as Source Control entities though)
2. The deployment is "complete" in the sense that it builds everything, redeploys everything (ALL extensions, ALL code for ALL sub-projects), re-configures everything, etc. -- this ensures that we can rerun it as many times as necessary, and that we can run it against an empty, or existing environment. There's no notion of "short" or "light" deployments, they are all "full", if you see what I mean. It makes life much easier.
3. There are no manual import/export from any of those environments
4. Developers only have access to their dev sandbox Composer
5. PR checklist acts as an implementation of the Definition of Done, and is the most important component in this whole setup
6. Use Semantic Versioning for your tags and extension versions -- this will save you a lot of time

With this setup, hotfixes branch from master, and the PR is opened back to master. Once it is merged, we tag it with another patch version, e.g. "1.0.1", and it gets deployed in the same way as any other deployment to any other environment.

As for the best practices -- read about Git Flow here or here -- it explains exactly how to handle bugfixes and hotfixes. ThingWorx is just a runtime, the process is the same.

/ Constantine

[MA8731174]

Thanks @Constantine for your detailed reply. I appreciate the clarity. I have few questions

1. Development Environment Usage
If developers are working on their own local instances, what is the exact purpose of the shared development environment? Is it still meant to be used for reviewing and validating PR changes before they move further in the pipeline, or is its role reduced in this setup?

---

2. Production Server Redundancy
Is maintaining a single production server sufficient? Given high traffic and occasional downtime, wouldn’t it be more reliable to operate multiple production servers (possibly in different regions) to ensure high availability and minimize service disruption?

---

3. Handling Critical Production Bugs
In cases where a severe bug appears in production and requires an urgent fix, is it considered acceptable from a professional standpoint to keep production running with the existing bug while the hotfix is prepared, reviewed, and deployed—especially if the fix might take time?

Current Goals and Practical Approach

At the moment, we’re not aiming for a very complex setup. Our primary goal is simply to enable multiple people to work on the same project at the same time. At the moment, we only have one shared development environment and no local instances, so collaboration and bug fixing is limited. Introducing local instances with Git already would solve most of these issues — including handling bug fixes through separate branches.

Do you think ? Since we’re not strictly following sprints in the beginning, a simpler workflow would also makes sense? : developers work locally, push to Git (Artifacts ), deploy to the Dev environment for initial review, then move to Test, and finally to Production. Do you think it keeps the process manageable while still supporting parallel development and proper version control?

[MA8731174]

Hi @VladimirRosu and @Constantine 

a little bit background that how / why i came into this topic of deployment..

Current Status:

At the moment, we have three servers: Dev, Test, and Prod. We develop our project directly on the Dev server and then deploy it to Test and Prod as extensions. Currently, we do not have any CI/CD pipeline in place. Initially, a single developer worked on the project for around six months before deploying it to Test and Prod. Since this was the start of our project, the process has been largely manual.

Now, with new feature requests coming in and development underway on Dev, we realized a potential problem: if a bug arises — as has happened twice before — we can no longer quickly fix it on Dev without impacting ongoing feature development. Previously, this was manageable because no new features were being developed at the same time, so bugs could be quickly fixed and deployed to Prod from developer who developed this project.

Additionally, the current approach has a major limitation: only one developer can work on the project at a time. As we plan to expand the team to 4–5 developers and scale the project across multiple locations, this single-Dev setup will not be sufficient. We need local instances of ThingWorx for each developer, along with a proper process for bug fixes and deployments. This is why we are looking to upgrade our deployment strategy and implement a more structured CI/CD workflow.