ModelCheck Usage - Experiences/Recommendations

[BrianMartin]

Hi Everyone,

I'm in a situation where I work for an organization using ModelCheck. However, in my humble opinion, it's being used incorrectly. While I believe I have the credentials and experience to make this judgment call, I wanted feedback from the greater PTC Community so I thought I'd pose the question here and see what the responses are.

This particular organization has three prongs to their implementation of ModelCheck:

1. Tight, restrictive settings. Different "codes" can be applied (via parameter) to each model as a way to ease restrictions. For example, a concept model has less restrictions than a model set for imminent release. However, in all cases the settings are very restrictive.
2. ModelCheck runs at every save operation. It takes between 5 seconds and 3 minutes for Creo to return control to the user during a save. Users save less so they don't have to suffer the performance hit. If ModelCheck crashes the session of Creo, your work is lost. There is no alternative method to save. Even "Save As --> Backup" can crash causing lost work.
3. Windchill GateKeeper is running. All errors must be resolved before you can check anything in. The organization hasn't yet identified all use cases where some easing of restrictions might be warranted - sometimes you simply have to DELETE work you wish to keep because you cannot check your file in otherwise. Some users get around this by simply refusing to check in.

In my personal opinion (I get a bit heated here), ModelCheck operating in Save Mode is among the top 5 dumbest ideas PTC has ever floated. They recommend it - but they don't recognize performance inefficiencies because they do not live with them. Their careers do not depend upon getting product out the door using Creo.

The organization feels:

• If they provide any wiggle room, people won't run ModelCheck
• If they make running the tool optional during the design phase - but required during release, people may wait to the last minute, not run the tool, then have to fix potentially dozens of errors with no time to do so
• Restrictive settings insure the models are clean and efficient.
• Running ModelCheck during every save is efficient because it's forcing the designers to fix problems multiple times a day.

In my experience and opinion, this restrictive implementation is hurting our schedule, budget, productivity, accuracy, and morale. Soliciting experiences from other professionals and organizations may help. If I were able to prove that the majority of companies do not use "ModelCheck Save Mode",  this would eliminate part of the problem. If I could further prove that easing restrictions during a design phase was commonplace - and that more restrictive standards are more appropriate just prior to release, that might also help.

What is your experience running and using ModelCheck in your enterprise? Do you run it at every save? How do you mitigate crashes? Do you have multiple levels of error-checking restrictions (concept more permissive, release-ready more restrictive)? And finally, what is your experience with user's "waiting until the last minute" then needing relief?

In my opinion, the concerns can be addressed with training and proper project management. If designers are reminded that they must allow time to clean up their models (and supervisors are aware this is part of the designer's job) no one should be caught off guard. If so, the designer will learn that hard lesson ONE TIME and in the future, they'll be wiser. I believe people will take the easiest path in most situations. They will learn to work with the tool as long as it does not represent a significant performance and productivity hit.

What do you think??

[lwestbrook]

Thanks for posting this, Brian. I'm eager to hear others' responses to your questions. 

Though I'm not in your situation, and therefore cannot comment from the customer perspective, I tend to agree with you that there must be a better way. ModelCHECK is tricky in that you want to ensure quality models, but in such a way that it isn't a prohibitive burden for users. It does seem like it might be beneficial to use warnings instead of errors earlier in the design process. That way, users can see the issues they'll need to fix later on, but don't need to worry about fixing them immediately. I can understand the frustration of having to wait for ModelCHECK to run after every save, but the alternative would be to leave it to users to run ModelCHECK in Interactive mode, which is a manual step that might get missed.

So in my view, the best path forward isn't straightforward, but hopefully we'll get good responses from other folks. 

Thanks again,

Luke

[BrianMartin]

Hi Luke...

I agree that the ModelCheck step might get missed. And I agree that there are good reasons to use the tool. My issue is that this particular organization has dialed up a bad combination of restrictions which equal lost work, lost productivity, less frequent saves, less frequent (or completely avoided) check-ins, and no strategy whatsoever for mitigating the problems being caused. Instead, it's "the designer's problem".

Someone else caused a problem with a file and it's impacting YOUR work? Tough.  Delete your work! Do it again a second time later! Or just figure it out and quit whining!

It's assumed the designers would prefer to check in crappy models. It's assuming everyone will shirk responsibilities. Or, worse, it's assuming most people will do their job properly - but a few bad apples will not  so therefore everyone must suffer equally.

It's just a really ridiculous stance - but IT management sees no problem with it at all! Hopefully we'll get some good responses. Maybe it'll help!

[BenLoosli]

I tend to agree with what you wrote. We are currently not running ModelCheck on a very large project and that is simply because I haev not had time to set it up. I have setup and configured ModelCheck at 2 other companies, so I do have some experience with it. I do have another system that does have ModelCheck enabled but I have only marginally tweaked that system configuration a few times in the last 11 years. 

I would not rely on a user setting a parameter to flag which ModelCheck rules to execute. They will leave them at the minimal check if at all possible, unless you have someone review the ModelCheck result files as part of the release process.

Having ModelCheck run at save and flagging errors may be a time consumer, but it is minimal. Also it might be easier to fix a flagged error earlier in the design process than later. ModelCheck rules should be reviewed with the users and management to be sure that they are checking the important things and not something mundane that doesn't matter. Do your designers review the ModelCheck results after each save to be sure they have clean models as they go along?

From your comments, it looks like PTC consultants set up ModelCheck per management 'goals' and did not consult with the end users about all of the issues it can cause. Setting the Windchill gatekeeper to not allow a check-in due to a single ModelCheck error is asking for trouble. Maybe set the threshold at 10 or 25 errors, so the file can be checked-in, but then it MUST be fixed. Marking a check to be an error versus a warning is critical to using the tool effectively.

[BrianMartin]

Hi Ben,

Thanks for your reply. You're totally right - having different codes to trigger different checks is a mess. It was implemented specifically so we could have the most restrictive environment possible. For instance, suppressed features are disallowed (why?) but in some tools like AFX (Advanced Framework Extension), a suppressed feature is part of the tool. Suppressed features are used as part of the AFX module. If you disallow those features, many beams and components generated by AFX cannot be checked in. So there's a code to allow a suppressed feature for AFX. But if you know that code (and can avoid other errors), you can use the AFX code to skirt the checks. It's a bad rabbit hole to go down.

Note that there are several ways this could be fixed:

• Make the ModelCheck errors simple and meaningful. Don't check irrelevant items.
  • This organization views everything as relevant (except a few things which actually ARE relevant but they ignore like buried features).
  • There is no wiggle room. Resolve ALL errors ALL the time...incessantly. If you do not, you cannot check in your file. If ModelCheck crashes your model and all work is lost - it's YOUR problem.
• Prior to release, allow models with a handful of errors to be checked in (maybe 3 is acceptable, for instance)
  • This organization says ALL or nothing. Show me one Creo model you've ever done that was so egregiously wrong you needed ModelCheck to rescue you before you checked the file in. Then, once fixed, show me a Creo file that gets so messed up between saves that you must re-re-run ModelCheck again just to make sure the universe won't rip in half after you check it in.
• Allow users to run on-demand but require all models to be correct prior to release
  • This is what many companies do. Everyone should be interested in adhering to basic standards but the user should be empowered to choose when and where the tool is used - up until release. 
  • At release, you adhere to standards or you just receive an override from the configuration management folks. There is no override in this case. Again, all or nothing with no strategy for mitigating the problems being caused.

In this case, PTC did not set up ModelCheck. I have to be careful what I say because this will likely make its way to the very people I am taking about so I'm torn to explain how it got this way. Our own people set it up. They did not consult users... "the lowly stupid user doesn't know what's good for him". It's a bad combination of egos, inexperience, misguided goals, misunderstanding of the tool, lack of confidence in the designers, poor training, poor management all around, and so many chips on so many shoulders it looks like a forest.

I do have to strongly disagree with you that ModelCheck running at each save is "not much productivity hit". In this environment I wait an average about 45 seconds per save and up to 3 minutes. If I save every 15 minutes, I lose about 5 minutes an hour (and sometimes more). At the end of every 12-hr day (and I work MANY of these) I have lost one hour to simply saving my model and waiting for ModelCheck. At four hours per week per designer and 25 designers - that's 100 hours a week - 400 hours a month - and nearly 5000 hours a year for nothing. We could hire 2.5 more engineers, make all of them checkers, and have the better results. 

The simple fact is... people do fix their models as they go along. But sometimes things happen! Someone else will remove a feature from a part - let's say they delete a hole and re-place a new one somewhere else rather than simply MOVING it. You lose a placement reference and your model assembly model is now "red" with feature errors and component placement errors. You cannot suppress the failed component while you go back to address the hole issue with the part designer. You cannot fix the failed component into position. Your only option - is to delete the failed feature. Otherwise, you cannot check your model in. Period.

We are checking mundane errors and relevant ones. But all are treated as dire and serious. Some model "codes" which can be applied allow one or two specific errors through  so people "code shop" trying to find any code which will allow them to save and check in their work. No one cares what code you use - we're just trying to get files checked in so we can collaborate. There must be 15 different codes and all of them are too restrictive.

I think if you worked with this system for a week, you'd be appalled and trying to alert someone that this combination of restrictions is hampering work. Your experience with the tool would be noted - and ignored. It's like saying:

"Thank you for your irrelevant opinion!" You're only complaining because you like to make bad models and we are stopping you! (See the adversarial way this plays out? You, the designer, are TRYING to cause problems and ModelCheck is forcing you to do your job - because unless you're forced, you're a moron who will purposely perpetuate bad models.)

[lwestbrook]

Hi @BrianMartin 

You made the comment "If ModelCheck crashes your model and all work is lost..."

I want to make sure you're aware of the config (SAVE_MC_PRE) that will cause ModelCHECK to run AFTER the save happens, rather than running ModelCHECK before the save and thus risking data loss.

Of course, the problem with this config is that if the model is saved before ModelCHECK runs, then the parameters from ModelCHECK don't get saved after ModelCHECK runs.

So I'm not really sure if this would be too helpful for you, but I at least wanted to let you know it exists, in you weren't already aware.

Luke

[rreifsnyder]

Hi Brian,

First, we have not totally implemented Modelcheck as I would like but here is my philosophy and where I'm trying to drive my company. First, we setup a button to make running Interactive Modelcheck readily accessible in the Quick Access Toolbar. We are NOT using gatekeeper to enforce absolutely error free models on checkin. There are a couple of reasons for this, first we don't want to discourage people from checking in and there simply are times that you can't completely eliminate a few specific errors. What we do have is the ability to force the user to run Modelcheck as the very last thing they do prior to doing a save if they are going to be checking in the model. So the user can save without running modelcheck but if they are going to check in the model then they MUST run modelcheck LAST. When MC is run it sets a parameter called MC_VERIFIED to YES and if any other change is made after that it the parameter changes to NO. This at least puts the results of the MC in front of the user on an ongoing basis. We then use Release Validators to prevent models with MC_ERRORS from being released. One major drawback is that there is still the issue of some errors that simply can't be fixed so our Configuration Managers can override the issue which of course can tempt them when deadlines approach.

There is simply no foolproof method to give the users the flexibility they need to get their work done but also force the discipline needed for good modeling. This is simply the balance I have tried to strike.

[BrianMartin]

I could not agree with this more if I tried.

A balance is what is needed. ModelCheck is the THIRD button on our Quick Access Toolbar. I put it there intentionally so people could access the tool quickly and often during their design work. The approach was going to be encouraging manual usage of the tool and making minimally restrictive rules for GateKeeper (for those who don't know, this is the tool which controls whether models can/cannot be checked in unless ModelCheck has been run). 

Unfortunately, this approach was abandoned in favor of "fix all errors - even minor ones  at once, all the time - no exceptions!"  Sadly, it's all ego. I recognize "ego" - because I have one myself but I always try to listen to differing feedback. I will admit a mistake and I will change my opinion when confronted with reality. In this case, the decision-makers are either completely inexperienced with Creo or are so insulated from the problems being caused that they do not see an issue. They're more invested in staying the course because any change might be taken as an admission a mistake was made. There are a dozen ways to mitigate the problem without damaging anyone's credibility but there's no interest in doing so.

I fully support the approach you shared. I support both running ModelCheck just prior to checkin (or prior to release - I am flexible on this point). I also support resolving all serious errors prior to release as long as there's an override capability for situations that cannot easily be resolved. I'm not saying everyone gets an easy PASS... I'm saying there are times when it's warranted to allow something to slide (imported model, COTS item, etc) and an administrator should have an override tool available.

In general though, I'm just trying to advocate for a balance but I am getting nowhere. Thank you for your feedback. This is the kind of reasonable approach I hoped to hear about.

[StephenW]

I'm strictly a user, I have never set up Modelcheck. I have had input on the setup. I've worked at 3 companies that use modelcheck in various ways. Company 1 set it up for the user to run (I doubt anyone ever ran it after the first few times). Company 2 had a much more restrictive setup and only ran when the user ran it AND they had a manual check afterwards to verify all errors were corrected. Company 3 (current company) is also doing the user run with a verification via an attached report by our documentation people. 

I agree that if left to the user to run, Modelcheck would never be run. I would skip it everytime. 

I also agree that an every save Modelcheck run is absolutely a waste of time and would directly result in fewer saves and more "lost work" due to crashes. I also feel that gatekeeper would also result in lost work due to the reluctance of checking in and the "work" it would cause.

From my standpoint, my current company implemented modelcheck about 5 years ago. We have models that date back 20 years, maybe more. When I run modelcheck on my large assembly, it can take an hour or more and generate a lot of errors. Our current method of enforcing Modelcheck "fixes" is upon completion of the drawing, model check is run and a report is generated and attached to our EN. Our release documentation people verify that the report is attached for each drawing to be released and that there are no errors. We also have limited exceptions for no modelcheck requirements on some drawings that would take a long time run modelcheck.

[BrianMartin]

Hi Steve...

I think you've done a good job of explaining what I would call a "Common sense approach" to using ModelCheck. In my experience, this is how most companies operate. Thank you for your feedback! I agree with points you raised. I wish I had more peers with ModelCheck experience to join forces and say "This is wrong" but there are too few of us to make a difference.

Thanks!

Brian

[Chris3]

We implemented model check this year and it is only implemented for the things that actually help users. It only checks:

1. Spelling
2. Checks drawing border bounds
3. Adds missing parameters
4. Removes old parameters / relations
5. Checks for default start part TBDs
6. Blanks layers
7. Provides a way to get X,Y,Z part size

We have large assemblies and having it run on every save or regen would be very problematic.

[BrianMartin]

I agree, Chris... checking for a few really important things would be much better than what we have now. Running the tool constantly has become a real burden with no benefits.

[lhoogeveen]

For our company, it’s worked well to have a group of superusers that represent various internal user groups, locations, etc. that can work with your Creo/Windchill admins to provide feedback before and after implementing changes.

ModelCheck at our organization:

• Errors are only for critical things. Things like required company parameters and values, improper files names, unit systems, allowable drawing formats, etc.. Our admins and superusers agreed on these together. We kept it very minimal and have slowly added over time as needed.
• We run ModelCheck primarily in Save mode but also allow Interactive. We did user testing before turning on ModelCheck and the teams preferred Save mode. Since we mostly focus in critical items, ModelCheck only popups (often with a GUI to help fix things) when something needs to get fixed. Otherwise, users just keep working. We have small to medium size models and haven’t had any major experience with crashing Creo.
• For errors/warnings relating to 3D modeling practices… we have been extremely cautious and careful about turning these on. We have slowly turned a handful of these on to be warnings and occasionally errors after power users representing our various users agreed it was worth adding or asked for them.
• We try to minimize the numbers of warnings – especially in Save mode. Save model popups that aren’t easy to fix drive users crazy.
• It’s possible to configure Save and Interactive modes to require different checks. We set up a couple groups where Interactive mode includes all the Save checks plus some additional modeling best practices that can be run as they get ready to release.
• Old models can be a huge pain for ModelCheck when it comes to modeling errors. This is another reason to be very cautious when considering adding modeling errors/warnings. ModelCheck can try to check when a file was created but we haven’t really pursued this as it makes the logic harder to explain to users.
• We maintain a handful of different ModelCheck groups to allow for different teams or regions to have different ModelCheck setups. For example, we have standard product teams that work on longer development times (months/years) and a group that make custom solutions for customers that need to turn things around in days.
• We used to store our ModelCheck setup files on a network drive but have switched to having the files on user’s computers. Having the files local provide the fastest run times for ModelCheck - typically less than 1 second for our models. Using network drives meant highly varied ModelCheck performance for our different locations. Some locations would be 1-2 seconds and others were 20-30 seconds.
• We have Windchill Gatekeeper turned on to make sure the critical things get fixed.
• Your Windchill admin can leave the ‘ModelCheck Configuration’ setting blank and then ModelCheck doesn’t have to run in a specific mode – this let’s our users use either Save or Interactive. 😊 Interactive can check an entire assembly quickly or fix things automatically (using ModelUpdate) so it can definitely add some flexibility.
• To prevent users from setting their own ModelCheck setup, we set modelcheck_enabled = yes and modelcheck_dir path in the Creo config.sup
• We created custom Windchill workspace views with the MC_ERRORS and MODEL_CHECK columns. This helps users find models with errors or that haven’t been checked recently.
• Waiting until late until the development process to fix errors could create problems. Our preference is to use a minimum level of errors and just fix them as you go. But every company/industry maybe different and should try to find a balance.
• Gatekeeper requires a time interval for when models last had ModelCheck run on them. We started at 3 weeks but have landed on 6 weeks. This lets users have models in their workspace for a while before checking in.

ModelCheck isn’t perfect but it’s helped us enforce some basic company standards. For Windchill relevant things like missing parameters or allowed drawing formats, I wish you could configure Windchill products and libraries to require the parameters or drawing formats there. Currently, I believe you can only set required parameters at the Org or Site level using the Type and Attribute Manager.

[BrianMartin]

Thank you for your detailed response, Luke. I appreciate you providing the benefits of your experience. Personally, I still hate SAVE mode - because when you do have a crash situation, there is no remedy whatsoever. The configuration is set to run off he user's local hard drive but still takes between 5 seconds and 2 or 3 minutes to run with no indication of what's happening or why. If it truly ran in only 1 second, I might be willing to live with the potential crash issue - but I feel very strongly there should be an emergency save button (perhaps a toolkit function) which will bypass MC and allow the the save regardless.

I don't believe checking incessantly for parameters and layers is a good use of time - but I can bend on these things. They're minor. I liken it to checking the tire pressure on your car every time you go for a drive. Heading to the store? Check the tires. Heading home from the same store? Check the tires. Engine on fire? Sorry, we don't check for THAT but the tires are 100% perfect!!

Alone, I do not have a problem with ModelCheck, GateKeeper, or the notion of fixing problems as you go. The problem I have is an extremely restrictive set of checks coupled with no alternatives. ModelCheck must run or else. Gatekeeper allows zero errors. And there are a list of errors as long as your arm. Warnings are everywhere, too. Everything demands to be resolved immediately. It's the lack of balance that troubles me so greatly.

For example... there are extremely valid reasons why someone might suppress a component during the rapid iterations of a design phase. I'm not saying I routinely junk up my model with a bunch of suppressed components - but there are occasionally very reasonable reasons to suppress something in targeted situations. This is completely disallowed. Sure, you can use a simplified representation to turn things off/on but on a massive project with thousands of assemblies, no one knows all those simp reps. So people open the master rep and are then confused by what they see. 

In our case, rules were created by fiat with no plan to slowly and methodically adopt new checks as they became necessary. Your choices are to live with it as-is and smile or be deemed a "ModelCheck denier" who simply wants to pollute the Windchill database with junk. 🙂  I'm being a little funny... but it's sadly not far from the truth.

Thanks again for your guidance! Take care!

[TomU]

---

@BrianMartin wrote:

Thank you for your detailed response, Pearl.

---

😂

Brian, for future reference, those 'rock' names are the user's rank on the community and have nothing to do with their actual names.  I'm pretty sure Luke (@lhoogeveen) doesn't typically get called 'Pearl'. 

[BrianMartin]

Thanks, Tom... that's simply me doing too many things at the same time. 🙂 I knew better.

Oh, and I'm super thrilled that after achieving Diamond status and (for awhile) being the third most proliferate provider of solutions on the old Community Board - that I'm back to being "dirt" level or whatever it's called now. I'm one step up from "manure" level. LOL

At least no one has mistaken my name for a rock - I'm really sorry Luke!! My brain has already left my head for the weekend, I apologize and I have corrected my post. 

Take care!

-Brian

[gbronke]

wow, just wow.

[mkron]

Brian,

I am currently in the process of implementing ModelCheck for one of the engineering groups at my company, working with a few SMEs from that team.

I used to be an engineer on that team but have been a Windchill/Creo Admin for the past four years so I can see both sides.

This team is kind of in the wild wild west. They only use CAD in Windchill and do not have any systematically release process (just set state) so there are no systematic checks on any of their data.

My perspective is

• ModelCheck should be run on Save
  • I believe 75% of the users will probably not run it ever if it is not forced
  • It should have the preference set to run after the Save
  • Only critical errors should be run at Save
• Additional warnings can be implemented when run with Interactive / Regen
• Any parameters or relations needed or not should be added/deleted automatically by ModelCheck

The thought is to have some data consistency before they eventually move to having WTParts and Windchill change process with business rules so they don't get a massive amount of errors when we get to that point. There have also been some quality issues tied to oversight in models that could have been caught with ModelCheck.