Programme Progress

[GeoffD]

Has anyone got a technique for monitoring the progress of a programme?

If you are running a long iterative programme involving many loops (OK Prime is not the best for this I know), but still, it would be great if it were possible to trigger any kind of output to indicate progress.

Thanks!

[Werner_E · Accepted solution]

In Mathcad you could use the debug mode but thats not available in Prime.

So I guess all you could do is to write info data to a file from time to time (every 10000 iterations or the like) and inspect that file outside of Prime.

Of course output to a file slows down execution significantly so you should use it sparely.

[GeoffD]

Alas, I cannot find a way of writing to an external file from within a programme - e.g. trying to use the WRITECSV function to write to an external ,CSV creates an error that stops the programme.

[Werner_E]

Without seeing what you did its not possible to track down the cause of the error you encounter.

Here is a demo program which creates a csv file from time to time.

Drawback is, that each call of WRITECSV overwrites the previous files.

If this is not desired, you may read in the csv file, add the current data and write all data again in the file. Of course the file reading would once again need some time and slows down execution a little bit.

Other options are

• Use WRITEPRN for writing header data and then APPENDPRN. The format of the file may be not what you desire, though.
• Add a counter, create a new filename everytime and create multiple files
• Use WRITEEXCEL because here you can save the data to a cell position of your choice and so you can use a counter to advance the row everytime you writ to the file. But when you open the file in Excel while your Prime program is still working, Prime will fail and throw an error.

[GeoffD]

Many thanks for this - I have things working now - it's the first time I have played with the FILE statements!

A thing worth noting:-  if the file is being inspected at the same time WRITECSV is being executed an error condition is created stopping the programme i.e. it has to be re-run.

I am not sure how Prime handles file access - but if instead of opening the file you run a READCSV statement within the worksheet every time you want an update (I guess with multi-threading activated) then you get the latest status info without crashing the programme. Another approach is to copy the file and open just the copy - but I guess there still is a small period of possible colliding access - not sure.

Many thanks for your prompt and insightful help.

PS I guess this file access capability can enable the near real-time animation of graphical data being generated within the programme by manually re-triggering a READCSV statement to feed a plotting routine? This would be very useful when dealing with problems with uncertain convergence . 

[Werner_E]

Oh yes, the problem of the Prime function failing if the file is opened externally when Prime want to write is not limited to Excel files (where I mentioned that problem) but affects all file types. I haven't tried but it may be possible to deal with that problem by wrapping the WRITECSV command in a "try ... on error" and skip writing if the command fails.

I am used to real Mathcad 15 and seldom use Prime for anything other than when answering in this forum. So I am not used to see it as a multi-thread able program. So I never would have thought of using READCSV in a separate region to inspect the written data while the generating program still is working. Doing so would not have been possible in good old Mathcad.

With this approach using WRITEPRN first  and then in the loop APPENDPRN might have the advantage that you don't need to read in the data every time before you write it.
Reading the written data and showing them in a plot while the generating program still is working might be possible in Prime and could be very handy, indeed.

P.S.: I remember a few older threads here in the forum reporting of Prime crashes and the usual advice was to turn multithreading off. But I am not sure if this still applies to the current versions of Prime.

[GeoffD]

The multi-threading stability issue is much better now (P8). I leave it on all the time and find it useful to work on other aspects whilst waiting for the programme(s) to finish running. Thanks !

[GeoffD]

Following up on the possibility to use READCSV and  WRITECSV to monitor the internal working of a long-running programme (in the absence of a P8 debugging tool) I have been able to get data back into the worksheet and plot the progress of parameters internal to the programme whilst it is running e.g. to immediately see if there is a convergence problem etc.

The following excerpt shows the two $Bump statements  (defined in the attached P8 file) used to monitor progress and push programme progress data to the status.csv file. The  $Splat statement below the programme extracts the status.csv data back into the worksheet for plotting. The plot is triggered by changing the Np parameter (which sets the limit on the amount of data extracted) - the multi-threading action picks up on this change and (re )runs $Splat to feed Plot with the latest progress in (near) real-time.

[GeoffD]

As a further refinement to seeing what is going on within a programme - without waiting for it to finish - I have added a $Stop statement.

If you interrupt a programme you lose all its data - which probably didn't matter as you couldn't see what it was anyway until it was completed. But with $Bump and $Splat you can now see what is happening - so it's useful to be able to stop the programme without losing all the data accumulated up to that point. The worksheet excerpt is shown below:

The $Bump, $Splat and $Stop functions are defined above the programme (the P8 files are attached). $Bump calls are made from within the programme and push status info to the external status.csv file.

$Splat is used below the programme to view the current status info in the status.csv file - this is triggered by you manually changing Np, the length of status info you wish to see. The multi-threading (if on) detects this change and re-runs $Splat - even though the programme is still running. If you open the .cvs file manually and the worksheet tries to access it the programme craters but this does not happen when using $Splat within a worksheet (well it hasn't happened yet !)- having the data within the worksheet also means you can plot it (see the Plot graph) or do additional processing of it.   

The $Stop function contaminates the status.csv file with a nonsense entry {"stop 10^30 10^30] - the next time $Bump interrogates the file it chokes on this entry and returns $call=10^30 which the programme trips over forcing it to stop the programme and return the current data set matloop- so no data is lost (provided it's specified in the return statement). The spike in the Plot is $Splat reading the nonsense entry. 

Although my programming is fairly clunky, it shows that not only is it possible to interrogate how things are going within a running programme - but also the opportunity to feed additional data back into the programme whilst it is running - to alter its direction of travel e.g. to change gains, convergence factors or data sets manually without stopping things. 

I am not sure how P8 controls external file access - a clash between accesses craters the programme - but it may allow two (or more) concurrently running P8 programmes to interact .....

[GeoffD]

with the correct .mcdx file this time...