Maximum array size

[Robert-B]

It’s been a while since anyone has asked, and the answers were a bit generic. In principle, yes it shouldn’t be limited by 32-bit, or perhaps limited to memory heap...

However, I have found a much smaller limit. I read in scope data, 20,000 points of 8-bit data, padded to 64-bits most likely, so is at most 160 kB. Doesn’t sound too daunting. In fact I can read it in, several columns in fact, manipulate it, plot it all, no problem.

However I cannot evaluate it as an array. Wasn’t really trying. But kept getting errors when I tried including this worksheet into a report style of worksheet... 20,000 points barfs on the include. Then I notice that I can’t even display the array as a table. Really? I have to find a file transfer workaround? It’s a lot of data files... or try to embed EVERYTHING within the report??

[Werner_E]

You should post the worksheet in question and explain in more detail, which problem you experience, what you mean by "However I cannot evaluate it as an array."

Its hard to impossible to make an educated guess without seeing what exactly you are doing.

[LucMeekes]

What version of Mathcad are you using?

The maximum array (or matrix) size in Mathcad 11 is 16 777 216 elements (not surprisingly: 4096^2). You can have more columns, but then the number of rows must come down, and vice versa. Another limit is on the maximum index value, which is set to around 16 000 000 if I remember well.

However if you write your own routine (UserEFI .dll) you may find that you run into problems with (sometimes much!) smaller arrays. My guess is: that is due to (errors in) garbage collection. To circumvent those problems it's wise to write out your matrix or array as a PRN file, and read it back in. (At least, that works for me).

Then you'll find that the maximum array size you can WRITEPRN out is 8 000 000 (sort of exactly). Mathcad just writes the last full row it can fit within the 8 milljon elements before it stops. So you're not stuck with half a row...

Nested matrices are a special breed. I was able to create an array containing 8 matrices of 4096^2 elements each...

Other limits apply when you start manipulating with matrices. If I'm right, mathcad wants to keep its elements in contiguous memory space, so when it cannot adjacently allocate the space needed to increase the size of an element, it 'll report an error.

I think (but just that) that this applies equally to Mathcad 15 (also a 32-bit application).

I haven't tried any of this in Prime, for obvious reasons.

Oh, and I haven't tried matrices of complex numbers either.

Hope this helps.

Success!

Luc

P.S. attach your worksheet. You may have to zip it first.

A serious bug in this forum often prevents from attaching a mathcad file. In Windows Explorer click with the right-mouse button on the mathcad file and select 'Send to' -> 'Compressed (zipped) folder'. The resulting .zip file can be attached.

[Robert-B]

Just to answer the questions...  

This is Prime 4.0, and I've not seen such problems before in v.15 or prior.

By displaying the array, I mean toggling off the button Collapse Nested Arrays.  When trying, it threw an exception and forced me to close MathCAD.  This repeated over and over again until I deleted and recreated the array region, suggesting that region was corrupted.  

This was a simple READCSV file command, but doubt that has any thing to do with it once read.  I restarted, opened a new file and did some testing, and found no serious limit to the size of the or displaying the array.  However, I still had problems reading the array from within the parent worksheet.  Somehow, MathCAD appeared to corrupt the child worksheet, and each such wierd instance seemed to be associated with the include function in the parent worksheet.

After a few hours sleuthing the problem, I gave up on using the include, writing the results off to an output file, and reading back in where I needed it.  I noticed a few issues using include, some workable, but others still a mystery:

1) A completely functioning standalone child or grandchild worksheet has to be careful with relative paths when being called by a parent.  The parent will execute the statements from the perspective of its own path.  (user beware?)  Static paths aren't the solution, and neither is putting them within a common directory.  However, even when the directory paths are deconflicted, there are still other problems.  *Some path functions are needed in MathCAD, defined dynamically, not statically by the user*

2) When including multiple child worksheets into a parent, the children could have problems with array variables with a common name. You would think that the latter would overwrite the former, which I didn't care about, so long as the aforementioned array output (uniquely named) could be transferred... But there still seemed to be problems indexing the arrays that already existed from a previous included worksheets.  *This is a serious problem with using includes*.  I was trying to use multiple copies of a worksheet acting on different data sets, but they seem to stumble on each other despite being sequential.

3) The parent worksheet continues trying to read the children (perhaps because of the error?).  Perhaps a child worksheet could get corrupted if the parent kept trying to read it while the child was being "fixed", corrupting the aforementioned array region. This corruption repeated, in exactly the same region (large array calculation), so maybe some day I will see it again...

Sorry, I can't include my worksheets and data.I've tried to repeat with fake test data and routines, but the simple things appear to check out ok.  The timely solution is to avoid includes if they cause any problems for you.

[LucMeekes]

Maybe this helps to understand:

When you include a (child) worksheet inside a (parent) worksheet, what effectively happens is that the contents of the child worksheet (all its statements, definitions and calculations) is effectively inserted at the position in the parent sheet where the 'include' statement is. You do NOT get a structure where definitions in the child sheet remain local to that child sheet.

Suppose you define a variable var:=1 in a parent sheet and var:=2 in a child sheet. Now you include the child sheet in the parent sheet to the right of, or below the var:=1 definition; then your var:=1 is overwritten with var:=2 by the child sheet. So to the right of and below the inclusion, var's value will be 2...until redefined.

If you use global definitions of the same variable (name) in both parent and child sheets, only trial and error will reveal what the result is.

Oh, and bear in mind: Mathcad Prime isn't perfect. Neither is Mathcad 15 or below, but the number of reports of Prime damaging sheet files is much larger than that of Mathcad....

Success!
Luc

[Robert-B]

Yes, knew that, thought I knew that, and finally changed all globals to local assignments.  But still had problems.  Some of the problems were indeed global statements, and those will obviously clash, even within a single worksheet.  What I've learned from this endeavor is that anything which might generate an error/warning on your single worksheet (and likely ignored, since so many of them are bordered in red) needs to be correct if you try inserting into another worksheet.

Still it doesn't solve the corruption issue with my array (and I suppose this subject title is inappropriate now).  I've witnessed it again, right this moment.  At first the include resolves as correct, and then after some time working on the parent, the include is bright red with an error, although I haven't touched the child at all.  I've been missing this for a while because the worksheet caching toggle keeps defaulting back to selected despite me unselecting it.  Everything looks fine because the variables are cached, until I try to assign them to another variable in the parent.  So I notice every variable is this way, and have to force re-reading the inserted child.  This is when (sometimes) the errors start persisting...

When I open the child, I find the corrupted region(s).  See the screen shot, where the inline cursor is on the number 3.5.  The whole bottom half of the page should display two arrays, and you see just the top element and some of the next.  If I hover on the right bracket of the array (to rescroll the displayed indexes) then some of the array region appears, as if it's there, but not being displayed.  There are actually several of these corrupted regions, not on every page, but quite a few.  Fortunately I have backups, but I have saved the child worksheets a few times with this weird corruption present, and it remains later on, saved int he file.  But if I close them without saving and reopen them without the parent, they seem ok.

So, the problem seems to originate from opening the referenced child for editing while the parent is also open.  Chicken and egg issue.

[LucMeekes]

"So, the problem seems to originate from opening the referenced child for editing while the parent is also open.  Chicken and egg issue."

So what you're saying is that when you have a parent sheet open and the child sheet as well, you see effect of calculation of the parent sheet in the open child sheet....I'm a fraid it's a nasty and (by the programmers) unforeseen side effect of saving calculation results in the sheet file.

I think you may have found a serious bug. Prime should NOT attempt to change a child sheet when it is calculating/evaluating the parent sheet. When you have the parent sheet open, it should only READ the child to get its contents and include that at the proper location in the parent sheet.

My guess is that Prime tries to write the calculation results that 'belong' to the child sheet into the child sheet. That should only occur when the child sheet is opened 'stand alone' as a top sheet AND only at the time that the sheet is actually being saved.

Luc

[tslewis]

I use READCSV when readign huge arrays as it is teh most efficient way of reading data I have spent hours exploring best way of doing this. READCSV is the fastest and less proen crashing method.

When you use mathcad built-in functiosn for file input output such as

READEXCEL READTEXT READCSV

if you are including functions and statements that use the above in the reference sheet, unless you put an absolute path for the file when passing argument from the child sheet, the fucntion used in the refrenced parent file will use the CWD variable paramter of the child sheet.

You probably get a green warning flag about the included sheet CWD variable  using teh current worksheet CWD  directory