Nested Matricies Editing

[MVP]

Hello,

I have a program I'm trying save a system states (a matrix with constituent masses for a chemical reaction) in a time-sequenced vector. I can access a single value in the nested state matrix, but I'm wondering if there's a way to edit the value without having to overwrite the nested matrix. Can anyone shed some light on this for me? I've uploaded a sample sheet that should give an idea of the error I'm running into (I'm trying to edit the 4th row cell in the nested matrix).

Thanks,

Matt

[AWesterman-disa]

Unfortunately as I only run mathcad 12 I can't see the exact nature of your problem, but best I can offer is attached.

Mathcad allows you to index into the array to read any element that you want, but when you want to modify it thats another problem.

Hopefuly this will give you a few ideas to try, any problems save as version 11 or 12 & i'll see if there's anything else I can add.

Regards

Andy

[MikeArmstrong]

Something like the attached?

Mike

[StuartBruff · Accepted solution]

Matt Paragano wrote:
Hello,
I have a program I'm trying save a system states (a matrix with constituent masses for a chemical reaction) in a time-sequenced vector. I can access a single value in the nested state matrix, but I'm wondering if there's a way to edit the value without having to overwrite the nested matrix. Can anyone shed some light on this for me? I've uploaded a sample sheet that should give an idea of the error I'm running into (I'm trying to edit the 4th row cell in the nested matrix).
Thanks,
Matt

Andy's worksheet gives the normal solution; Mathcad doesn't allow that form of nested assignment at worksheet level (ie, := ), but it will (in some versions) allow it within a program. There is a long-standing request to allow it in an ordinary definition. The attached worksheet just expands a little on Andy's worksheet.

Stuart

[MikeArmstrong]

Weird. Your last function 'changeval' is basically the same function as I proposed and I didn't even see your sheet

Mike

[MVP]

Thanks for the help everyone. It appears this will work backwards to at least MathCAD 11, which is what I need.

Maybe I should start a petition for the normal assignment of values in embedded matricies

[RichardJ]

Another slight variation on the theme.

[wayne]

Anicillary question;

Nested arrays are mainly for holding information, and can be used to return matrices of different size from a program, But what about the memory requirements of nested arrays? Is all the information held in contiguous memory, or are the nested arrays held separatly and just pointed to from the main matrix. I am just gussing, but I suspect that they are not efficient (memory ) ways to store information?

I am thinking that it is attractive to use nested arrays to return results from a program, but for larger matrices, it might be better to just to return a large matrix and parse that, rather than parsing the nested arrays?

[RichardJ]

Is all the information held in contiguous memory, or are the nested arrays held separatly and just pointed to from the main matrix.

That's a good question. Unfortunately, I don't know the answer. Perhaps it's possible to figure it out by creating the largest matrix possibe, and then the largest second matrix possible. Then see if you can nest them into a third matrix.

[RichardJ]

I guess that answers that question.

After assigning M1, M2 is was large as I could make it without running out of memory. There is no problem if I then put bot matrices into a nested array though, so the nested array cannot occupy a single contiguous memory block.

Strangely, even though it can create the nested array OK, if you try and display it you get an out of memory error.

[wayne]

Richard,

I took you lead and did the following:

1) I made 5 matrices, with 5700x5700 for a total of 1.625x10^8 elements.

2) I made 4 matrices, with 6500x6500 for a total of 1.68x10^8 elements and put these into a 5th array, nested.

Each case was in a file by itself and I rebooted in each case to determing the maximum size I could get.

Therefore, the nested arrays must be pointed to and not stored in one big block.

It follows that it doest not matter, as far as memory is concerned, which way you store the data.

I attaced a file with both methods combined, and with the results from the independent files copied so you can look at, if you want.

(I wasn't smart enough to know if that was the conclusion you came to or not)

Actually, after remembering previous posts about memory fragmentation and that matrices need contiguous memory, I think that passing large matrices as nexted arrays is actualy better; if it can be done efficiently.

What say you?

(or anyone else)

[RichardJ]

(I wasn't smart enough to know if that was the conclusion you came to or not)

I guess I should have made the effort to write something! I just updated my post.

I also tried another experiment that I didn't bother to report. Create a matrix, M1, that is as large as possible. I think it was 3500x3500. Then nest M1 into a new matrix, M2, 100,000 times. That works, and my PC does not have 3500 x 3500 x 100,000 x 4 = 4,900,000,000,000 bytes of RAM! So it can only be an array of pointers.

[wayne]

Richard, (and others)
Is this a logical conclusion?

--------------------------

Matrices are stored in conitguous blocks of memory (as opposed to memory with gaps).

A nested array in a matrix acutally contains a pointer to the corresponding matrix. Therefore, a matrix with nested arrays is actually a small matrix of pointers.

In a long file, and especially with other applications running, memory can become fragmented and it may happen that a large matirx cannot be created because a sufficient block of memory cannot be found. Therefore, it can be advnatageous to partation a large matrix and store independeltly or as a matrix of nested arrays.

When passing a large matrix result back from a program, it can be more efficient to partation the matrix and pass back a nested array, thereby using smaller blocks of non-contiguous memeory.

-------------------------

Since I don't know how or what form the nested array is passed back from a program, this may or may not be true?

I think a test would be to make two short programs to pass a matrix and another to pass a nested array, and to place this at the end of a long worksheet and see how many elements can be passed using each method. (re-booting and running each independently, with no other applications running).

I will do this a little later, but would like any thoughts you migth have before I do.

Thanks

Wayne

[RichardJ]

Matrices are stored in conitguous blocks of memory (as opposed to memory with gaps).

A nested array in a matrix acutally contains a pointer to the corresponding matrix. Therefore, a matrix with nested arrays is actually a small matrix of pointers.

Both of these are certainly true.

In a long file, and especially with other applications running, memory can become fragmented and it may happen that a large matirx cannot be created because a sufficient block of memory cannot be found. Therefore, it can be advnatageous to partation a large matrix and store independeltly or as a matrix of nested arrays.

In principle at least, yes. I don't know how much advantage you might actually get in practice though.

When passing a large matrix result back from a program, it can be more efficient to partation the matrix and pass back a nested array, thereby using smaller blocks of non-contiguous memeory.

-------------------------

Since I don't know how or what form the nested array is passed back from a program, this may or may not be true?

A nested array must be passed back as a set of smaller arrays and an array of pointers, just as if you had created it directly in the worksheet. That must be true, because when you exit a program all the arrays created in the program are destroyed and the memory freed.

[wayne]

Richard,

Thanks for your response.

I did do the test I indicated, and it supports the previous conclusions.

At the end of a very long set of calculations (32 compact pages with 4 files containing functions referenced in).

The largest matrix I could return was 1750x1750.

The largerst matrix of nested arrays was 4, each 1750x1750.

So the largest block of memory available allowed for a 1750x1750, but I had at least 4 blocks that size available for the nested array.

Again, thanks for your help.

[RichardJ]

Very interesting result!

I assume you made sure the 4 arrays were different; i..e you are not returning a nested array with 4 "copies" of the same array, because of course that will be one array with 4 identical pointers to it

What happens if you just use some range variables to create some arrays at the end of the same worksheet? Do you see the same thing?

[wayne]

Richard,

Of course just using the same array 4 times would be really stupid, so that is exactly what I did.

But I did catch it before I posted the results.

Surprisingly, to me any way, am not able to duplicate outside of a program.
I can't make 4 matrices, unless they are much smaller( 1000x1000), runs out of stroage space. And even then I can't nest them unless even samller.

I'm going to go have a drink.

[RichardJ]

Of course just using the same array 4 times would be really stupid, so that is exactly what I did.

But I did catch it before I posted the results.

Just checking

Surprisingly, to me any way, am not able to duplicate outside of a program.

I can't make 4 matrices, unless they are much smaller( 1000x1000), runs out of stroage space. And even then I can't nest them unless even samller.

That surprises me too. A lot. I need to think about that.

I'm going to go have a drink.

I've already had one. It obviously hasn't lubricated the brain cells enough though. I guess I'll just have to go and get another

[RichardJ]

See my post here:

http://communities.ptc.com/message/155365#155365

I think I need another drink!

[wayne]

Richard, and others

It seems that the two threads are merging, Nested arrays and the Import Binary file.

I can do much more inside a program than outside, I am wondering if the memory management inside a program is better than outside, Is it possible that inside a program, .NET is used to manage the memeory and ouside it is not, a consequence of the free form?

Another reason that for large projects or standardized calculations, the more work done inside a program the better.

We seem to be burning a lot of time, iterating around to hone in on how Mathcad manages memory,

But,
It would seem to me that its about time for a muckety-muck from Mathcad to shed some light on this?

[MikeArmstrong]

for a muckety-muck from Mathcad

Muckety-muck????

Mike

[wayne]

Mike,

Chief, head cook and bottle washer, big cheese, what ever you like.

We built up a lot of indirect evidence regarding how memeory is managed

But the answer is known and I think it could be cleared up easily by someone from Mathcad.

[RichardJ]

We built up a lot of indirect evidence regarding how memeory is managed

Yes. Badly.

But the answer is known and I think it could be cleared up easily by someone from Mathcad.

Only one of the programmers could answer a question like that. They do not frequent the forums though (or if they do, they are lurkers). I thought about sending an e-mail to one of them, but since the answer would have to be something along the lines of "yes, you are right, it doesn't work very well", I suspect what I would get would be no answer at all.

I'm at a conference next week, but when I get back I might play with programs a bit, and see what I can figure out there..

[wayne]

Thanks Richard.

This may be a stupid question (certainly an ignorant one) but is the memory management different inside a program verses outside?
I know I already querried that, but it may make a difference.

Wayne

[MikeArmstrong]

This may be a stupid question (certainly an ignorant one) but is the memory management different inside a program verses outside?

I know I already querried that, but it may make a difference.

The help file states it is a better stragegy to perform intermediate array operations within a program because the array is only created when the program is executed, once the program is finished any intermediate arrays are cleared from memory.

Each array created at worksheet level takes up memory while the worksheet is open, which means less virtual memory for calculations.

Mike

[wayne]

Thanks Mike,

But I know that, I was referring to some earily posts where I was able to make 4 large matrices inside a program, and return a nested array.

But in the worksheet, I could not make four matrices any where near the size I could inside the program.

So, why could the program find the necessary memory to make 4 large matrices, but the work sheet could not. Note that the program does not clean the memory until it is exited, so up to that point, I would expecte the same availabe memory.

FYI, the test was doen at the end of a very long worksheet, the purpose of which was to fragment the memory. Eash test was performed after closing and restarting Mathcad, and done several times.

Is it because the overhead in the work sheet for a matrix is much higher than the overhead required inside a program? Not just performing operations, but the matrix itself?

If this is the case, then when working with large data sets or with long worksheets, it suggests to me that you should use the worksheet with small data sets to develope what you want, then translate into porgrams before working in the large data sets. I think this is the case anyway, but it puts even more emphasis.

Thanks,

Wayne

[MikeArmstrong]

But I know that, I was referring to some earily posts where I was able to make 4 large matrices inside a program, and return a nested array.

But in the worksheet, I could not make four matrices any where near the size I could inside the program.

So, why could the program find the necessary memory to make 4 large matrices, but the work sheet could not. Note that the program does not clean the memory until it is exited, so up to that point, I would expecte the same availabe memory.

Oh I get you what you where referring too now.

That is a strange issue and would probably need one of the programmers from PTC to shed some light on it.

Is it because the overhead in the work sheet for a matrix is much higher than the overhead required inside a program? Not just performing operations, but the matrix itself?

Possibly, but you would have thought it was the same.

Mike

[LucMeekes]

Richard,

In Mathcad 11 I can increase n1 to 4095 with no pain (other than running out of physical memory, so the system is swapping).

Then I can increase n2 to the same 4095, and still M3 is displayed as a nested matrix. (The system is now using 700+ Mbytes on my 512 Mbytes machine).

Now i can add a M4:stack(M1,M2) and gives me Rows(M4)= 8192. Memory usage is now up to 950MB.

As you can guess, the machine is slow now, but no error message from Mathcad yet. I think that will happen when I try to display M4=

Yep: "Warning: An internal error has occurred. Please svae your worksheet and exit."

I can save it, and then when I close the sheet, I get another popup notifying me that due to errors, windows is closing Mathcad.

Luc