Skip to main content
M
MikeArmstrong1-Visitor
1-Visitor
December 28, 2010
Question

Large Data Handling - File Input

  • December 28, 2010
  • 2 replies
  • 7094 views

Further to a recent post I have decided to create another thread due to my findings.

I tried to insert an Excel file, which had a Matrix of size 36000 x 100 with no success. I then decided to convert the Excel file to a *.txt file and still could not insert the data.

Strangely, if the built in function submatrix is used the data can be displayed. I also tried pre-allocating the Matrix as suggested in the Mathcad quicksheet without any succes.

Can anybody shed some light on this?

I have attached the worksheet and a copy of the original data.

Mike

2 replies

W
wayne12-Amethyst
12-Amethyst
December 28, 2010


Mike,

I was able to read the data file in twice, data and DATA. Tried to add them together, but got a memory error.

data is already a matrix (displayed as a table), so the program to construct a matrix is not usefull.

Did you try closing mathcad, and any other porgrams, to clear the memory, and start fresh? (maybe even reboot).

M
MikeArmstrong1-VisitorAuthor
1-Visitor
December 28, 2010
Did you try closing mathcad, and any other porgrams, to clear the memory, and start fresh? (maybe even reboot).

No because the submatrix function worked, so I thought my bother.

It must depend on the machine one is using and as you said allocated memory.

Can you you try increasing the data size until the computer runs out of memory and see if the submatrix function allows a bigger matrix.

Mike

W
wayne12-Amethyst
12-Amethyst
December 28, 2010

Mike,

We sort of went through this exercise a couple of months ago with Richard.

I am rarely accused of being logical, but, the submatrix operator operates on a matrix, so the matrix "data" must have already been there. For some reason that is beyond me, you apparently were able to display the matrix that way and not by typing data= ?

Anyway, if you use the submatrix to create another matrix, then you will have two copies of "data" in memory, which is not good.

Wayne

T
tietjee15-Moonstone
15-Moonstone
December 28, 2010

I am currently with an array that an Excel spreed sheet 44000 x 15 and 16 worksheets.Make sure the spreedsheet is save as 97-2003 file format. For data inport use "Excel". It will allow selection of the file and worksheet. You will get a warning "only able to preview 19999 row". Ignor the warning, all the rows are imported. It will take awhile (5 minutes)

Cheers

W
wayne12-Amethyst
12-Amethyst
December 28, 2010

Just for discussion:

When working with the matricies, mathcad needs a contiguous block of memory to store the large matrix. Mathcad does not clean up the memeory outside of a program unitl it is closed, but it does for calculations done inside a program, when the program is closed. It gets more and more difficult to find the large blocks of contiguous memeory as you work with the data.

A possible solution:

import the date in smaller sections, probably by column since Mathcad seems to perform operations columnwise better.
Then use a program to perform operations.

maybe, if the data is used in sequence, read in a block, do calculations, read in the next bock, overwriting the provious block, and continuing.

I don't know if importing a text file or a excell data file makes much difference, but it would seem the simpler the data the better, and you can't get much simpler than a dilimited text file, but I might be wrong.

M
MikeArmstrong1-VisitorAuthor
1-Visitor
December 28, 2010
I don't know if importing a text file or a excell data file makes much difference, but it would seem the simpler the data the better, and you can't get much simpler than a dilimited text file, but I might be wrong.

Jean always argued that the text file was the right way to go.

A possible solution:

import the date in smaller sections, probably by column since Mathcad seems to perform operations columnwise better.
Then use a program to perform operations.

Are you suggesting to call the data in column by column? The data provided has 100 columns.

Mike

Terms & ConditionsAccessibility statement