cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Large Data Handling - File Input

MikeArmstrong
4-Participant

Large Data Handling - File Input

Further to a recent post I have decided to create another thread due to my findings.

I tried to insert an Excel file, which had a Matrix of size 36000 x 100 with no success. I then decided to convert the Excel file to a *.txt file and still could not insert the data.

Strangely, if the built in function submatrix is used the data can be displayed. I also tried pre-allocating the Matrix as suggested in the Mathcad quicksheet without any succes.

Can anybody shed some light on this?

I have attached the worksheet and a copy of the original data.

Mike

12 REPLIES 12


Mike,

I was able to read the data file in twice, data and DATA. Tried to add them together, but got a memory error.

data is already a matrix (displayed as a table), so the program to construct a matrix is not usefull.

Did you try closing mathcad, and any other porgrams, to clear the memory, and start fresh? (maybe even reboot).

MikeArmstrong
4-Participant
(To:wayne)

Did you try closing mathcad, and any other porgrams, to clear the memory, and start fresh? (maybe even reboot).

No because the submatrix function worked, so I thought my bother.

It must depend on the machine one is using and as you said allocated memory.

Can you you try increasing the data size until the computer runs out of memory and see if the submatrix function allows a bigger matrix.

Mike

Mike,

We sort of went through this exercise a couple of months ago with Richard.

I am rarely accused of being logical, but, the submatrix operator operates on a matrix, so the matrix "data" must have already been there. For some reason that is beyond me, you apparently were able to display the matrix that way and not by typing data= ?

Anyway, if you use the submatrix to create another matrix, then you will have two copies of "data" in memory, which is not good.

Wayne

MikeArmstrong
4-Participant
(To:wayne)

Anyway, if you use the submatrix to create another matrix, then you will have two copies of "data" in memory, which is not good.

I thought that to be honest.

Very strange.

Mike

tietjee
14-Alexandrite
(To:MikeArmstrong)

I am currently with an array that an Excel spreed sheet 44000 x 15 and 16 worksheets.Make sure the spreedsheet is save as 97-2003 file format. For data inport use "Excel". It will allow selection of the file and worksheet. You will get a warning "only able to preview 19999 row". Ignor the warning, all the rows are imported. It will take awhile (5 minutes)

Cheers

wayne
1-Newbie
(To:tietjee)

Just for discussion:

When working with the matricies, mathcad needs a contiguous block of memory to store the large matrix. Mathcad does not clean up the memeory outside of a program unitl it is closed, but it does for calculations done inside a program, when the program is closed. It gets more and more difficult to find the large blocks of contiguous memeory as you work with the data.

A possible solution:

import the date in smaller sections, probably by column since Mathcad seems to perform operations columnwise better.
Then use a program to perform operations.

maybe, if the data is used in sequence, read in a block, do calculations, read in the next bock, overwriting the provious block, and continuing.

I don't know if importing a text file or a excell data file makes much difference, but it would seem the simpler the data the better, and you can't get much simpler than a dilimited text file, but I might be wrong.

MikeArmstrong
4-Participant
(To:wayne)

I don't know if importing a text file or a excell data file makes much difference, but it would seem the simpler the data the better, and you can't get much simpler than a dilimited text file, but I might be wrong.

Jean always argued that the text file was the right way to go.

A possible solution:

import the date in smaller sections, probably by column since Mathcad seems to perform operations columnwise better.
Then use a program to perform operations.

Are you suggesting to call the data in column by column? The data provided has 100 columns.

Mike

Mike,

If Jean argued that, then I feel better about saying it.

As for importing, depends on the order of the data. But I was just relating to previous threads (with Richard) that Mathcad operates more efficiently using columns as opposed to rows, or even element by element.

So I am suggesting that if the data allows, import blocks using full columns of data, what ever number of columns you want.

But if the data is sequenced by row, than I would use blocks than consist of complete rows

MikeArmstrong
4-Participant
(To:tietjee)

I was using the data provided in another thread. My machine wouldn't runs the Excel file, but would run the *.txt file with the submatrix function.

Strangle behavior.

Mike

tietjee
14-Alexandrite
(To:MikeArmstrong)

I did try exporting my Excel file to a .cvs to make it easier for MathCAD to handle the data. Excel only exported half the data. I went back to import the spreadsheet/worksheet. I does take along time and I am havinmg memory issues. Many reboots.

Cheers

MikeArmstrong
4-Participant
(To:tietjee)

I did try exporting my Excel file to a .cvs to make it easier for MathCAD to handle the data.

What makes you think a *.cvs file would make it easier for Mathcad to read?

I went back to import the spreadsheet/worksheet. I does take along time and I am havinmg memory issues. Many reboots.

Have you tried using the submatrix function and selecting the data that way?

Mike

tietjee
14-Alexandrite
(To:MikeArmstrong)

I need all the data in the excel spreed sheet. Once it has been import, I use the submatrix, stack, and augment commands to manipute the data. Acutally, my project has to excel spreadsheet per workbook and 8 work books. It has been a job, but I am almost all done. End of the year.

Happy New Year.

Announcements
Check out the latest
Mathcad Tip
"PTC Mathcad 15 / Prime 1-6 Update."