Skip to main content
J
JXBWk14-Alexandrite
14-Alexandrite
August 7, 2021
Solved

Dealing with large a .csv file - advice neeeded

  • August 7, 2021
  • 1 reply
  • 3592 views

to all

 

I am trying to compare set of data and I imported it as a csv. file. Essentially a "matrix" of Nrow =300,000 by 60 columns. The data represents combined (aggregated) csv files (300000x4)

Mathcad 15 is struggling and I got an 'out of memory error'

Short of using a different software any advice on manipulating large matrices in mathcad 15?

I am going to do a test using the 4th column of data from each file

 

Regards

 

Best answer by terryhendicott

Hi,

 

Mathcad can make a matrix 300,000 by 60 and fill it with data.  It can save the matrix to text file and read it back.  Patience is required as it is slow.  The data file is just shy of 300 MB

 

One memory trick is to manipulate the data using programming to create any intermediate matrices, as once these programs exit the intermediate variables and matrices are deleted from memory.  So process the data as much as possible in programming steps.

 

If you are looking for software to process really large matrices then the free SciLab is suitable to handle them.  It is neither as easy to use nor does it produce the excellent documentation of MathCad.

 

Cheers

Terry

1 reply

T
21-Topaz II
terryhendicott21-Topaz IIAnswer
21-Topaz II
August 7, 2021

Hi,

 

Mathcad can make a matrix 300,000 by 60 and fill it with data.  It can save the matrix to text file and read it back.  Patience is required as it is slow.  The data file is just shy of 300 MB

 

One memory trick is to manipulate the data using programming to create any intermediate matrices, as once these programs exit the intermediate variables and matrices are deleted from memory.  So process the data as much as possible in programming steps.

 

If you are looking for software to process really large matrices then the free SciLab is suitable to handle them.  It is neither as easy to use nor does it produce the excellent documentation of MathCad.

 

Cheers

Terry

J
JXBWk14-AlexandriteAuthor
14-Alexandrite
August 9, 2021

Spoke to quickly. Just found the time to actually open the kindly provided example and where mathcad 15 is trying to create the matrix a I get 'Not enough memory for this operation'. My machine has 100Gb+ of RAM !

L
LucMeekes23-Emerald IV
23-Emerald IV
August 9, 2021

"My machine has 100Gb+ of RAM"

That doesn't matter. Mathcad 15 is a 32-bit application. It cannot address more than 4 GB... But it also needs memory for other stuff, so your data structure must be (much) less than 4 GB. And it must fit in a contiguous piece of (available) memory, it cannot be scattered around...

 

Considering that Terry got it running, your PC's memory must be overloaded with other data and/or you're not on the latest service release of Mathcad 15 (you should be running M050).

 

Success!

Luc

Terms & ConditionsCookie settingsAccessibility statement