Dealing with large a .csv file - advice neeeded

[JXBWk]

to all

I am trying to compare set of data and I imported it as a csv. file. Essentially a "matrix" of Nrow =300,000 by 60 columns. The data represents combined (aggregated) csv files (300000x4)

Mathcad 15 is struggling and I got an 'out of memory error'

Short of using a different software any advice on manipulating large matrices in mathcad 15?

I am going to do a test using the 4th column of data from each file

Regards

[terryhendicott · Accepted solution]

Hi,

Mathcad can make a matrix 300,000 by 60 and fill it with data.  It can save the matrix to text file and read it back.  Patience is required as it is slow.  The data file is just shy of 300 MB

One memory trick is to manipulate the data using programming to create any intermediate matrices, as once these programs exit the intermediate variables and matrices are deleted from memory.  So process the data as much as possible in programming steps.

If you are looking for software to process really large matrices then the free SciLab is suitable to handle them.  It is neither as easy to use nor does it produce the excellent documentation of MathCad.

Cheers

Terry

[JXBWk]

Spoke to quickly. Just found the time to actually open the kindly provided example and where mathcad 15 is trying to create the matrix a I get 'Not enough memory for this operation'. My machine has 100Gb+ of RAM !

[LucMeekes]

"My machine has 100Gb+ of RAM"

That doesn't matter. Mathcad 15 is a 32-bit application. It cannot address more than 4 GB... But it also needs memory for other stuff, so your data structure must be (much) less than 4 GB. And it must fit in a contiguous piece of (available) memory, it cannot be scattered around...

Considering that Terry got it running, your PC's memory must be overloaded with other data and/or you're not on the latest service release of Mathcad 15 (you should be running M050).

Success!

Luc

[Werner_E]

---

@JXBWk wrote:

Spoke to quickly. Just found the time to actually open the kindly provided example and where mathcad 15 is trying to create the matrix a I get 'Not enough memory for this operation'.

---

You mean you get the error when Mathcad executes this region

in Terrys sheet without having modified his sheet?

Did you close Mathcad completely before opening the sheet, no other worksheets opened?

Which release of Mathcad 15 are you using (current maintenance release is M050, dating back to November 2017).

There was an issue with handling matrices with 256 or more elements in M040 which was fixed in M045

M020 fixed a memory issue that occur while performing multiple recalculations of a program that reads or writes data to an external data file

M020 also fixed an "Out of Memory" error which occurs at some symbolic operations, but this seems not apply here as no symbolic operations are in use

BTW, I opened Terrys sheet and it executes without problems (and yes, its slow).

[JXBWk]

Thanks to all. I closed all the othe rapp running, mathcad and re-started and indeed mathcad created the "large" matrix. A few seconds of work.

I then had a go a my real input matrix and it worked ! Must have been of those days!

[Werner_E]

Mathcads garbage collection is below par, so after you experience a memory error chances are that you will see the same error with rather simple expressions, too. So the cure often is to close MC completely and restart.