Openning a big file (excel or .txt)

[pjoyeux]

hello,

I need your help,

I made different algorithms to exploit data. This program works with small file (excell export 600ko). Now I'd like to use a bigger data file (arround 32Mo inexcell file or 28Mo in txt file), but I can't import this kind of file. so I solicits your help

thanks for you answer

[Werner_E]

but I can't import this kind of file

Why? Cause of a Mathcad error - which one? An what version of Mathcad are you using?

Do you need all that data at once in your calculations or would it be possible to split the problem into parts and read in just smaller junks of data?

I don't know what limits Mathcad has in terms of matrix size (don't think so), but I guess that available memory used by Mathcad would be the limit. Anybody having more cobcrete information on this?

P.S.: FIrst I thought that the "o" would be a typo, but I guess you call a byte an octet in France, right? Unfamiliar to me but octet is more correct anyway, as a byte could be seven or nine bits, too, in some context.

[pjoyeux]

thank you for your response
I use the version mathcad 15.0 M010. I have no error message (it appears only in red).
my operating system is seven professional 64-bit with 16GB of RAM
The original file has 2000 lines and 1500 columns
I decreased the size of the file and it opens up 7.5Mo, Number column of this file is 388
if I create a file with zeros up to 1500 column (the same 388 colmn and zero until to 1500 column)sometimes has open (to 7.8Mo), so I feel a problem of gesion of memory. you can increase the memory to be mathcad?
I can try to cut out the original file into part but if I could know the limits of what will be better mathcad

best regards

PS MB = Mega Byte in Frenchthank you for your response

[Werner_E]

I use the version mathcad 15.0 M010. I have no error message (it appears only in red).

No black on yellow error message if you click on the red?

The original file has 2000 lines and 1500 columns

That size of that matrix should be no problem at all! I just created two matrices, both with 6000x6000 elements without problems (using M[6000,6000:1, 32 bit, just 1 GB of RAM ).

So I guess its not a problem with too less memory. The memory menagement in Mathcad is somewhat strange and and obscure. You cannot influence the amount of mem Mathcad would use.(which is not strange at all). The program is based on the .NET framework and so the whole memory management and garbage collection could be left to .NET. But it seems that Mathcad uses its own memory management on top of it and this seems not to be top notch and garbage collection won't work always as expected. When playing with the creation of large matrices one often experiences strange errors as of "Not enough memory for this operation" even when the larga data structures are already deleted or even the worksheet containig them is already closed. So only closing MC and restarting the program cures that problem.

But in your case it may another error, but without the sheet (and maybe even with) its hard to say what may cause an error.

[pjoyeux]

hello,

je vous confirme qu'il n'y a pas de message d'erreur,

je vous mets en lien un fichier compressé qui contient un fichier excel, un fichier mathcad qui ouvre le fichier excel et une capture d'ecran montrant le l'erreur du fichier

http://9giga.sfr.fr/n/50-17/share/LNK8407521632c59fdb1/

best regards

[Werner_E]

Nun ja, das Öffnen Ihrer Datei hat meinen schwachen Rechner ziemlich lahm gelegt. Ich musste den Vorgang nach 15 bis 20 Minuten abbrechen, noch bevor ein Fehler aufgetaucht ist. Es war ständig Festplattenaktivität zu beobachten - wohl der ständige Zugriff auf die Auslagerungsdatei.

Ich habe dann ein 2000x1000 Zellen großes Blatt erstellt und dieses wurde in sehr erträglichen 2-3 Minuten geladen.

Sie könnten versuchen, anstelle der Excel-Komponente den Befehl READEXCEL zum Einlesen zu verwenden. Er ist nicht schneller, aber Sie würden vermutlich zumindest eine Fehlermeldung erhalten.

EDIT: An interesting detail: When I tried to read just a small junk of data it takes the same time as reading in the whole sheet.

That means that the splitting of the data can't be done in Mathcad by reading in smaller chunks one after the other, but the data file itself would have to splitted in parts (which is bader, I guess).

So my guess is that Mathcad will first read in the whole Excel sheet in some temporary memory and only afte doing so the data needed is copied to the variable. The whole read-in seems to be accomplished in a VERY inefficient way concerning both memory usage and time needed.

[Werner_E]

Here is a possible solution.

I splitted the Excel sheet into two 2001x750 sheets and read them in one after the other. Works like a charm even on my machine. You may keep the splitted sheets in the same file as different sheets, as you can specify not only the range but also the sheet in READEXCEL (and in the EXCEL component as well).

[LucMeekes]

See http://communities.ptc.com/message/51236#51236

I guess Mathcad 15 has different limitations on array size than good old Mathcad 11.

One improvement for Mathcad 15...

Success!
Luc

[Werner_E]

No, I don't think that the matrix size is the problem - a 2000x1500 matrix should be no problem, not even on my machine ;-).

See my answer to Patrice above.

Mathcad's data read-in seems to be implemented in a very inefficient way. I was able to read-in a 2000x1000 Excel sheet in about 2-3 minutes but had to stop the reading process (killing the Mathcad task in task manager) of Patrice's 2000x1500 sheet after 15-20 Minutes as it blocked my machine completely, swapping data in and out (only 1 GB of RAM) continuously. I couldn't wait until I get the error Patrice was mentioning, but I guess it would be some kind of "out of memory". That memory hunger is by no means justified by that amout of data. I have no idea how this data read could be implemented.

As Patrice had his problems with txt files as well I guess its not a problem of Mathcads Excel interface but will affect all readins.

[tdevaraj]

I know you've been looking at how Mathcad 15 reads large external files.

Have you tried reading this file in Prime 2.0 64 bit?

I observed that Prime 2.0 reads the excel data in one go ( without spiltiing the data into different sheets) after a short delay ( 5 - 10 seconds).

You would be less likely running into memory limitations when processing large arrays in Prime compared with Mathcad 15.0

Making use of the READEXCEL function.

Here is the data

BR,

Thomas

[Werner_E]

You would be less likely running into memory limitations when processing large arrays in Prime compared with Mathcad 15.0

As already written its not the array size which makes the trouble but it seems to be the way file read in is implemented in MC15.

I am glad to hear it works better in Prime.

Think Patrice will have to decide if that advantage of Prime is worth to him the step backwards in useabilty and functionality.