Skip to main content
jhardy11-Visitor
1-Visitor
November 8, 2010
Solved

Import binary file to Mathcad 14 - size limit?

  • November 8, 2010
  • 11 replies
  • 11742 views
I have created a Mathcad 14 worksheet which reads and processes a large binary data file - specifically, a tile of NASA SRTM terrain elevation data. The worksheet works fine using the generally available 3 arc-second resolution files, which are 2.8 MB each, with over 1,000,000 records in each file. (Each file is a vector containing one integer entry for the elevation data of each point on a 1,201 x 1,201 grid, for 1,442,401 integers in total.)

I would like to use the same worksheet to process some 1 arc-second data files, which have approximately 9 times the number of records (3,601 x 3,601 = 12,9607,201 grid points), and 9 times the file size (approx. 25 MB each).

When I try to point the "Data Import Wizard" at one of the larger files, it first seems to work fine, but then it loses the data from the imported data file, and the worksheet stops working, as it has no data vector in the main data array. (The array shows only a single entry with a zero value, instead of the expected vector of data entries - see attached screen grab.)

I suspect this is an issue with the file size exceeding some limit of Mathcad for handling large binary files, and / or exceeding memory limits.

Does anyone know what are the effective limits for handling large binary files in Mathcad 14.0 M020? I couldn't find anything in the documentation.

Thanks for any help!
(Note: I have also posted this question on the Eng-Tips forum http://www.eng-tips.com/viewthread.cfm?qid=285290&page=1 - please don't feel obliged to respond to both! )
Best answer by RichardJ

In Mathcad 11 the maximum array index is 16,000,000-1 exactly. That's a program(med) limitation.

I don't think so. The MC 11 help says:

The effective array size limit depends on the memory available on your system. Approximately 8 bytes of memory are needed per array element. For most systems, the array limit is at least 1 million elements. In no system is it higher than 8 million elements.

Except that is definitely not correct, because I just created an array bigger than 8 million elements. Maybe that piece of the help is a left over from an earlier version. I thought the size was limited only by memory (which is certainly the case in MC 14 and MC15).

11 replies

RichardJ19-Tanzanite
19-Tanzanite
November 9, 2010

With 13 million points it's likely you have simply run out of memory.

An array requires a contiguous block of memory, so it's possible that if you reboot your computer, then start only Mathcad, you may be able to read in the file.

LucMeekes23-Emerald IV
23-Emerald IV
November 9, 2010

You may have hit a (or the) maximum allowed array or matrix size.

In Mathcad 11 the maximum array index is 16,000,000-1 exactly. That's a program(med) limitation. Mathcad 11 tells you when an array is too large. Don't know for Mcad 14.

Regards,

Luc

RichardJ19-TanzaniteAnswer
19-Tanzanite
November 9, 2010

In Mathcad 11 the maximum array index is 16,000,000-1 exactly. That's a program(med) limitation.

I don't think so. The MC 11 help says:

The effective array size limit depends on the memory available on your system. Approximately 8 bytes of memory are needed per array element. For most systems, the array limit is at least 1 million elements. In no system is it higher than 8 million elements.

Except that is definitely not correct, because I just created an array bigger than 8 million elements. Maybe that piece of the help is a left over from an earlier version. I thought the size was limited only by memory (which is certainly the case in MC 14 and MC15).

MikeArmstrong1-Visitor
1-Visitor
November 9, 2010

I just tried to create an array with an index of 16,000,000 and it failed, then again my work laptop is painfully slow.

Mike

ptc-13715561-Visitor
1-Visitor
November 14, 2010

Hi

This issue of running out of memory has been raised a number of times over the years in the collaboratory. Often blame is fobbed upon NET but this is not right simply because many other programmes run in Windows without memory issues like in Mathcad. I think we need to get right to the issue here and not skirt around it all the time. The limitations should be documented in its manuals etc as users are getting a false impression of what can be done.

I have used Mathcad v11, v13, v14 and v15 and the same issue exists across all these versions. Mathcad falls over when you use large arrays and when you start processing them - after all just creating a large array and doing nothing with it is a totally pointless exercise and proves nothing. Mathcad must have an internal structural problem with regard to memory, particularly stressing its internal structure when processing large arrays. It really needs to fix the issue once and for all. As time goes on, we do more computation and process ever larger sources of data: this is a non-reversible trend. Mathcad or any other software provider must keep up with these real life demands. Mathcad has hit the limit so far as I am concerned and it seems for a number of other users as well. For PTC / Mathcad trading as an provider of high level technical software, this is a poor situation and undermines their credibilty.

Do not get me wrong, I really like the Mathcad programme.

However, Mathcad is getting very frustrating for me to use and it is even more frustrating to see Mathcad / PTC do so little software development over the last few years to push Mathcad further along the envelope and make it generally more powerful. Forum users has raised many useful suggestions on advancing Mathcad, many of which should be fairly straightforward to implement. I do not know what is in Prime (I am concerned by some of the comments in this forum that Prime has some limitations compared to even Mathcad 15), but I think PTC really need to pull their finger out and get some serious development work done. I suggest that PTC should make some announcements to its users about what it is doing. Has PTC properly resourced its programming team on software development? - I suspect not given the limited development output over the last few years. Perhaps there are also issues on how PTC is running on a business management aspect? It seems to me Mathcad may be good for a certain level of work, but it will get left behind for larger scale work.

I have gone through my learning curve with Mathcad and written a number of programmes over the last 7 years or so for my work. I am loath to dump all that and re-learn something else like Maple or Mathematica. So I am hoping that Mathcad does progress and go to another level. As soon as possible, please!!!

Regards

David

Terms & ConditionsAccessibility statement