cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Showing results for 
Search instead for 
Did you mean: 

Community Tip - Need to share some code when posting a question or reply? Make sure to use the "Insert code sample" menu option. Learn more! X

How to Read a Binary COMTRADE Data File

ppal
17-Peridot

How to Read a Binary COMTRADE Data File

Does anyone know how I can read a binary comtrade data file.  In this case "File.dat"

What is comtrade? See https://en.wikipedia.org/wiki/Comtrade.

I am interested in displaying the File.dat The others are easy to open in mathcad.

Once I can view it then I can do further processing to it in mathcad

Thanks.

ACCEPTED SOLUTION

Accepted Solutions
LucMeekes
23-Emerald III
(To:LucMeekes)

The data file information is specified in IEEE standard C37.111, which has become an IEC standard in 2001.

See here:https://webstore.iec.ch/preview/info_iec60255-24%7Bed2.0%7Db.pdf

Success!

Luc

View solution in original post

67 REPLIES 67
LucMeekes
23-Emerald III
(To:ppal)

The .hdr (header) file is an XML formatted file.

The .dat (data) file appears to be a binary data file.

The .cfg (configuration) file is a comma separated values (.csv) file.

While Mathcad can 'read' all these files (the first two with the READBIN function, the last one with READPRN), it cannot interpret them.

You'll have to know the format and meanings of fields, then you can write Mathcad programs to actually interpret the files.

Alternatively, if you have c-code that can interpret these files, you could wrap that into a User-DLL that adds the specific functions to Mathcad.

Success!

Luc

LucMeekes
23-Emerald III
(To:LucMeekes)

The data file information is specified in IEEE standard C37.111, which has become an IEC standard in 2001.

See here:https://webstore.iec.ch/preview/info_iec60255-24%7Bed2.0%7Db.pdf

Success!

Luc

LucMeekes
23-Emerald III
(To:ppal)

It's done.

Results:

Let me know if you like it.

Luc

ppal
17-Peridot
(To:LucMeekes)

Looks great. But where do you actually give the full path to the files. When I try it sees the "." cfg then does a subscript. Could you give a screen capture of where you actually point the mathcad to the actual files. Thanks for  your effort. You have put a lot of work into it. Genius!

ppal
17-Peridot
(To:ppal)

I get this error:

File Error.png

LucMeekes
23-Emerald III
(To:ppal)

You have an exclamation mark '!' in your path. That (apparently) is an illegal character.

Instead of using CWD, define the you path string explicitly, like: 

            file:= "D:\....\H035"

Success!

Luc

LucMeekes
23-Emerald III
(To:LucMeekes)

Ah, better still.... assuming you have the mathcad file(s) in the same directory as the .cfg and .dat files, just define:

       file:="H035"

That should do it.

Success!

Luc

P.S. I see that you're using Prime (and probably used Mathcad 15 to open my source file).

Does the utility function in Mathcad 15?

ppal
17-Peridot
(To:LucMeekes)

Hi The conversion to Prime 4 looks fine. The "!" Is a sub folder. But it also failed to open in the d: root drive.  I'll give it another go tomorrow. I really appreciate your work on this. When I get the prime 4 working I'll post it here.

LucMeekes
23-Emerald III
(To:ppal)

Even if you don't get it to work, I may be able to help to find the problem.

Though I cannot execute programs in Prime 4, I can read the file and inspect it.

Success!
Luc

ppal
17-Peridot
(To:LucMeekes)

Hi

Errors I got in Mathcad 15 and prime 4.0.

Hope you can shed some light. Much appreciated.

Werner_E
25-Diamond I
(To:ppal)

I can confirm the error in Mathcad 14, M045.

Its the vec2str(W) statement in the READCSVI routine which throws the error because it obviously can't convert a vector containing a zero into a string.

@Luc

Haven't digged into the logic of your program and can't tell why you set W to a 1x1 matrix containing a zero from time to time.

How does vec2str handles a zero byte in MC11? Maybe as a workaround we could write a new vec2str in MC11 which simulates that behaviour.^

!!! EDIT:

Solution found!

Simply replace the two occurrences of W <-- (0) by W <-- 0 and it seems to work OK. At least it works in MC15

I can't give it a try with P4, but I guess the problem there is the very same.

Below you see the difference between vec2str(0) and vec2str of a 1x1 matrix containing 0. The first returns an empty string (which is needed) but the second throws an error. Guess this is different in MC11

ppal
17-Peridot
(To:Werner_E)

DAT ERROR.pngHi The Cfg now works but Dat file says : "can't open the data file"

Werner_E
25-Diamond I
(To:ppal)

I can't replicate that error using Mathcad 15) - just changed the two program lines mentioned and it works.

Are you sure you have both files, "file.cfg" AND "file.dat" in the same directory as your Mathcad file? "file.hdr" seems not to be used by Luc's sheet.

LucMeekes
23-Emerald III
(To:Werner_E)

The .hdr file is not part of the 'critical' comtrade data, but the .cfg file is needed in order to be able to interpret the .dat file.

LucMeekes
23-Emerald III
(To:Werner_E)

Werner,

I remembered once having had a bad experience with supplying an integer to the vec2str() function.

(The error message says that the argument must be a vector)

But I've checked vec2str() can live with a zero, also in Mathcad 11:

So I could change single element vector containing a zero into a simple zero.

A message to the OP:

In Prime you might not need the READCSV function, because Prime has READCSV, which does the same in this application.

Luc

LucMeekes
23-Emerald III
(To:LucMeekes)

Yes, it works,

here is the adapted version that should also work in versions of Mathcad above 11.

Luc

ppal
17-Peridot
(To:LucMeekes)

Yes! Mathcad 15 works but Prime 4.0 has errors:

Prime 4.0 Comtrade error.png

LucMeekes
23-Emerald III
(To:ppal)

I also corrected an error, which means that the data file is now read including the very last sample

(I had to omit the last sample because the function was apparently reading past the end of the data file).

Attached are corrected Mathcad 11 and Prime 4 files.

I don't think that the corrected Prime 4 file will solve the problem you show above.

Luc

LucMeekes
23-Emerald III
(To:ppal)

Does prime have a trace function, and can you use that to pinpoint the location of the error inside the RaedComtradeDat function more precisely?

ppal
17-Peridot
(To:LucMeekes)

I don't think so. I could not see  any option.

Werner_E
25-Diamond I
(To:ppal)

There IS a trace option in Prime. Not so convenient as in Mathcad, but its there -> "Calculation ribbon" -> "Error Tracing"

Werner_E
25-Diamond I
(To:LucMeekes)

Ah, I see from your screenshot that this

is the difference to later Mathcad version which throw an error (maybe we could call that a bug).

Still not sure why the OP now has problems reading the *.dat file.

Only explanation I can thinks of is, that he forgot to copy it to the same locations as the cfg and MC file. Also possible that the file is open and in use by another program - not sure if MC would choke on hat.

LucMeekes
23-Emerald III
(To:Werner_E)

Prime 4 does not choke on opening (built in READCSV) the .cfg file while that is open in MsExcel.

And it's got no trouble opening first part or entire .dat file while that is also open in MsWord.

Luc

ppal
17-Peridot
(To:Werner_E)

Working fine in Mathcad 15. Problems when i converted to Prime 4.0

LucMeekes
23-Emerald III
(To:ppal)

I'm afraid I can't solve the problem that Prime has with my (translated) function. I've started my 30 day evaluation period in order to be able to run programs in Prime 4. It appears that Prime has a big problem with one of the statements in the READCSV program.

I've checked all tricks that the statement uses, and verified that they work in (other) Prime test programs that I made. So that is not the problem. The values of r, c and W just before the statement are as expected, and Prime does allow such assignments to R, effectively creating the matrix. So I cannot see why this should be a problem.

I think that simply Prime 4 is the problem: "system error"...

Anyway, it works in Mathcad 15, I suggest you use that to analyse the Comtrade data.

Success!
Luc

ppal
17-Peridot
(To:LucMeekes)

Thank you. I will take it up with the vendor. Time to get some value out of the maintenance and support fees.

Werner_E
25-Diamond I
(To:LucMeekes)

I agree that the problem seems to be buggy Prime.

Same error here with Prime 3.0.

Crazy enough that the following workaround(?) solves the problem:

At least it works in Prime 3.0:

LucMeekes
23-Emerald III
(To:Werner_E)

Certainly odd, because Prime4 says:

Luc

Werner_E
25-Diamond I
(To:LucMeekes)

Yes, same error as in Mathcad 15 as I noted above.

OK, maybe its not to be seen as bug as a string containing the NUL character is not the same as a string containing no character at all, but the error message is misleading and it would be quite convenient if vec2str would accept a scalar as argument.

But this vec2str behavior does not explain why my workaround with the if statement is necessary. It should work as it was written by you because vec2str(0) returns the empty string anyway. First idea was that it isn't en empty string but rather a string containing an invisible NUL character, but Mathcad told me otherwise

So I believe that the strange error message we get in Prime is the result of a a Prime bug.

Anyway, our "Mathcad User" should now be able to use your program in Prime, too. He seems to have another Problem (in MC15) when using a different input file.

Announcements

Top Tags