I don't think that attribute matters. I left it with the default value, and the file compiled just fine. The important thing is to make sure each <trans-unit> element has the "restype" attribute set to "message".
One thing that tripped me up initially: I was editing the XLIFF file in Arbortext, and trying to compile it *while it was open in Arbortext*. That was a mistake, because Arbortext puts a file lock on the file while it's open, and then the xmsgfmt.exe compiler can't access the file. Once I closed Arbortext I could use the xmsgfmt.exe utility to compile the XLIFF into an AMO file with no problem.