If you are looking specifically at primary content documents (documents for which the user is uploading a Word document, PDF, etc.) you might not have to add any kind of attribute restraint.
If the Attachments > Document Name preference is set to Yes, then the document name is automatically generated from the file name. Windows explorer does not allow the following characters in file or folder names: \ / : * ? " < > | and therefore Windows would do the job of restricting those characters for you.
If you are creating documents from templates, you can set the Document > Document from Template Filename preference to (or similar values that do not allow user input).
You can also enable the document Title attribute if you want users to still be able to manually specify some kind of document identifier string (but, depending on your use case, you might need to use attribute constraints on the title characters). The title is also enabled using a preference. See the topic "Content Management Preferences" in your Windchill Help Center (or the help center on ptc.com).
There are also 3 properties in wt.properties that affect characters in object names, but those only apply to HTML markup characters (e.g. transforming "<" to "<") -- com.ptc.netmarkets.outputFilter, com.ptc.netmarkets.inputFilter, and com.ptc.netmarkets.filterAllHTML.
However -- to restrict characters for the Title attribute or user-specified document names, and for "no content" documents, URL document labels, and "external storage" document types - you'd need to use the attribute restraints Patrick mentions.
