We do it a little differently; due to our process, the DTDs aren't in \custom, but we update the catalog path to point to the directory that holds our catalog, and everything works.
If you put everything in your \custom directory, you may not even need to update the catalog path.
Best practice is usually to use the "custom" folder. Think of your shell DTD as any other DTD, and follow the usual Arbortext process.
It goes something like this: 1. Create folder inside custom/doctypes named "myDoctype" 2. Copy in the "myDoctype.dtd" 3. Setup catalog file to link the PUBLIC ID for your doctype to "myDoctype.dtd" (good idea to setup your .ent and .mod here too) 4. Setup DCF
With step #4 I recently discovered you can inherit from the base Arbortext DITA stuff. Have a look at $ARBORTEXT$\application\com.arbortext.sma\doctypes\servicemanual\servicemanual.dcf for pointers.
I think by default Arbortext will recognise your specialisation and apply the out-of-the-box DITA styling but you can also make your own stylesheet(s) too.
You can distribute your custom folder as a folder or (since 6.0?) as a ZIP. The client will use the APTCUSTOM environment variable to locate your customisations.
We already distribute using a custom zip so it's good to know that we should be able to use this also for DITA specializations. I will now set off optimistically in that direction.