I did an upgrade from 5.3 M060 to 5.4 M070 last year. I remember that I needed to set the APTAPPLICATION environment variable to the same directory as the APTCUSTOM variable for everything to work properly. I had to set this variable on both the upgraded PE and the upgraded client PCs with Editor that were referencing the central custom folder. This took some time to figure that out because APTAPPLICATION wasn't required in 5.3 for our customizations to work.
If you don't have that variable set, give it a try, reboot and see if that helps.
I pointed it to the custom directory because that is where the baseapplication.xml file is stored. I'm not sure if you need a separate application.xml for each of your custom doctypes or not.