I tried to run the sample application Install_test provided with j-link, which resulted a message "Startup of InstallTest failed" when running Creo (Check attachment), everything I did was as per the j-link user guide
CLASSPATH: C:\Program Files\PTC\Creo 3.0\M040\Common Files\text\java\pfc.jar;C:\Program Files (x86)\QuickTime\QTSystem\QTJava.zip;C:\Program Files\Java\jdk-9.0.1\bin\java.exe;
PATH: C:\ProgramData\Oracle\Java\javapath;C:\Program Files\Java\jdk-9.0.1\bin;C:\Program Files\PTC\Creo 3.0\M040\Common Files;
I first attempted to compile the java files (expect Asynch'') in that folder using windows command prompt (javac *.java) it resulted in an error (see attachment). I assumed may be its because we don't hold the "write" rights over that directory and copied the folder to different location and did the same, the java files got compiled and 4 class files got created. Now opening Creo (from that directory) said the same "Startup of InstallTest failed". I copied the 4 created class files to the original load-point and did run Creo and it resulted the same though. I have attached few screenshot of folders, config file and etc.
Using Creo 3.0, Windows 8
Looking forward for a response,
see my reply from 2017-12-12 01:28 PM on https://community.ptc.com/t5/Creo-Modeling-Questions/HELP-J-LInk-can-t-setup/m-p/494098 page. Maybe it helps you to resolve your problem.