This depends a bit upon the release and where you're using the custom jar.
If this is for a release prior to R10, then put the jar in WEB-INF/lib if you want it accessible to the servlet engine or WT_HOME/lib if you want it accessible only to the method server. In R10 and higher, either location will work fine, though it's worth noting that WT_HOME/lib jars are after those in WEB-INF/lib in the classpath lookup.
In any case if it is to be used by applets, then a copy will be needed somewhere in codebase but outside WEB-INF. The standard location in this case would be codebase/lib.
In any case, double-check the Windchill 3rd-party support matrix to be sure that the jar you're adding isn't already bundled with Windchill -- as Windchill requires the versions it bundles be the versions used.
