Number of files in the Attachments table

This was just my solution. There may be others that are easier but this should allow you to create that hook or place to insert your custom logic. I modified the create and edit jsp wizards for WTDocument to add the following custom step.

<%-- adds attachments to the document --%>
<jca:wizardStep action="attachments_step" type="attachments" />

<%--- Customizations for Electrical Eng Docs --%>
<%-- process if matches drawings as documents --%>
<jca:wizardStep action="process_drw_docs" type="document" />

The action "process_drw_doc" is a reference that I then defined in config/actions/custom-actions.xml:

<objecttype name="document" class="wt.doc.WTDocument" resourceBundle="com.ptc.windchill.enterprise.doc.documentResource">
 <action name="process_drw_docs" preloadWizardPage="false" hidden="true">
 <command class="customization.electrical.DrawingDocumentProcessorDelegate" windowType="no_content"/>
 </action>
 </objecttype>

That "delegate" class is executed but noticed that hidden=true so users never see this step. This is the basic form of that class implementing the postProcess method where you can put your checks.

public class DrawingDocumentProcessorDelegate extends DefaultObjectFormProcessorDelegate { 

	/**
	 * Delegate uses only postProcess hook to inspect WTDocument. postProcess
	 * will evaluate the type against list in ee.properties.
	 * @param bean NmCommandBean
	 * @param list Object Bean list
	 * @return FormResult
	 * @throws wt.util.WTException
	 */

	@Override
	public FormResult postProcess (NmCommandBean bean, List list) throws WTException
	{
		log.debug("entering postProcess");
...

 You can get the reference to the WTDocument from the list which are ObjectBeans and finding the one that has the WTDocument on it. 

/**
	 * returns WTDocument associated with CreateEdit Form. Should only be called from Edit.
	 * @param list Object Bean list
	 * @return WTDocument
	 */
	private WTDocument getDoc(List list)
	{
		//should be only one document, therefore one object bean
		for (Iterator<ObjectBean> beans=list.iterator();beans.hasNext();)
		{
			ObjectBean objBean=beans.next();
			//if object on bean is a WTDocument, continue
			if ((objBean.getObject() != null)&&(objBean.getObject() instanceof WTDocument))
				return((WTDocument)objBean.getObject());
		}

		log.debug("Could not find WTDocument. Returning null");
		return null; //WTDocument not found in bean.
	}

From there its standard method to inspect attachments. If they violate your rules, return a FormResult that indicates such.