Tolerance height is controlled by the drawing setup option tol_text_height_factor. If you are using anything other then standard or "1", then you tolerances will show as a different size than the dimension font. The text added in using this method will always display as dimension font size and not tolerance size.
The only other way I can think of to do this is to just add a separate note with tolerance (with desired font size) and manually position it next to the dimension (I would attach it to the dimension or use relate to object so it moves with the dimension)
