Prediction of lift/drag values, dynamic stability, etc. in the interface between hull, air, and water surface via mathematical formulae quickly becomes VERY esoteric and is best practiced by post-graduate physicists and hydrodynamicists - check out papers on the topic at The Society of Naval Architects & Marine Engineers (SNAME). The usual practice among lesser mortals is to use general formulae from texts such as Lord's "Naval Architecture of Planing Hulls" and others to get a baseline data set, then refine the design with public and proprietary hull data from previous designs or towing tank experiments. Usually the designer has a "good idea" and then builds either a full-size or scale model and performs calibrated trials to prove and improve the initial design. Mostly seat of the pants (in an expensive and formal way) engineering, I'm afraid.