To limit the sail area is a good starting point in my opinion, and even more I would start from a full defined sailplan, e.g. the one of an existing popular cruising sailboat to have it at a low cost, and doing so to focus the design exercise on what hull(s) for such saiplan.
A rule based on a sum of hull length + etc ... reminds the old Sonder klasse rules, where :
Lwl + Beam + Draft < 9,75 m ; Sails area < 51 m2 : Displacement (all up, without crew) > 1830 kg (+ minimum of thickness for the hull and deck planking : > 16 mm).
International Sonder Class (sonderklasse-yachten.de)
How can we modernise such box rule, some ideas :
** Here the waterline Lwl is a problem, that favors too much the overhangs. But to take the Lhull has the reverse effect, to favor no overhang, the alas current unaesthetic trend also unknowingly supported by the various license and marina taxes based on length overall. If we drop Lwl, same effect, no overhang will be the final optimum. May be one can try something like : Beam + Draft - 0,1 * Loverhangs < ... , i.e. some overhangs can be rewarded by more beam or more draft (the hull length being anyway constrained by the Displacement and the necessary beam).
** To add also a maximum for the draft , for example 1,8 m , or perhaps better to use Draft^2 in the formula to penalise the over draft.
** To add that the keel is made of cast iron or lead or a combination of the two (wing in cast iron + bulb in lead)
** As already suggested, to change the sails area by the sailplan full definition (and weight)
** For the displacement (without crew), to change the minimum thickness by average mass/squared meter of the hull surface and of the deck one. And to add a minimum of mass dedicated for the accomodation + the mass of the selected sailplan.
** And last but not least, the need to fulfill the stability requirements for a classification in a sailing category, for example A2/B6 (2 people for an ocean crossing, 6 people for a coastal cruising/racing at < 200 NM), implying thresholds for the AVS, the STIX, the righting moment energy.