As others have mentioned, we DO have mains that aren't triangular. Look at J/70s, catamarans, many high performance dinghies and most skiff types, etc.
As has already been pointed out, there are practical issues with squaretops etc. For one, to hold the leach out you need battens, and those are surprisingly heavy (modern high tech windsurfer sails, for example, are often twice as heavy as the low-tech original windsurfer sails because even carbon fibre battens are comparatively heavy), can be expensive, and can be a hassle. For example, many squaretops need you to take out the top batten every time the main is set or dropped. The tension that battens place on the mainsail luff can also mean that the sail is very hard to drop and hoist unless you have expensive (and heavy) battens cars.
The extra weight of battens, like the extra weight of wing masts, means that all else being equal, you have to reduce mast height or suffer more heeling force. However, since tall and skinny rigs often work more efficiently it can often be pretty much as good to put up a simpler, lighter but taller rig of the same heeling moment.
If you try to make a cheaper squaretop sail you run into major problems with cloth stretch, which can mean that the whole head of the sail falls away to leeward. To keep the leach taut with a fat head sail you may need to use very high vang and mainsheet tensions, so you may need to spend a lot more in mainsheet blocks, vang controls, etc. I have some "high tech" windsurfer squaretop sails of about 5m that would probably carry the same sort of downhaul tension as the pinhead dacron mainsail of my 36 footer. That 36 footer uses the old mainsheet system off our 20 foot cat, which had a squaretop.
This is a fascinating area, and here on BDF we have been incredibly lucky in the past to have several regular posters who are top aerodynamics experts. Search
"tspeer" and "mdrela". I won't out them apart from saying that one of them is a professional wing designer with America's Cup expertise while the other is a leading MIT professor of aerodynamics and an America's Cup designer. Also look up SHC, a world champ in development classes and Mikko Brummer, a sailmaker.
The really interesting thing about the people mentioned above is that unlike many "experts", what they say about sails actually fits in with what we actually see happening on the water. For example, Tom points out that we don't need an elliptical sail outline to achieve what we want aerodynamically, which is an elliptical "span loading". The "span loading" is created not just by the sail's shape and area, but also by its twist, the apparent wind, and the depth. We can therefore get many of the aerodynamic advantages of a non-triangular sail by merely setting the correct leach twist and depth in a triangular sail.
Another relevant point that Mdrela has neatly pointed out is that for many boats, reducing aero drag is not a major speed factor. What matters, he notes, is not the lift/drag factor of the rig, but the lift/drag factor of the whole boat. If you have a boat with high hull drag then the aero drag can be of little importance, and it's worthwhile creating a rig that has higher drag but more power. Perhaps the analogy can be between a truck and a bicycle - one needs lots of power and aero drag is largely irrelevant, while the other has very little power so reducing aero drag can be critical. You can really see this in windsurfers. This means that the fact that a certain type of sail works on really high performance craft does NOT mean that the same type will work on "normal" boats which have different characteristics.