why had some ancient boats such high stem- and sternposts ?

I think this use is a lot later than the viking boat, and a different derivation; The 'galleon' and 'man of war' with the fo(re)castle and aft(er)castle on top heavy, high windage vessels, like the Mary Rose, whose sinking may have been caused, in part by top heavy modifications:

 

I googled images of viking ship bows and after separating modern kayaks branded "Viking" from viking ships and then fake replicas from the genuine ones and ceremonial boats from "working boats", seagoing, coastal and inland use, I have the impression that the high dragon heads are ceremonial where scaring the gods of Valhala was more important than reducing windage.

Also, proportionally, the rudders look minimal, again stressing the point of having the boat balanced in most conditions unless of course rowing would the general fallback. In one replica I counted 72 oars. If in sync these should produce decent horsepower for maneuvering.

 

There are some very good reasons why a lot of the ancient boats have high sterns and stems.
1. Most of the boats are not decked over completley, they are open boats.
2. They were beached and needed to go trough surf.
3. Side rudders allowed freedom of form.
4. Very early boats had tension ropes along the centerline (or that is what some believe).
5. Double ended is the simplest and strongest form of wood construction.
6. The posts are often used as decoration.

This all applies to high L/B ratio boats used mainly for war. Commercial boats were wider, higher freeboard and had not so high ends (relatively speaking).
Take an ancient war galley. It has a high curved stern for wave protection. The boat can not take enough supplies for its crew so it is mandatory to stop regularly. This would often have been on a beach. Having a high stern prevented the boat to be swamped when beaching bow to, and even beach stern to.
The stem looks concave to straight because there is a ram extension just at the waterline extending the keel. The stempost is usually vertical but there is a knee between ram and stempost giving the concave apperance. The stempost is left high over the planks to provide protection from debree while ramming and provide a handhold for boarding soldiers. The decoration it carries is very important for identification in battle and needs to be highly visible.
Wiking ships are about the same in function but without the ram their stems have no need to look concave, so they use a convex bow.

 

The plot thickens. Apparently no masts nor rigging from viking times were ever recovered. One theory holds that the most fitting sail would not have been the square sail as usually depicted but a 3:1 rectangular sail, horizontal and maybe canted depending on the conditions. One reference discussing the Sulawesi pajala-style ship with tripod masts and canted rectangular sails explained that if a bowsprit was lacking the top of the stempost would be used.

 

As a rank novice on a design forum frequented by professionals, I'm a little hesitant to stick my oar in here. But I can think of several possible reasons why ancient builders liked high bows and sterns on double ended boats.

1. At sea they parted the waves and helped keep the boat dry, both when headed into waves and when hit with following seas.
2. Those old boats were probably beached more often than they were tied up, which means they had to deal with surf on a regular basis (remember the old lifeguard surf dories?)
3. Having longer stem and stern posts meant more planking could be carried onto them, thereby stiffening and strengthening the overall structure.
4. It just by golly looked good.... never discount aesthetics as a motivating force.

 

Being even more of a novice, it seems to me that the advantages of high stem/stern far outweighed any disadvantages; hence their existence and persistence.

They could have very well been used in the initial building and planking, but rather than cut them off later, they could be used for tying lines, rigging, lookout posts (by the able-bodied sailors shimmying up the "neck"), tying a line stem to stern to dry the laundry, beaching, climbing steep banks to raid villages, etc. Why get rid of such a handy protrusion?

Plus, I'd always been told they wanted the ships to look like sea serpents to fool the landlubbers.

 

We have been told a lot about the Vikings. For example, they did not wear large horns on their helmets.
Also, there is a bias in the boats we recovered. The best preserved seem to be burial, thus ceremonial, boats. The remains of half a dozen of boats sunk on purpose to block a harbor show different sizes and do not have hints of high bow- or stern-posts. If these posts had good uses in daily practice I would expect structural details (like steps or holds or holes ...) worked out or marks of wear and tear from chafing ropes supporting such uses.
So ceremonial boats would be built to impress according to the taste/popular myth at the time but not need to pass the test of time of actual use.

 

IMO about 90% of the time they have a line running from stem to stern high-points and turn a big rectangle sail into a tent. IMO all sail boat designs need to include pre-engineered, pre-planned schemes for deploying their various sails as expedient sun/rain tarps, and even Expanded Living Space Deck Tent.

"climbing steep banks to raid villages" You might be onto something. There is a lot of coastline that has a small, but steep, little cliff where a soft sandy pasture meets waves and tides.

 

Warfare, as in shooting arrows and guns down upon the enemy. It is like being in a high tower.