I use a calc program that comes very close to the Wageningen series, and when feeding the gnome with your measured values for speed, rpm, fuel flow and the resulting real power, the values show that the wake factor is ~0,8, meaning that the average inflow into the propeller is 80% of the boat speed through the water.
By calculating the thrust values up to the 8,1 knots measured, I get a rough value for the hull resistance, showing that with the 110 hp Yanmar and the prop alternatives possible, you would have a maximum speed of just shy of 11 knots. Operating this engine at 3500 rpm with the 17"x14" prop means it is running on the fuel stop curve. I'd estimate it will produce something around 106-108 hp there. As noted by PhilS this prop is too "light" for the engine.
Based on numbers above, you should aim at a 17"x16" screw with the same blade area as you have now. This will allow the engine to work with slightly higher effective pressures, which is better for the combustion, avoiding "glazing" of cylinder walls and keeping the turbo clean. If you look at the power curve, you see that the max power is delivered already at ~2900 rpm. Aiming at a max rpm around 3000-3100 will give you better cruising performance.
Considering the low inflow velocity, the effective angle of attack on the blade leading edge is on the high side already with the 14 pitch prop. Thus it would be better to use this prop and add a "coarse" cup to it, rather that going for a 16 pitch with no or small cup. The point here is that the blade leading edge should operate with a low angle of attack in the present case, where you have a combination of disturbed inflow, high thrust load and possible ventilation from behind.
With the propeller aperture available, there is little to gain by increasing diameter to 18", which would increase the risk for blade frequency vibrations. As said, the best option for performance improvement is a big dump box...... the hull is overweight for its length and the underwater shape is bad. Your propeller efficiency now is about 38% while cruising at 8 knots. The higher pitch will not be radically different efficiency-wise, but much better for the engine.