I was going to ride my first RAGBRAI on my mountain bike. Could I have done it, yes, hands down. I had rode lots of longs days with it before. Am I glad I bought a road bike instead… yeah. It’s not necessary the speed, but the ease. Given the different road conditions and weather conditions. I would suggest at least trying one out for a few days and do some longer rides and then see what you think.
As for your question, how fast do your ride…it depends, and that’s where a bike you know you can take it easy on and still keep up and have a leisurely conversation w/o worrying about making to the end in time or straining. Try a road bike and see if it makes you more comfortable with your speed.