Its 100% fail safe. I could use a 100 people and be fine. I am hoping to spend around 10 minutes total for the effect. So I think I will try for 5. I mainly just need to work on some routing and then do a run through. the 10 minutes will include the transition from the 1st effect to this one, getting audience etc. as well as the actual trick part.
Time the trick when you practice it for the first time. It will probably run a bit longer than ten minutes, though not much longer. Also bear in mind that you're introducing an utterly random factor to your stage - five complete and total strangers, any of whom could be a bit dim-witted, a bit drunk, a heckler-in-waiting, take forever to make a drawing, scared to immobility in front of an audience, etc., and you can't toss the fish back once you've reeled it in without looking a bit silly. Have your "staff" (or you, if there is no assistants) do a little pre-screening as people enter the venue so you can at least attempt to minimize the level of "personality randomness" you'll have to deal with in your performance.
I recall seeing a performer doing this who had the volunteers drawing with their backs to the audience onto easel-mounted pads. This probably kept them more focused on the task at hand and less concerned about who's watching them, because out of sight does quickly become out of mind. He had himself blindfolded in a very secure fashion while they made their artworks - or he was off-stage, I can't remember! An alternate way to do this would be to step in front of the stage at audience level, facing them the whole time and having no opportunity to see the drawings too soon without getting caught.