Completed another round of early playtesting this past weekend, with interesting results.
I had a simple outline for the group that evening; give a brief rundown of the core rules, explain character creation rules and create characters, and end with running some combat simulations. The group understood the rules easily enough, created some diverse characters to test the combat system, and we were off.
Combat flowed fairly well, with some good input from the players. I was doing one-on-one combat with each player instead of group combat, mainly because I wasn't quite ready to test out that mechanic yet. Players gave their input after a few turns of combat, and I jotted down notes during and after each combat.
The playtest was pretty much done, and there were some good conversations about the mechanic and how to implement it in this setting or that; I have to admit, I was feeling pretty stoked.
Then one of the players asked a question about the resolution mechanic. It gave me pause, because I didn't have an immediate answer. The group discussed the issue, and delved deeper into the root cause and potential fixes. More testing started up to quantify the issue, and how to resolve it in the future.
This may seem like a failure, but I count it as a huge win. Sometimes, you're so close to what you're working on, you fail to see what may be wrong; that whole, "Forest for the trees" thing. I can't stress enough how important playtesting is when developing. Even when you think you're done, do it some more.
I'm also very fortunate to have some very sharp individuals for playtesting. They really dove in with little guidance, and put the mechanic through its paces. That last part about testing and trying to resolve it? That was all them.
I've run some simulations, using the mechanic as is, as well as a revised version. Interesting results thus far, and there may not be as much of an issue as previously thought. I'm hoping to do some extensive testing this weekend, using some scripting to get a large batch of results. AnyDice is great, but I need something a little meatier. Hopefully I won't have to reinvent the wheel; developing an RPG system is enough work as is, thank you very much. :)