Exactly my opinion. The deviation curve requires an order of magnitude
increase for results to meet the minimum for accuracy of +/- 1% At 71 with a
range of 100 (percentile) the deviation range is something more in the
neighborhood of 15%. Any result with less than a 15% difference is
therefore invalid. All his results showed less than 15% difference
rendering them all invalid. Since the 'perfect' result is likely less than
15% in any event, the sample size would have to be large enough for a
standard deviation of 1 to be larger than the 'perfect' resultant difference
noted. This means that doing less than 1000 tests, enough for a 1%
variation to be detected consistently, is not really bona fide.
Kitasi
-----Original Message-----
From: John Robertson [mailto:
johnr@...]
Sent: Thursday, October 28, 1999 11:33 AM
To:
eqbards@onelist.com
Subject: Re: [eqbards] Interesting thread I found on a Monk board
From: John Robertson <
johnr@...>
Rokenn Swiftsong wrote:
> From: "Rokenn Swiftsong" <rokenn@...>
>
> On a monk board I have been reading lately (started playing my monk again)
I
> found a link to an interesting discussion on the benefits (or lack there
of)
> of stat buffs. The Shaman that ran the test came to the conclusion that
> that buffing Str had a negative impact on damage. Here is a link to the
> thread:
>
> http://www.soerbaird.com/forums/Forum16/HTML/001893.html
>
> I would love to read John's comments on this :)
Well, different John here... but I'll comment nonetheless.
BTW, I really dislike that John Kim has usurped the name John here. :)
Anyway, speaking as someone who does this sort of testing professionally, I
can tell you that
this person did a lot of work into the test, but personally, I wouldn't take
the results to the
bank.
The main reason is that he just didn't have a big enough sample size, in my
opinion. His
smallest test case had only 71 attempted hits, and of those, only 55% hit.
That combined with
the variability of the possible results (1 damage to 39 damage) makes it
quite possible for
random variation to infuence the results more than they should.
For example: We all know that the numeric mean of rolling a six sided die is
3.5. If you rolled
a 6 sided die an infinite number of times, assuming the die is perfectly
balanced, you should get
a mean of 3.5 -- but you don't need to do it that many times. The way
randomness works means
that you can roll it a lesser number of times, and depending on how many
times you roll that die,
be PRETTY sure that the result you get will accurately reflect the true
result. That's what
statistics and probabability is all about.
Now, using the 6 sided die, if you rolled it 71 times, would you get 3.5 as
the mean? I'd guess
probably. 71 times is a lot of rolls when there are only 6 possible
results.
On the other hand, if you roll it 71 times and it's a 40 sided die, do you
think you'd get the
true mean of 20.5? Maybe.... but I'd expect a result something in the
neighbourhood of 18-24.
That's the effect of variability. And this test didn't even get 71 hits, he
got 39. Do you
think you could roll a 40 sided die and get an accurate reading of the
average roll with 39
tries? Try rolling a six sided die 5 times and seeing if you get an average
of 3.5.
If you've taken a lot of stats courses, you'll know what I mean by the
concept of "power". Power
determines the size of sample you'll need to get within a certain allowable
variability. I could
whip out my stats book and figure out the power needed to get an acceptable
result for this
test... but to be honest, I hate stats, so I'll wimp out. :)
These tests MAY be valid, but given the weird results, I would take them
with a grain of salt.
The person may be on to something here, I won't deny it, but I'd like to see
the test repeated
and compare the results.
Jyzan
Please send submissions for the eqbards newsletter to
toren@... with
the subject lol.