2.P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

【略】
Retraction Watch: Some of the principles seem straightforward, but I was curious about #2 ? I often hear people describe the purpose of a p value as a way to estimate the probability the data were produced by random chance alone. Why is that a false belief?

Ron Wasserstein: Let’s think about what that statement would mean for a simplistic example. Suppose a new treatment for a serious disease is alleged to work better than the current treatment. We test the claim by matching 5 pairs of similarly ill patients and randomly assigning one to the current and one to the new treatment in each pair. The null hypothesis is that the new treatment and the old each have a 50-50 chance of producing the better outcome for any pair. If that’s true, the probability the new treatment will win for all five pairs is (?)⁵ = 1/32, or about 0.03. If the data show that the new treatment does produce a better outcome for all 5 pairs, the p-value is 0.03. It represents the probability of that result, under the assumption that the new and old treatments are equally likely to win. It is not the probability the new treatment and the old treatment are equally likely to win.

This is perhaps subtle, but it is not quibbling.  It is a most basic logical fallacy to conclude something is true that you had to assume to be true in order to reach that conclusion.  If you fall for that fallacy, then you will conclude there is only a 3% chance that the treatments are equally likely to produce the better outcome, and assign a 97% chance that the new treatment is better. You will have committed, as Vizzini says in “The Princess Bride,” a classic (and serious) blunder.

両側，片側のどちらを用いるかは，ほんらいデータを集める前に決めておくべきことである．両側検定をするつもりだったが，データを集めた後に平均がＡ＞Ｂだったから片側検定にするなどというのは，検定の大原則に反する．
　片側検定（例：Ａ≦Ｂを棄却してＡ＞Ｂを見出す）は，本質的にＡ＜Ｂがありえない場合［補注２］，あるいはＡ＜Ｂを考慮に入れる必要がない場合［補注３］に用いられる．これら以外の場合は，原則として両側検定が推奨される［補注４］．
　両側検定の場合は“ＡとＢの差は有意であった”とは言えるが，“ＡよりＢのほうが有意に大であった”とは言えない．Ａ＞Ｂと結論することは実際的には問題がないが（近藤・安藤, 1967, p.16），これは検定の結果ではなくて，信頼限界に基づく推定の結果であることを理解しておかなければならない．

［補注３］：例えば，ある添加物に発癌性があるかどうかを検定するために，その添加物を投与したラットにおける癌の発生率Ａと投与しなかった対照群のラットにおける発生率Ｂを比較したとしよう．この場合，Ａ＜Ｂとなる可能性，つまりその添加物には癌の発生を押さえる効果があるかどうかということは当面の議論とは無関係であるから考慮に入れる必要はない．
［補注４］：χ^２検定やＦ検定では原則として片側確率のみが意味をもつが，検定結果は“Ａ＞Ｂ”ではなくて“有意差あり”という両側検定的な表現となる．ただし１標本の検定でχ２検定を用いて“Ａ＞Ｂ”と結論される場合や，母分散が既知の値に等しいか否かを検定するような場合を除く．