Author: Eiko

Time: 2024-12-28 15:59:49 - 2025-01-03 12:56:59 (UTC)

Reference:

• Theoretical Statistics by Robert W. Keener

• Probability Theory by Eiko

• Foundations of Modern Probability by Olav Kallenberg

Let $\theta \in \mathrm{\Omega }$ be some parameter space and $\mathrm{\Omega }={\mathrm{\Omega }}_0\cup {\mathrm{\Omega }}_1$ be a partition of this space, $X\sim \mathbb{P}(X|\theta )$ be certain law. $H_i$ is a hypothesis that $\theta \in {\mathrm{\Omega }}_i$.

Hypothesis testing aims to tell which of the two competing hypotheses $H_0$ or $H_1$ is correct by observing $X$.

Test Functions

Non-Randomized Tests

• A non-randomized test of $H_0$ versus $H_1$ can be specified by a critical region $S$, so if $X\in S$ we reject $H_0$ in favor of $H_1$.

• Power function ${\beta }_S:\mathrm{\Omega }\to \mathbb{R}$ describes the probability of rejecting $H_0$ given $\theta$

  $${\beta }_S(\theta )=\mathbb{P}(X\in S|\theta )$$

• Significant level $\alpha$ is the small error calculating the worst error rate of falsely rejecting $H_0$ when it is true.

  $${\alpha }_S=\underset{\theta \in {\mathrm{\Omega }}_0}{sup}{\beta }_S(\theta ).$$

  In theory we would want ${\beta }_S(\theta )=1_{{\mathrm{\Omega }}_1}$ which would imply ${\alpha }_S=0$, but this is not possible in practice.

Randomized Tests

Sometimes instead of giving a critical region $S$ or equivalently a function $1_S$, we give a critical function $\phi (x)$ instead, reflecting the probability of rejecting $H_0$. Then a non-randomized test is just a special case of $\phi =1_S$.

In this case, the power function is

$${\beta }_{\phi }(\theta )=\mathbb{E}(\phi (X)|\theta )$$

and the significant level is

$${\alpha }_{\phi }=\underset{\theta \in {\mathrm{\Omega }}_0}{sup}{\beta }_{\phi }(\theta )=\underset{\theta \in {\mathrm{\Omega }}_0}{sup}\mathbb{E}(\phi (X)|\theta ).$$

The main advantage of randomized tests is that they can form (convex) linear combinations.

Simple Hypothesis And Simple Tests

A hypothesis is simple if ${\mathrm{\Omega }}_i$ is a singleton.

Neyman-Pearson Lemma

Assume $H_0$ and $H_1$ are both simple, in this case there is a Neyman-Pearson Lemma describing all reasonable tests. Let ${\mu }_1=\mathbb{P}(X|{\theta }_1)$ and ${\mu }_0=\mathbb{P}(X|{\theta }_0)$ be the distributions of $X$ under $H_1$ and $H_0$ respectively.

We have

$${\alpha }_{\phi }={\mu }_0(\phi )=\int \phi (x){\mu }_0(dx)$$ $${\beta }_{\phi }({\theta }_i)={\mu }_i(\phi )=\int \phi (x){\mu }_i(dx).$$

We would want to choose $\phi$ such that ${\mu }_0(\phi )\to 0$ and ${\mu }_1(\phi )\to 1$. Consider maximizing ${\beta }_{\phi }({\theta }_1)$ subject to ${\alpha }_{\phi }\le \alpha$.

Lagrange Multiplier Lemma

• Let $k\ge 0$ be any constant, then maximizing ${\mu }_1(\phi )-k{\mu }_0(\phi )$ gives the function ${\phi }^{\ast }$ maximizing ${\mu }_1(\phi )$ subject to ${\mu }_0(\phi )\le \alpha$, here $\alpha ={\mu }_0({\phi }^{\ast })$.

  $$\begin{array}{rl}{\phi }^{\ast }& \in {\mathrm{argmax}}_{\phi }({\mu }_1(\phi )-k{\mu }_0(\phi ))\\ & \subset {\mathrm{argmax}}_{{\mu }_0(\phi )\le \alpha }{\mu }_1(\phi ).\end{array}$$

• Moreover, any function ${\phi }^{\ast \ast }$ maximizing ${\mu }_1(\phi )$ subject to ${\mu }_0(\phi )\le \alpha$ must have ${\mu }_0({\phi }^{\ast \ast })=\alpha$.

  $${\phi }^{\ast \ast }\in {\mathrm{argmax}}_{{\mu }_0(\phi )\le \alpha }{\mu }_1(\phi )\subset \{\phi :{\mu }_0(\phi )=\alpha \}.$$

Note that ${\phi }^{\ast }$ and $\alpha$ depend on $k$ here.

Proof.

• Let ${\phi }^{\ast }$ be the maximizing function. It suffices to prove that ${\mu }_0(\phi )\le \alpha \Rightarrow {\mu }_1(\phi )\le {\mu }_1({\phi }^{\ast })$.

  We have

  $${\mu }_1(\phi )-k({\mu }_0(\phi )-\alpha )\le {\mu }_1({\phi }^{\ast })-k(\mu ({\phi }^{\ast })-\alpha )$$

  So ${\mu }_0(\phi )-\alpha \le 0\Rightarrow -k({\mu }_0(\phi )-\alpha )\ge 0$ therefore

  $$\begin{array}{rl}{\mu }_1(\phi )& \le {\mu }_1(\phi )-k({\mu }_0(\phi )-\alpha )\\ & \le {\mu }_1({\phi }^{\ast })-k({\mu }_0({\phi }^{\ast })-\alpha )\\ & ={\mu }_1({\phi }^{\ast }).\end{array}$$

• We know that ${\phi }^{\ast \ast }$ and ${\phi }^{\ast }$ are both in ${\mathrm{argmax}}_{{\mu }_0(\phi )\le \alpha }{\mu }_1(\phi )$. Therefore ${\mu }_1({\phi }^{\ast \ast })={\mu }_1({\phi }^{\ast })$. The fact that ${\phi }^{\ast }\in \mathrm{argmax}({\mu }_1(\phi )-k{\mu }_0(\phi ))$ implies

  $$\begin{array}{rl}{\mu }_1({\phi }^{\ast })-k{\mu }_0({\phi }^{\ast })& \ge {\mu }_1({\phi }^{\ast \ast })-k{\mu }_0({\phi }^{\ast \ast })\\ & ={\mu }_1({\phi }^{\ast })-k{\mu }_0({\phi }^{\ast \ast }).\end{array}$$

  Therefore ${\mu }_0({\phi }^{\ast \ast })\ge {\mu }_0({\phi }^{\ast })=\alpha$ by definition.

How To Maximize ${\mu }_1(\phi )-k{\mu }_0(\phi )$ ?

We know that ${\mu }_1-k{\mu }_0$ is a finite signed measure, so according to Hahn decomposition, any finite signed measure can be uniquely decomposed into the difference of two mutually singular finite measures

$${\mu }_1-k{\mu }_0={\nu }_{+}-{\nu }_{-}.$$

So maximizing ${\mu }_1(\phi )-k{\mu }_0(\phi )$ is equivalent to maximizing ${\nu }_{+}(\phi )-{\nu }_{-}(\phi )$. From which it is clear that we can pick $\phi =1_{A_{+}}$ where $A_{+}$ is the set where ${\nu }_{+}$ is concentrated, and there is a freedom for us to pick anything from $[0,1]$ on a set of measure zero in $|{\mu }_1-k{\mu }_0|$.

If ${\mu }_i$ can be written as density functions, then the set $A_{+}$ is simply $\{x:\frac{\mathrm{d}{\mu }_1}{\mathrm{d}\mu }>k\frac{\mathrm{d}{\mu }_0}{\mathrm{d}\mu }\}$. This can be seen as a slight generalization of a likelihood ratio test, if we ignore the division by zero problem, it can be written as $\{\frac{\mathrm{d}{\mu }_1}{\mathrm{d}{\mu }_0}>k\}$.

The Neyman-Pearson Lemma

The Lemma states that, for a simple test scenario, given any level $\alpha \in [0,1]$, there exists a likelihood ratio test (which means $1_{L>k}\le \phi \le 1_{L\ge k}$ and potentially some other function values on a measure zero set) ${\phi }_{\alpha }$ with exactly level $\alpha$ (i.e. ${\mu }_0({\phi }_{\alpha })=\alpha$). The likelihood ratio test ${\phi }_{\alpha }$ is chosen to be maximizing ${\mu }_1(\phi )-k{\mu }_0(\phi )$ and any likelihood ratio test maximizes the power function ${\beta }_{{\phi }_{\alpha }}({\theta }_1)$ subject to the significant level ${\alpha }_{{\phi }_{\alpha }}\le \alpha$.

Some Detailed Results Relating To Neyman-Pearson Lemma

• For $\alpha \in [0,1]$, let $k$ be a critical value for a likelihood ratio test ${\phi }_{\alpha }$ in the sense of Neyman-Pearson Lemma, i.e.

  $${\phi }_{\alpha }=1_{\{x:\frac{\mathrm{d}{\mu }_1(x)}{\mathrm{d}{\mu }_0(x)}>k\}}\text{ a.e. in }|{\mu }_0-k{\mu }_1|.$$

  Then ${\mu }_0({\phi }_{\alpha })=\alpha$ and ${\mu }_1({\phi }_{\alpha })={\beta }_{{\phi }_{\alpha }}({\theta }_1)$.

  We have

  $${\phi }^{\ast \ast }\in {\mathrm{argmax}}_{{\mu }_0\le \alpha }{\mu }_1\Rightarrow {\phi }^{\ast \ast }={\phi }_{\alpha }\text{ a.e. in }|{\mu }_0-k{\mu }_1|.$$

• If ${\mu }_0\ne {\mu }_1$ or $k\ne 1$, with ${\phi }_{\alpha }$ a likelihood ratio test with level $\alpha \in (0,1)$, then ${\mu }_1({\phi }_{\alpha })>\alpha$.

$${\mu }_0\ne {\mu }_1\Rightarrow {\mu }_1({\phi }_{\alpha })>\alpha .$$

Proof.

• We already proved that ${\mu }_0({\phi }^{\ast \ast })={\mu }_0({\phi }_{\alpha })=\alpha$. Since

  $$({\mu }_1-k{\mu }_0)({\phi }^{\ast \ast })=({\mu }_1-k{\mu }_0)({\phi }_{\alpha })$$

  and by the construction of ${\phi }_{\alpha }$, we know that ${\phi }_{\alpha }-{\phi }^{\ast \ast }\ge 0$ a.e. in $|{\mu }_0-k{\mu }_1|$. This implies ${\phi }^{\ast \ast }={\phi }_{\alpha }$ a.e. in $|{\mu }_0-k{\mu }_1|$.

• Consider the constant test ${\phi }_c=\alpha \in (0,1)$, by ${\phi }_{\alpha }\in {\mathrm{argmax}}_{{\mu }_0\le \alpha }{\mu }_1$ we know ${\mu }_1({\phi }_{\alpha })\ge {\mu }_1({\phi }_c)=\alpha$. If equality holds then ${\phi }_c$ is also in the set, thus ${\phi }_c={\phi }_{\alpha }$ a.e. in $|{\mu }_0-k{\mu }_1|$, but this equality never hold since ${\phi }_{\alpha }\in \{0,1\}$ a.e. in $|{\mu }_0-k{\mu }_1|$. The only possible case is ${\mu }_0={\mu }_1$ and $k=1$.

Examples

• Suppose we are testing

  $$\mathbb{P}(X|\theta )\sim \text{Exponential}(\theta )\sim \theta e^{-\theta x}{1}_{x\ge 0}\mathrm{d}x$$ with hypothesis $H_0:\theta ={\theta }_0$ and $H_1:\theta ={\theta }_1$, for simplicity assume ${\theta }_1>{\theta }_0$. The likelihood ratio test is of the form

  $$\frac{{\theta }_1{e}^{-{\theta }_{1}x}}{{\theta }_0{e}^{-{\theta }_{0}x}}>k\iff x<\frac{1}{{\theta }_1-{\theta }_0}\log \frac{{\theta }_1}{k{\theta }_0}=x_k.$$

  $$\alpha ={\mu }_0\phi ={\int }_0^{x_k}{\theta }_0{e}^{-{\theta }_{0}x}\mathrm{d}x=1-e^{-{\theta }_0{x}_k}.$$

  $$x_k=\frac{1}{{\theta }_0}\log \frac{1}{1-\alpha }.$$

  And the test with level $\alpha$ is simply given by ${\phi }_{\alpha }=1_{x<\frac{1}{{\theta }_0}\log \frac{1}{1-\alpha }}$. Some magic is happening here, this test is optimal as it maximizes ${\mu }_1(\phi )$ among level $\le \alpha$, but is independent of ${\theta }_1$! (This is an example of Uniformly Most Powerful Test. An interesting question is when does this happen?)

• Consider a very simple random variable $X\sim \text{Bernoulli}(p)$, with $H_0:p=\frac{1}{2}$ and $H_1:p=\frac{1}{4}$. The likelihood ratio is

  $$L(x)=\{\begin{array}{ll}\frac{1}{2}& x=1\\ \frac{3}{2}& x=0.\end{array}$$

  Then clearly there are $5$ different regions of $k$ we can take to form different tests $\phi =\{\begin{array}{ll}1& L(x)>k\\ \gamma & L(x)=k\\ 0& L(x)<k\end{array}$

  $$[0,\frac{1}{2}),\left\{\frac{1}{2}\right\},(\frac{1}{2},\frac{3}{2}),\left\{\frac{3}{2}\right\},(\frac{3}{2},\mathrm{\infty }).$$

  The corresponding significant levels are

  $$\alpha ={\mu }_0({\phi }_{k,\gamma })=\{\begin{array}{ll}1& k\in [0,\frac{1}{2})\\ 1\cdot \gamma +\frac{1}{2}\cdot (1-\gamma )& k=\frac{1}{2}\\ \frac{1}{2}& k\in (\frac{1}{2},\frac{3}{2})\\ \frac{1}{2}\cdot \gamma +0\cdot (1-\gamma )& k=\frac{3}{2}\\ 0& k\in (\frac{3}{2},\mathrm{\infty })\end{array}.$$