ControlNet/docs/faq.md

FAQs

Q: If the weight of a conv layer is zero, the gradient will also be zero, and the network will not learn anything. Why "zero convolution" works?

A: This is wrong. Let us consider a very simple

$$y=wx+by=wx+b$$

and we have

$$\mathrm{\partial }y\mathrm{/}\mathrm{\partial }w=x,\mathrm{\partial }y\mathrm{/}\mathrm{\partial }x=w,\mathrm{\partial }y\mathrm{/}\mathrm{\partial }b=1\backslash partial\; y/\backslash partial\; w=x,\; \backslash partial\; y/\backslash partial\; x=w,\; \backslash partial\; y/\backslash partial\; b=1$$

and if $w=0$ and $x \neq 0$, then

$$\mathrm{\partial }y\mathrm{/}\mathrm{\partial }w\mathrm{\ne }0,\mathrm{\partial }y\mathrm{/}\mathrm{\partial }x=0,\mathrm{\partial }y\mathrm{/}\mathrm{\partial }b\mathrm{\ne }0\backslash partial\; y/\backslash partial\; w\; \backslash neq\; 0,\; \backslash partial\; y/\backslash partial\; x=0,\; \backslash partial\; y/\backslash partial\; b\backslash neq\; 0$$

which means as long as $x \neq 0$, one gradient descent iteration will make $w$ non-zero. Then

$$\mathrm{\partial }y\mathrm{/}\mathrm{\partial }x\mathrm{\ne }0\backslash partial\; y/\backslash partial\; x\backslash neq\; 0$$

so that the zero convolutions will progressively become a common conv layer with non-zero weights.