```
#hide
#skip
! [ -e /content ] && pip install -Uqq self-supervised
```

```
#default_exp vision.supcon
```

# SupCon

> **SupCon**: [Supervised Contrastive Learning](https://arxiv.org/pdf/2004.11362.pdf)

```
#export
_all_ = ['UnsupMethod']
```

```
#export
from fastai.vision.all import *
from self_supervised.augmentations import *
from self_supervised.layers import *
```

## Algorithm

#### SupCon

SupCon introduces a generalized form for contrastive losses, and shows that self-supervised loss from SimCLR, N-Pair loss and triplet margin loss are special cases. In this repo, we leverage this general form and formalize a loss that can transition from self-supervised contrastive loss to supervised contrastive loss:

$$
\begin{equation}
\begin{split}
\mathcal{L} & = \hspace{30mm} \mathcal{L}^{unsup} \hspace{16mm}+ \hspace{30mm} \lambda\mathcal{L}^{sup} \\
            \\
            & = - \sum_{i \in I} \log \frac{\exp \left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{j(i)} / \tau\right)}{\sum_{a \in A(i)} \exp \left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{a} / \tau\right)} + \lambda \sum_{i \in I} \frac{-1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp \left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{p} / \tau\right)}{\sum_{a \in A(i)} \exp \left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{a} / \tau\right)}
\end{split}
\end{equation}
$$

We use supervised signal as regularization which has an associated weight $\lambda$, this allows to pretrain with a dataset mixed with labelled and unlabelled data. This regularization can help to learn more generic features which in turn can help with the downstream task. In `SupCon` callback you can choose to use all samples (`UnsupMethod.All`) for unsupervised loss or only use the ones doesn't have a label (`UnsupMethod.All`). Supervised loss will use the samples with labels.

Therefore, positive samples come form two disjoint categories:

    (1) Other view of the anchor sample after augmentation (self-supervised case)
    (2) All views of the samples that have the same class id with anchor, including the other view of the same sample (supervised case)

*Note that self-supervised and unsupervised are used interchangably in this context*

SimCLR model consists of an `encoder` and a `projector (MLP)` layer. The definition of this module is fairly simple as below.

```
#export
class SupConModel(Module):
    "Compute predictions of concatenated xi and xj"
    def __init__(self,encoder,projector): self.encoder,self.projector = encoder,projector
    def forward(self,x): return self.projector(self.encoder(x))
```

Instead of directly using `SupConModel` by passing both an `encoder` and a `projector`, `create_simclr_model` function can be used by minimally passing a predefined `encoder` and the expected input channels.

```
#export
def create_supcon_model(encoder, hidden_size=256, projection_size=128, bn=False, nlayers=2):
    "Create SupCon model"
    n_in  = in_channels(encoder)
    with torch.no_grad(): representation = encoder(torch.randn((2,n_in,128,128)))
    projector = create_mlp_module(representation.size(1), hidden_size, projection_size, bn=bn, nlayers=nlayers)
    apply_init(projector)
    return SupConModel(encoder, projector)
```

You can use `self_supervised.layers` module to create an encoder. It supports all **timm** and **fastai** models available out of the box.

We define number of input channels with `n_in`, projector/mlp's hidden size with `hidden_size`,  projector/mlp's final projection size with `projection_size` and projector/mlp's number of layers with `nlayers`.

```
encoder = create_encoder("tf_efficientnet_b0_ns", n_in=3, pretrained=False, pool_type=PoolingType.CatAvgMax)
model = create_supcon_model(encoder, hidden_size=2048, projection_size=128, nlayers=2)
out = model(torch.randn((2,3,224,224))); out.shape
```

    torch.Size([2, 128])

## SupCon Callback

The following parameters can be passed;

- **aug_pipelines** list of augmentation pipelines List[Pipeline] created using functions from `self_supervised.augmentations` module. Each `Pipeline` should be set to `split_idx=0`. You can simply use `get_supcon_aug_pipelines` utility to get aug_pipelines.
- **temp** temperature scaling for cross entropy loss (defaults to paper's best value)

SupCon algorithm uses 2 views of a given image, and `SupCon` callback expects a list of 2 augmentation pipelines in `aug_pipelines`.

You can simply use helper function `get_supcon_aug_pipelines()` which will allow augmentation related arguments such as size, rotate, jitter...and will return a list of 2 pipelines, which then can be passed to the callback. This function uses `get_multi_aug_pipelines` which then `get_batch_augs`. For more information you may refer to `self_supervised.augmentations` module.

Also, you may choose to pass your own list of aug_pipelines which needs to be List[Pipeline, Pipeline] where Pipeline(..., split_idx=0). Here, `split_idx=0` forces augmentations to be applied in training mode.

```
#export
@delegates(get_multi_aug_pipelines)
def get_supcon_aug_pipelines(size, **kwargs): return get_multi_aug_pipelines(n=2, size=size, **kwargs)
```

```
aug_pipelines = get_supcon_aug_pipelines(size=28, rotate=False, jitter=False, bw=False, blur=False, stats=None, cuda=False)
aug_pipelines
```

    [Pipeline: RandomResizedCrop -> RandomHorizontalFlip,
     Pipeline: RandomResizedCrop -> RandomHorizontalFlip]

```
#export
mk_class('UnsupMethod', **{o:o.lower() for o in ['All', 'Only']},
         doc="Whether to use all (sup+unsup) or only unsup data in a batch for unsup loss")
```

```
#export
class SupCon(Callback):
    order,run_valid = 9,True
    def __init__(self, aug_pipelines, unsup_class_id, unsup_method=UnsupMethod.All,
                 reg_lambda=1.,
                 temp=0.07,
                 print_augs=False):
        assert_aug_pipelines(aug_pipelines)
        self.aug1, self.aug2 = aug_pipelines
        if print_augs: print(self.aug1), print(self.aug2)
        store_attr('unsup_class_id,unsup_method,reg_lambda,temp')

    def before_fit(self):
        self.learn.loss_func = self.lf

    def before_batch(self):
        xi,xj = self.aug1(self.x), self.aug2(self.x)
        self.learn.xb = (torch.cat([xi, xj]),)

    def _remove_diag(self, x):
        bs = x.shape[0]
        return x[~torch.eye(bs).bool()].reshape(bs,bs-1)

    def unsup_lf(self, pred, yb):
        "Self-Supervised contrasitve loss with all or only unlabelled batch samples"
        targ = torch.cat([yb,yb])
        unsup_mask = (targ == self.unsup_class_id)
        if self.unsup_method == UnsupMethod.All:
            pass
        elif self.unsup_method == UnsupMethod.Only:
            pred = pred[unsup_mask]
        else:
            raise Exception(f"{self.unsup_method} is not a valid UnsupMethod")

        if len(pred) == 0: return 0

        pred = F.normalize(pred, dim=1)
        bs = pred.shape[0]
        targ = torch.arange(bs, device=pred.device).roll(bs//2)
        sim  = self._remove_diag(pred @ pred.T) / self.temp
        targ = self._remove_diag(torch.eye(targ.shape[0], device=pred.device)[targ]).nonzero()[:,-1]
        return F.cross_entropy(sim, targ)

    def sup_lf(self, pred, yb):
        "Supervised contrasitve loss with labelled batch samples"
        targ = torch.cat([yb,yb])
        unsup_mask = (targ == self.unsup_class_id)
        pred = pred[~unsup_mask]
        targ = targ[~unsup_mask]

        if len(pred) == 0: return 0

        # exclude anchor from loss calc
        ohe_labels = (targ[...,None] == targ[None, ...]).float()
        pred = F.normalize(pred, dim=1)
        sim  = self._remove_diag(pred @ pred.T) / self.temp
        targ = self._remove_diag(ohe_labels)

        return (F.cross_entropy(sim, targ, reduction='none')/targ.sum(1)).mean()

    def lf(self, pred, *yb):
        unsup_loss = self.unsup_lf(pred, *yb)
        sup_loss   = self.sup_lf(pred, *yb)
        return unsup_loss + self.reg_lambda*sup_loss

    @torch.no_grad()
    def show(self, n=1):
        bs = self.learn.x.size(0)//2
        x1,x2  = self.learn.x[:bs], self.learn.x[bs:]
        idxs = np.random.choice(range(bs),n,False)
        x1 = self.aug1.decode(x1[idxs].to('cpu').clone()).clamp(0,1)
        x2 = self.aug2.decode(x2[idxs].to('cpu').clone()).clamp(0,1)
        images = []
        for i in range(n): images += [x1[i],x2[i]]
        return show_batch(x1[0], None, images, max_n=len(images), nrows=n)
```

### Tests

```
# No unsupervised, all samples are labelled, but use all for unsup loss
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="all")
yb = torch.tensor([1,1,2,2])
pred = torch.randn((yb.shape[0]*2,128))
```

```
loss1 = supcon.unsup_lf(pred, yb)
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
loss2 = torch.mean(tensor([nll[i,idx] for i, idx in enumerate([3,4,5,6,0,1,2,3])]))
assert torch.isclose(loss1,loss2)
```

```
loss1 = supcon.sup_lf(pred, yb)
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
ohe = supcon._remove_diag(tensor([[1,1,0,0,1,1,0,0],
                                 [1,1,0,0,1,1,0,0],
                                 [0,0,1,1,0,0,1,1],
                                 [0,0,1,1,0,0,1,1],
                                 [1,1,0,0,1,1,0,0],
                                 [1,1,0,0,1,1,0,0],
                                 [0,0,1,1,0,0,1,1],
                                 [0,0,1,1,0,0,1,1]]))
loss2 = (tensor([(row[idxs.bool()].sum()/idxs.sum()) for row, idxs in zip(nll, ohe)])).mean()
assert torch.isclose(loss1, loss2)
```

```
# No unsupervised, all samples are labelled, use only unlabelled for unsup loss
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="only")
yb = torch.tensor([1,1,2,2])
pred = torch.randn((yb.shape[0]*2,128))
```

```
loss1 = supcon.unsup_lf(pred, yb)
assert loss1 == 0
```

```
loss1 = supcon.sup_lf(pred, yb)
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
ohe = supcon._remove_diag(tensor([[1,1,0,0,1,1,0,0],
                                 [1,1,0,0,1,1,0,0],
                                 [0,0,1,1,0,0,1,1],
                                 [0,0,1,1,0,0,1,1],
                                 [1,1,0,0,1,1,0,0],
                                 [1,1,0,0,1,1,0,0],
                                 [0,0,1,1,0,0,1,1],
                                 [0,0,1,1,0,0,1,1]]))
loss2 = (tensor([(row[idxs.bool()].sum()/idxs.sum()) for row, idxs in zip(nll, ohe)])).mean()
assert torch.isclose(loss1,loss2)
```

```
# All samples are unlabelled, but use all for unsup loss
yb = torch.tensor([0,0,0,0])
pred = torch.randn((yb.shape[0]*2,128))
```

```
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="all")
loss1 = supcon.unsup_lf(pred, yb)
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="only")
loss2 = supcon.unsup_lf(pred, yb)
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
loss3 = torch.mean(tensor([nll[i,idx] for i, idx in enumerate([3,4,5,6,0,1,2,3])]))
assert torch.isclose(loss1, loss2) and torch.isclose(loss2, loss3)
```

```
loss1 = supcon.sup_lf(pred, yb)
assert loss1 == 0
```

```
# Mixed samples, but use all for unsup loss
yb = torch.tensor([1,1,2,0])
pred = torch.randn((yb.shape[0]*2,128))
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="all")
```

```
loss1 = supcon.unsup_lf(pred, yb)
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
loss2 = torch.mean(tensor([nll[i,idx] for i, idx in enumerate([3,4,5,6,0,1,2,3])]))
assert torch.isclose(loss1, loss2)
```

```
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="only")
loss1 = supcon.unsup_lf(pred, yb)
assert loss1 == 0 # log(1) -> 0, there is no negative sample
```

```
loss1 = supcon.sup_lf(pred, yb)
targ = torch.cat([yb,yb])
unsup_mask = (targ == supcon.unsup_class_id)
pred = pred[~unsup_mask]
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
ohe = supcon._remove_diag(tensor([[1,1,0,1,1,0],
                                  [1,1,0,1,1,0],
                                  [0,0,1,0,0,1],
                                  [1,1,0,1,1,0],
                                  [1,1,0,1,1,0],
                                  [0,0,1,0,0,1]]))

loss2 = (tensor([(row[idxs.bool()].sum()/idxs.sum()) for row, idxs in zip(nll, ohe)])).mean()
assert torch.isclose(loss1, loss2)
```

## SupConMOCO Callback [Experimental]

The following parameters can be passed;

- **aug_pipelines** list of augmentation pipelines List[Pipeline] created using functions from `self_supervised.augmentations` module. Each `Pipeline` should be set to `split_idx=0`. You can simply use `get_supcon_aug_pipelines` utility to get aug_pipelines.
- **temp** temperature scaling for cross entropy loss (defaults to paper's best value)

SupCon algorithm uses 2 views of a given image, and `SupCon` callback expects a list of 2 augmentation pipelines in `aug_pipelines`.

You can simply use helper function `get_supcon_aug_pipelines()` which will allow augmentation related arguments such as size, rotate, jitter...and will return a list of 2 pipelines, which then can be passed to the callback. This function uses `get_multi_aug_pipelines` which then `get_batch_augs`. For more information you may refer to `self_supervised.augmentations` module.

Also, you may choose to pass your own list of aug_pipelines which needs to be List[Pipeline, Pipeline] where Pipeline(..., split_idx=0). Here, `split_idx=0` forces augmentations to be applied in training mode.

```
#export
class SupConMOCO(Callback):
    order,run_valid = 9,True
    def __init__(self, aug_pipelines, unsup_class_id, unsup_method=UnsupMethod.All,
                 K=4096,
                 m=0.999,
                 reg_lambda=1.,
                 temp=0.07,
                 print_augs=False):
        assert_aug_pipelines(aug_pipelines)
        self.aug1, self.aug2 = aug_pipelines
        if print_augs: print(self.aug1), print(self.aug2)
        store_attr('unsup_class_id,unsup_method,K,m,reg_lambda,temp')

    def before_fit(self):
        "Create key encoder and init queue"
        if (not hasattr(self, "encoder_k")) and (not hasattr(self, "queue")):
            # init key encoder
            self.encoder_k = deepcopy(self.learn.model).to(self.dls.device)
            for param_k in self.encoder_k.parameters(): param_k.requires_grad = False

            # init queue
            nf = self.learn.model.projector[-1].out_features
            self.emb_queue = torch.randn(self.K, nf).to(self.dls.device)
            self.emb_queue = nn.functional.normalize(self.emb_queue, dim=1)
            self.label_queue = torch.zeros(self.K).to(self.dls.device) + self.unsup_class_id
            self.queue_ptr = 0
        else:
            warnings.warn("Key encoder and queue are already defined, keeping them.")

        self.learn.loss_func = self.lf

    def before_batch(self):
        "Generate query and key for the current batch"
        q_img,k_img = self.aug1(self.x), self.aug2(self.x.clone())
        self.learn.xb = (q_img,)
        with torch.no_grad():
            self.encoder_k.eval()
            self.learn.yb = (F.normalize(self.encoder_k(k_img)), self.y) # query and labels

    @torch.no_grad()
    def _momentum_update_key_encoder(self):
        for param_q, param_k in zip(self.learn.model.parameters(), self.encoder_k.parameters()):
            param_k.data = param_k.data * self.m + param_q.data * (1. - self.m)

    @torch.no_grad()
    def _dequeue_and_enqueue(self):
        bs = self.x.size(0)
        key_embs, key_labels = self.yb
        assert self.K % bs == 0  # for simplicity
        self.emb_queue[self.queue_ptr:self.queue_ptr+bs, :] = key_embs
        self.label_queue[self.queue_ptr:self.queue_ptr+bs] = key_labels
        self.queue_ptr = (self.queue_ptr + bs) % self.K  # move pointer

    def unsup_lf(self, pred, key_embs, key_labels):
        "Self-Supervised contrasitve loss with all or only unlabelled batch samples"
        query_embs = F.normalize(pred, dim=1)
        queue_embs, queue_labels = self.emb_queue, self.label_queue

        if self.unsup_method == UnsupMethod.All:
            key_embs = torch.cat([key_embs, queue_embs])

        elif self.unsup_method == UnsupMethod.Only:
            query_embs = query_embs[(key_labels == self.unsup_class_id)]
            key_embs   = key_embs[(key_labels == self.unsup_class_id)]
            queue_embs = queue_embs[(queue_labels == self.unsup_class_id)]

            key_embs   = torch.cat([key_embs, queue_embs])

        else:
            raise Exception(f"{self.unsup_method} is not a valid UnsupMethod")

        if len(query_embs) == 0: return 0

        labels = torch.arange(len(query_embs), device=pred.device)
        sim  = query_embs @ key_embs.T / self.temp
        return F.cross_entropy(sim, labels)

    def sup_lf(self, pred, key_embs, key_labels):
        "Supervised contrasitve loss with labelled batch samples"
        query_embs = F.normalize(pred, dim=1)
        queue_embs, queue_labels = self.emb_queue,  self.label_queue

        query_embs = query_embs[(key_labels != self.unsup_class_id)]
        key_embs   = key_embs[(key_labels != self.unsup_class_id)]
        queue_embs = queue_embs[(queue_labels != self.unsup_class_id)]

        key_embs   = torch.cat([key_embs, queue_embs])

        key_labels   = key_labels[(key_labels != self.unsup_class_id)]
        queue_labels = queue_labels[(queue_labels != self.unsup_class_id)]

        labels   = torch.cat([key_labels, queue_labels])

        if len(query_embs) == 0: return 0

        # exclude anchor from loss calc
        ohe_labels = (key_labels[...,None] == labels[None, ...]).float()
        sim  = query_embs @ key_embs.T / self.temp
        return (F.cross_entropy(sim, ohe_labels, reduction='none')/ohe_labels.sum(1)).mean()

    def lf(self, pred, *yb):
        unsup_loss = self.unsup_lf(pred, *yb)
        sup_loss   = self.sup_lf(pred, *yb)
        return unsup_loss + self.reg_lambda*sup_loss

    def after_step(self):
        "Update momentum (key) encoder and queue"
        self._momentum_update_key_encoder()
        self._dequeue_and_enqueue()

    @torch.no_grad()
    def show(self, n=1):
        bs = self.learn.x.size(0)//2
        x1,x2  = self.learn.x[:bs], self.learn.x[bs:]
        idxs = np.random.choice(range(bs),n,False)
        x1 = self.aug1.decode(x1[idxs].to('cpu').clone()).clamp(0,1)
        x2 = self.aug2.decode(x2[idxs].to('cpu').clone()).clamp(0,1)
        images = []
        for i in range(n): images += [x1[i],x2[i]]
        return show_batch(x1[0], None, images, max_n=len(images), nrows=n)
```

### Tests

```
from fastai.test_utils import *
```

```
class ContrastiveModel(Module):
    def __init__(self):
        self.encoder   = nn.Parameter(tensor([1.]))
        self.projector = nn.Linear(1,5, bias=False)
        self.projector.weight.data.zero_()
        self.projector.weight.data += 1
        self.projector = nn.Sequential(self.projector)

    def forward(self, x): return self.projector(x*self.encoder)
```

```
# No unsupervised, all samples are labelled, but use all for unsup loss
supcon = SupConMOCO([Pipeline([noop],0),Pipeline([noop],0)], unsup_class_id=0, unsup_method="all", K=8, m=0.999, reg_lambda=1.0, temp=0.07)
yb = torch.tensor([1,1,2,2])
pred = torch.randn((yb.shape[0]*2,128))
```

```
learner = synth_learner(cbs=supcon, data=synth_dbunch(a=0,b=0,bs=4), model=ContrastiveModel())
```

```
learner.sup_con_moco.aug1, learner.sup_con_moco.aug2
```

    (Pipeline: , Pipeline: )

```
learner.sup_con_moco.__dict__['__stored_args__']
```

    {'unsup_class_id': 0,
     'unsup_method': 'all',
     'K': 8,
     'm': 0.999,
     'reg_lambda': 1.0,
     'temp': 0.07}

```
learner('before_fit')
```

```
assert learner.sup_con_moco.emb_queue.shape == (8,5)
assert torch.all(learner.sup_con_moco.label_queue == torch.zeros(8))
assert not any(list(o.requires_grad for o in learner.sup_con_moco.encoder_k.parameters()))
assert torch.all(learner.sup_con_moco.encoder_k.projector[0].weight == 1)
```

```
b = tensor([1,1,-1,1]).reshape(-1,1),tensor([1,1,2,2])
learner._split(b)
learner('before_batch')
```

```
key_embs, labels = learner.sup_con_moco.yb
assert torch.equal(F.normalize(learner.sup_con_moco.encoder_k(b[0])), key_embs)
assert torch.equal(labels, b[1])
```

```
learner.model.encoder.data += 0.1 # pseudo param update 1.0 -> 1.1
learner.model.projector[0].weight.data += 0.1 # pseudo param update 1.0 -> 1.1
```

```
learner('after_step')
```

```
assert torch.equal(learner.sup_con_moco.emb_queue[:4], key_embs)
```

```
newval = 1*supcon.m + 1.1*(1-supcon.m)
assert torch.all(learner.sup_con_moco.encoder_k.encoder.data == newval) and torch.all(learner.sup_con_moco.encoder_k.projector[0].weight.data==newval)
```

```
b = tensor([-1,-1,1,-1]).reshape(-1,1),tensor([1,1,2,2])
learner._split(b)
learner('before_batch')
```

```
key_embs, labels = learner.sup_con_moco.yb
assert torch.equal(F.normalize(learner.sup_con_moco.encoder_k(b[0])), key_embs)
assert torch.equal(labels, b[1])
```

```
learner('after_step')
```

```
assert torch.equal(learner.sup_con_moco.emb_queue[-4:], key_embs)
assert torch.equal(learner.sup_con_moco.label_queue, tensor([1,1,2,2,1,1,2,2]).float())
```

```
newval = newval*supcon.m + 1.1*(1-supcon.m)
assert torch.all(learner.sup_con_moco.encoder_k.encoder.data == newval) and torch.all(learner.sup_con_moco.encoder_k.projector[0].weight.data==newval)
```

```
pred = F.normalize(learner.model(learner.x))
loss1 = learner.sup_con_moco.unsup_lf(pred, *learner.yb)
key_embs, labels = learner.yb
logits = pred @ torch.cat([key_embs,learner.sup_con_moco.emb_queue]).T / learner.sup_con_moco.temp
loss2 = F.cross_entropy(logits, tensor([0,1,2,3]))
assert loss1 == loss2
```

```
learner.sup_con_moco.unsup_method = UnsupMethod.Only
learner.sup_con_moco.unsup_class_id = 1
loss1 = learner.sup_con_moco.unsup_lf(pred, *learner.yb)
logits = pred[labels==1] @ torch.cat([key_embs[labels==1],learner.sup_con_moco.emb_queue[learner.sup_con_moco.label_queue==1]]).T / learner.sup_con_moco.temp
loss2 = F.cross_entropy(logits, tensor([0,1]))
assert loss1 == loss2
```

```
learner.sup_con_moco.unsup_class_id = 0
pred = F.normalize(learner.model(learner.x))
loss1 = learner.sup_con_moco.sup_lf(pred, *learner.yb)
logits = pred @ torch.cat([key_embs,learner.sup_con_moco.emb_queue]).T / learner.sup_con_moco.temp
ohe_labels = tensor([[1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1,],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1]])
loss2 = (F.cross_entropy(logits, ohe_labels, reduction='none') / ohe_labels.sum(1)).mean()
assert loss1 == loss2
```

```
learner.sup_con_moco.unsup_class_id = 2
pred = F.normalize(learner.model(learner.x))
loss1 = learner.sup_con_moco.sup_lf(pred, *learner.yb)
logits = pred[labels != 2] @ torch.cat([key_embs[labels != 2],learner.sup_con_moco.emb_queue[learner.sup_con_moco.label_queue != 2]]).T / learner.sup_con_moco.temp
ohe_labels = tensor([[1., 1., 1., 1., 1., 1.],
                     [1., 1., 1., 1., 1., 1.]])
loss2 = (F.cross_entropy(logits, ohe_labels, reduction='none') / ohe_labels.sum(1)).mean()
assert loss1 == loss2
```

```
key_embs, labels = learner.yb
learner.sup_con_moco.unsup_class_id = 3
learner.xb = (torch.cat([learner.x, tensor([[0]])]),)
key_embs, labels = torch.cat([learner.y[0], learner.y[0][:1]]), torch.cat([learner.y[1],  tensor([3])])
learner.yb = (key_embs, labels)
pred = F.normalize(learner.model(learner.x))
loss1 = learner.sup_con_moco.sup_lf(pred, *learner.yb)
logits = pred[labels != 3] @ torch.cat([key_embs[labels != 3],learner.sup_con_moco.emb_queue[learner.sup_con_moco.label_queue != 3]]).T / learner.sup_con_moco.temp
ohe_labels = tensor([[1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1,],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1]])
loss2 = (F.cross_entropy(logits, ohe_labels, reduction='none') / ohe_labels.sum(1)).mean()
assert loss1 == loss2
```

```
key_embs, labels = learner.yb
learner.sup_con_moco.unsup_class_id = 3

learner.xb = (torch.cat([learner.x[:2],tensor([[0]]), learner.x[2:]]),)
key_embs, labels = torch.cat([learner.y[0][:2], learner.y[0][:1], learner.y[0][2:]]), torch.cat([learner.y[1][:2],  tensor([3]), learner.y[1][2:]])

learner.yb = (key_embs, labels)
pred = F.normalize(learner.model(learner.x))

loss1 = learner.sup_con_moco.sup_lf(pred, *learner.yb)
logits = pred[labels != 3] @ torch.cat([key_embs[labels != 3],learner.sup_con_moco.emb_queue[learner.sup_con_moco.label_queue != 3]]).T / learner.sup_con_moco.temp
ohe_labels = tensor([[1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1,],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1]])
loss2 = (F.cross_entropy(logits, ohe_labels, reduction='none') / ohe_labels.sum(1)).mean()
assert loss1 == loss2
```

### Example Usage

```
path = untar_data(URLs.IMAGEWANG_160)
items = get_image_files(path)
```

```
items = np.random.choice(items, size=1000)
```

```
tds = Datasets(items, [[PILImage.create, ToTensor, RandomResizedCrop(112, min_scale=1.)],
                       [parent_label, Categorize()]], splits=RandomSplitter()(items))
dls = tds.dataloaders(bs=5, after_item=[ToTensor(), IntToFloatTensor()], device='cpu')
```

```
unsup_class_id = dls.vocab.o2i['unsup']
```

```
fastai_encoder = create_encoder('xresnet18', n_in=3, pretrained=False)
model = create_supcon_model(fastai_encoder, hidden_size=2048, projection_size=128)
aug_pipelines = get_supcon_aug_pipelines(size=28, rotate=False, jitter=False, bw=False, blur=False, stats=None, cuda=False)
learn = Learner(dls, model, cbs=[SupCon(aug_pipelines,
                                        unsup_class_id,
                                        unsup_method=UnsupMethod.All, reg_lambda=1.0, temp=0.07,
                                        print_augs=True),ShortEpochCallback(0.001)])
```

    Pipeline: RandomResizedCrop -> RandomHorizontalFlip
    Pipeline: RandomResizedCrop -> RandomHorizontalFlip

Also, with `show_one()` method you can inspect data augmentations as a sanity check. You can use existing augmentation functions from `augmentations` module.

```
b = dls.one_batch()
learn._split(b)
learn('before_batch')
axes = learn.sup_con.show(n=5)
```

```
learn.fit(1)
```

<style>
    /* Turns off some styling */
    progress {
        /* gets rid of default border in Firefox and Opera. */
        border: none;
        /* Needs to be in here for Safari polyfill so background images work as expected. */
        background-size: auto;
    }
    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {
        background: #F44336;
    }
</style>

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>epoch</th>
      <th>train_loss</th>
      <th>valid_loss</th>
      <th>time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>0</td>
      <td>00:02</td>
    </tr>
  </tbody>
</table>

```
learn.recorder.losses
```

    [TensorCategory(1.7556)]

```
fastai_encoder = create_encoder('xresnet18', n_in=3, pretrained=False)
model = create_supcon_model(fastai_encoder, hidden_size=2048, projection_size=128)
aug_pipelines = get_supcon_aug_pipelines(size=28, rotate=False, jitter=False, bw=False, blur=False, stats=None, cuda=False)
learn = Learner(dls, model, cbs=[SupConMOCO(aug_pipelines,
                                            unsup_class_id,
                                            unsup_method=UnsupMethod.All, K=25, reg_lambda=1.0, temp=0.07,
                                            print_augs=True),ShortEpochCallback(0.001)])
```

    Pipeline: RandomResizedCrop -> RandomHorizontalFlip
    Pipeline: RandomResizedCrop -> RandomHorizontalFlip

```
learn.fit(1)
```

<style>
    /* Turns off some styling */
    progress {
        /* gets rid of default border in Firefox and Opera. */
        border: none;
        /* Needs to be in here for Safari polyfill so background images work as expected. */
        background-size: auto;
    }
    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {
        background: #F44336;
    }
</style>

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>epoch</th>
      <th>train_loss</th>
      <th>valid_loss</th>
      <th>time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>0</td>
      <td>00:06</td>
    </tr>
  </tbody>
</table>

## Export -

```
#hide
from nbdev.export import notebook2script
notebook2script()
```

    Converted 01 - augmentations.ipynb.
    Converted 02 - layers.ipynb.
    Converted 03 - distributed.ipynb.
    Converted 10 - simclr.ipynb.
    Converted 11 - moco.ipynb.
    Converted 12 - byol.ipynb.
    Converted 13 - swav.ipynb.
    Converted 14 - barlow_twins.ipynb.
    Converted 15 - dino.ipynb.
    Converted 16 - supcon.ipynb.
    Converted 20 - clip.ipynb.
    Converted 21 - clip-moco.ipynb.
    Converted 70 - vision.metrics.ipynb.
    Converted 90 - models.vision_transformer.ipynb.
    Converted index.ipynb.

```

```


Sveučilište u Zagrebu

Fakultet elektrotehnike i računarstva

# Strojno učenje

http://www.fer.unizg.hr/predmet/su

Ak. god. 2015./2016.

# Bilježnica 8: Stroj potpornih vektora (SVM)

(c) 2015 Jan Šnajder

Verzija: 0.3 (2015-12-11)

```python
import scipy as sp
import scipy.stats as stats
import matplotlib.pyplot as plt
import pandas as pd
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

### Sadržaj:

* Uvod

* Problem maksimalne margine

* Optimizacija uz ograničenja

* Metoda Lagrangeovih multiplikatora

* Dualna formulacija problema maksimalne margine

* Meka margina

* Gubitak zglobnice

* Jezgreni trik

* Mercerove jezgre

* Optimizacija hiperparametara

# Uvod

* Vrlo učinkovit **diskriminativan model**

* Podsjetnik: Algoritam strojnog učenja definiran je
  1. modelom
  * pogreškom
  * optimizacijskim postupkom

* Model:

$$
    h(\mathbf{x}) = \mathbf{w}^\intercal\boldsymbol{\phi}(\mathbf{x})
$$

* Dakle, to je poopćeni linearan model bez aktivacijske funkcije $f$

* Gornja definicija modela je tzv. **primarna formulacija**

* Postoji i **dualna formulacija**:
  * Umjesto da značajke novog primjera množimo težinama $\mathbf{w}$, možemo za novi primjer izračunati koliko je sličan primjerima iz skupa za učenje i na temelju toga odrediti klasifikaciju
  * Model efektivno postaje **neparametarski**!

* U dualnoj formulaciji možemo iskoristiti tzv. **jezgreni trik**, koji nam omogućava jeftino preslikavanje primjera u prostor više dimenzije (a time i nelinearnost modela)
  * SVM je jedna vrsta tzv. **jezgrenog stroja** (engl. *kernel machine*)

* Model je jednostavan, no definicija pogreške i optimizacijskog postupka su nešto složeniji

* Osnovna ideja: primjere dviju klasa razdvojiti tako da je prostor između njih što veći $\Rightarrow$ tzv. **maksimalna margina**

* Opravdanje: generalizacija će biti najbolja onda kada granicu između klasa povućemo točno po sredini margine

* Do sada smo optimizacijski problem definirali tako da smo
  * krenuli od log-izglednosti pa izveli MLE (naivan Bayes)
  * krenuli od funkcije gubitka pa izveli funkciju pogreške (linearna regresija) i napravili analitičku minimizaciju
  * krenuli od funkcije pogreške pa izveli funkciju gubitka (logistička regresija, perceptron) i iskoristili je za gradijentni spust

* Kod SVM-a, krenut ćemo odmah od onoga što u konačnici želimo dobiti: maksimalnu marginu $\Rightarrow$ to je ono što SVM optimizira

* Ići ćemo "klasičnim pristupom": formalizirati to kao problem **kvadratnog programiranja**

* Naknadno ćemo iz toga izvesti funkciju gubitka (i funkciju pogreške), ali samo radi usporedbe s drugim algoritmima

> **Sinopsis:**
>
> * Prvo ćemo se fokusirati na **linearan model** i **linearno odvojive probleme** (tvrda granica)
  * Matematika: Lagrangeovi multiplikatori
>
> * Zatim ćemo proširiti **linearan model** tako da može raditi s **linearno neodvojivim problemima** (meka granica)
>
> * Na kraju ćemo proširiti na **nelinearan model** (jezgreni trik)
>   * Matematika: Mercerove jezgre

# Problem maksimalne margine

* Model:
$$
h(\mathbf{x}) = \mathbf{w}^\intercal x + w_0
$$

* Oznake primjera za učenje: $y\in\{-1,+1\}$

* Granica između klasa: hiperravnina $h(\mathbf{x})=0$

* Predikcija klase: $y=\mathrm{sgn}(h(\mathbf{x}))$

* Pretpostavimo da su primjeri iz $\mathcal{D}$ **linearno odvojivi**

* Onda postoji $\mathbf{w}$ i $w_0$ takvi da
$$
\begin{align*}
h(\mathbf{x}^{(i)}) \geq 0 & \quad\text{za svaki $y^{(i)}=+1$}\\
h(\mathbf{x}^{(i)}) < 0 & \quad\text{za svaki $y^{(i)}=-1$}\\
\end{align*}
$$

* Kraće, postoje $\mathbf{w}$ i $w_0$ takvi da
$$
\forall(\mathbf{x}^{(i)},y^{(i)}) \in \mathcal{D}.\ y^{(i)}h(\mathbf{x}^{(i)})\geq 0
$$

* Postoji beskonačno mnogo rješenja za $\mathbf{w}$ i $w_0$ (prostor inačica je beskonačan)

* No nas zanima rješenje **maksimalne margine** $\Rightarrow$ induktivna pristranost preferencijom

* **Margina = udaljenost hiperravnine do najbližeg primjera**

* Ako maksimiziramo marginu, onda će hiperravnina prolaziti točno na pola puta između dva primjera

#### Formulacija optimizacijskog problema

* Predznačena udaljenost primjera od hiperravnine je

$$
d = \frac{h(\mathbf{x})}{\|\mathbf{w}\|}
$$

* Nas zanimaju samo hiperravnine koje ispravno klasificiraju primjere. U tom slučaju **apsolutna** udaljenost primjera do hiperravnine je:

$$
\frac{y^{(i)}h(\mathbf{x})}{\|\mathbf{w}\|} =
\frac{y^{(i)}(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0)}{\|\mathbf{w}\|}
$$

* Po definiciji, margina je udaljenost hiperravnine do najbližeg primjera:

$$
\frac{1}{\|\mathbf{w}\|}\mathrm{min}_i\big\{y^{(i)}(\mathbf{w}^\intercal\mathbf{x} + w_0)\big\}
$$

* Tu udaljenost želimo maksimizirati:

$$
\mathrm{argmax}_{\mathbf{w},w_0}\Big\{\frac{1}{\|\mathbf{w}\|}\mathrm{min}_i\big\{y^{(i)}(\mathbf{w}^\intercal\mathbf{x} + w_0)\big\}\Big\}
$$

* Ako je $\mathcal{D}$ linearno odvojiv, onda postoji samo jedna takva margina

#### Pojednostavljenje optimizacijskog problema

* Gornji problem teško je riješiti izravno (min unutar max)

* Vektor $(\mathbf{w},w_0)$ možemo pomnožiti s proizvoljnom konstantom, a da to ne utječe na udaljenosti između primjera i hiperravnine:

$$
d=\frac{h(\mathbf{x})}{\|\mathbf{w}\|}=
\frac{\color{red}{\alpha}\mathbf{w}^\intercal\mathbf{x}+\color{red}{\alpha}w_0}{\|\color{red}{\alpha}\mathbf{w}\|}=
\frac{\color{red}{\alpha}(\mathbf{w}^\intercal\mathbf{x}+w_0)}{\color{red}{\alpha}\|\mathbf{w}\|}=
\frac{\mathbf{w}^\intercal\mathbf{x}+w_0}{\|\mathbf{w}\|}
$$

* Kako bismo pojednostavili problem, možemo definirati da za primjer $\mathbf{x}^{(i)}$ koji je najbliži margini vrijedi

$$
y^{(i)}(\mathbf{w}^\intercal\mathbf{x}+w_0)=1
$$

* Svi ostali primjeri jednako su blizu margine ili su od nje još udaljeniji:

$$
y^{(i)}(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0) \geq 1, \qquad i=1,\dots,N
$$

* Za primjere za koje $y^{(i)}h(\mathbf{x})=1$ kažemo da su ograničenja **aktivna**, dok su za ostale primjere ograničenja neaktivna
  * (Primjeri za koje su ograničenja aktivna zovemo potpornim vektorima, ali o tome više kasnije)

* Uvijek će postojati barem **dva** aktivna ograničenja

* [Skica: maksimalna margina]

* Dakle, umjesto
$$
\mathrm{argmax}_{\mathbf{w},w_0}\Big\{\frac{1}{\|\mathbf{w}\|}\underbrace{\mathrm{min}_i\big\{y^{(i)}(\mathbf{w}^\intercal\mathbf{x} + w_0)\big\}}_{=1}\Big\}
$$
mi sada maksimiziramo
$$
\mathrm{argmax}_{\mathbf{w},w_0}\frac{1}{\|\mathbf{w}\|}
$$
uz ograničenja
$$
y^{(i)}(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0) \geq 1, \qquad i=1,\dots,N
$$

* Maksimizator od $\frac{1}{\|\mathbf{w}\|}$ ekvivalentan je minimizatoru od $\|\mathbf{w}\|=\sqrt{\mathbf{w}^\intercal\mathbf{w}}$, a taj je ekvivalentan minimizatoru od $\|\mathbf{w}\|^2$. Još ćemo pomnožiti s $\frac{1}{2}$ radi kasnije matematičke jednostavnosti

* Konačna formulacija optimizacijskog problema maksimalne margine:

> $\mathrm{argmin}_{\mathbf{w},w_0}\frac{1}{2}\|\mathbf{w}\|^2$

> tako da $\quad y^{(i)}(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0) \geq 1, \quad i=1,\dots,N$

* Naš optimizacijski problem sveo se na ciljnu funkciju koju želimo optimirati i ograničenja koja pritom moramo poštovati: tipičan problem **konveksne optimizacije uz ograničenja**, točnije **kvadratnog programiranja**
  * "Programiranje" = matematičko programiranje = (matematička) optimizacija

# Optimizacija uz ograničenja

* Optimizacijski problem $\Rightarrow$ Optimizacija uz ograničenja $\Rightarrow$ Konveksni optimizacijski problem $\Rightarrow$ Kvadratno programiranje

* Optimizacijski problem uz ograničenja (standardni oblik):

$$
\begin{align*}
\text{minimizirati} &\quad f(\mathbf{x})\\
\text{uz ograničenja} &\quad g_i(\mathbf{x})\leq0,\quad i=1,\dots,m\\
&\quad h_i(\mathbf{x}) = 0,\quad i=1,\dots,p
\end{align*}
$$

* $f : \mathbb{R}^n\to\mathbb{R}$ je **ciljna funkcija** (engl. *objective function*
* $h_i : \mathbb{R}^n\to\mathbb{R}$ su **ograničenja
jednakosti** (engl. *equality constraints*)
* $g_i:\mathbb{R}^n\to\mathbb{R}$ su **ograničenja nejednakosti** (engl. *inequality constraints*)

* **NB:** Sva ograničenja (ne)jednakosti mogu se se svesti na standardni oblik

* Tražimo minimum koji zadovoljava sva ograničenja
* Točke koje zadovoljavaju rješenja zovemo **ostvarivim
točkama** (engl. *feasible points*)

* Konveksni optimizacijski problem:

$$
\begin{align*}
\text{minimizirati} &\quad f(\mathbf{x})\ \color{red}{\text{$\Rightarrow$ konveksna}}\\
\text{uz ograničenja} &\quad g_i(\mathbf{x})\leq0,\quad i=1,\dots,m\\
&\quad \mathbf{a}_i^\intercal\mathbf{x} - b_i = 0,\quad i=1,\dots,p
\end{align*}
$$

* Kvadratni program:

$$
\begin{align*}
\text{minimizirati} &\quad \frac{1}{2}\mathbf{x}^\intercal P\mathbf{x} + \mathbf{q}^\intercal\mathbf{x} + r\ \color{red} {\text{$\Rightarrow$ kvadratna funkcija}}\\
\text{uz ograničenja} &\quad G\mathbf{x}\leq\mathbf{h}\\
&\quad A\mathbf{x}=\mathbf{b}\\
\end{align*}
$$

# Metoda Lagrangeovih multiplikatora

* Kvadratni program može se riješiti metodom **Langrangeovih multiplikatora**

* Lagrangeovi multiplikatori mogu se koristiti općenito za optimizaciju s ograničenjima (problem ne mora biti konveksan)

* Ideja: preformulirati ograničen problem tako da se u ciljnu funkciju ugrade ograničenja

* Alternative: **metode unutarnje točke** (engl. *interior-point methods*, *barrier methods*) ili **metode s kaznom** (engl. *penalty methods*)

#### Lagrangeova funkcija

* **Lagrangeova funkcija** izravno ugrađuje ograničenja u ciljnu funkciju

* [Skica ideje]

* Početni problem:

\begin{align*}
\text{minimizirati} &\quad f(\mathbf{x})\\
\text{uz ograničenja} &\quad g_i(\mathbf{x})\leq0,\quad i=1,\dots,m\\
 &\quad h_i(\mathbf{x}) = 0,\quad i=1,\dots,p
\end{align*}

* Lagrangeova funkcija:

$$
\begin{equation}
L(\mathbf{x},\color{red}{\boldsymbol\alpha},\color{red}{\boldsymbol\beta}) = f(\mathbf{x}) + \sum_{i=1}^m\color{red}{\alpha_i} g(\mathbf{x}) + \sum_{i=1}^p\color{red}{\beta_i}
h(\mathbf{x})
\end{equation}
$$

* Dobili smo neograničeni problem optimizacije: rješenje izvornoga ograničenog problema jest točka koja mimimizira $\mathbf{x}$ i maksimizira $\boldsymbol\alpha$ (sedlo)

* Vrijednosti $\alpha_i$ i $\beta_i$ su **Lagrangeovi multiplikatori** (množitelji) za ograničenja nejednakosti odnosno jednakosti

* Za vrijednosti $\alpha_i$ vrijede takozvani **Karush-Kuhn-Tuckerovi (KKT)** uvjeti:

$$
\begin{align}
\alpha_i &\geq 0,\quad i=1,\dots,m\\
\alpha_i g_i(\mathbf{x}) &= 0,\quad i=1,\dots,m
\end{align}
$$

#### Lagrangeova dualnost

* Načelo dualnosti u teoriji optimizacije:
  * **Primarni problem** (engl. *primal problem*): minimizacija funkcije $f(\mathbf{x})$
  * **Dualni problem**: nalaženje donje granice primarnog problema ("minimum ne može biti manji od $\mathbf{x}$")

* [Skica: primal-dual sedlo]

* Općenito, rješenja primarnog i dualnog problema se ne preklapaju već postoji **dualni procjep**

* Uz određene uvjete, kod konveksne optimizacije dualni procjep jednak je nuli $\Rightarrow$ **jaka dualnost**

* To znači da je rješenje dualnog problema ujedno i rješenje primarnog problema

* Dakle, ako nam je tako pogodnije, možemo rješavati dualni problem umjesto primarnog

* U nastavku promatramo Lagrangeovu dualnost

* Lagrangeova funkcija (BSO: samo s ograničenjima jednakosti):

$$
L(\mathbf{x},\boldsymbol\alpha) = f(\mathbf{x}) + \sum_i\alpha_i h_i(\mathbf{x})
$$

* To je fukcija od $\mathbf{x}$ (**primarne varijable**) i $\boldsymbol\alpha$ (**dualne varijable**)

* Optimum:
$$
L(\mathbf{x}^*,\boldsymbol\alpha^*) = \min_{\mathbf{x},\boldsymbol\alpha} L(\mathbf{x},\boldsymbol\alpha)
$$

* Vrijednost za $\mathbf{x}^*$ nalazimo rješavanjem sustava:

$$
\begin{equation}
\nabla f(\mathbf{x}) + \nabla \sum_i\alpha_i h_i(\mathbf{x}) = 0
\end{equation}
$$

* To rješenje ne mora dovesti do uklanjanja dualnih varijabli! Općenito, dobit ćemo rješenje koje minimizira $\mathbf{x}$ za neki zadani $\boldsymbol\alpha$:
$$
\tilde{L}(\boldsymbol\alpha)= \min_{\mathbf{x}}L(\mathbf{x},\boldsymbol\alpha) = \min_{\mathbf{x}}\Big(f(\mathbf{x}) + \sum_i
\alpha_i h(\mathbf{x})\Big)
$$
$\Rightarrow$ **Dualna Lagrangeova funkcija**

* Sigurno vrijedi:
$$
\tilde{L}(\boldsymbol\alpha) \leq L(\mathbf{x}^*,\boldsymbol\alpha)
$$
$\Rightarrow$ Dualna Lagrangeova funkcija je **donja ograda** primarnog problema

* U točki $\boldsymbol\alpha^*$ vrijedi $\tilde{L}(\boldsymbol\alpha^*)=L(\mathbf{x}^*,\boldsymbol\alpha^*)$

* Kako bismo pronašli $\boldsymbol\alpha^*$, moramo maksimizirati
donju ogradu, tj. riješiti sljedeći konveksni problem:

$$
\begin{align*}
\text{maksimizirati} &\quad \tilde{L}(\boldsymbol\alpha)\\
\text{uz ograničenja} &\quad \alpha_i\geq 0,\quad i=1,\dots,p
\end{align*}
$$

* **NB:** minimizacija ciljne funkcije $\Leftrightarrow$ maksimizacija dualne funkcije

# Dualna formulacija problema maksimalne margine

* Optimizacijski problem maksimalne margine:

> $\mathrm{argmin}_{\mathbf{w},w_0}\frac{1}{2}\|\mathbf{w}\|^2$

> tako da $\quad y^{(i)}(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0) \geq 1, \quad i=1,\dots,N$

* Odgovarajuća Lagrangeova funkcija:

$$
L(\mathbf{w},w_0,\color{red}{\boldsymbol\alpha})=\frac{1}{2}\|\mathbf{w}\|^2 -
\sum_{i=1}^N\color{red}{\alpha_i}\Big\{y^{(i)}\big(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0\big)-1\Big\}
$$

*  $\color{red}{\boldsymbol\alpha=(\alpha_1,\dots,\alpha_N)}$ je vektor Lagrangeovih multiplikatora, po jedan za svako ograničenje

* Prelazimo na **dualnu formulaciju** problema jer je ta formulacija jednostavnija (optimirat ćemo samo po $\boldsymbol\alpha$, a postoje i neke druge prednosti)

* Minimizator $(\mathbf{w}^*, w_0^*)$: deriviranje po $\mathbf{w}$ odnosno $w_0$ i izjednačavanje s nulom:
$$
\begin{align}
\mathbf{w} &= \sum_{i=1}^N \alpha_i y^{(i)}\mathbf{x}^{(i)}\\
0 &= \sum_{i=1}^N\alpha_i y^{(i)}
\end{align}
$$

* Dualna Lagrangeova funkcija:

$$
\begin{align*}
\tilde{L}(\boldsymbol\alpha) &=
\frac{1}{2}\|\mathbf{w}\|^2 -\sum_{i=1}^N\alpha_i\Big\{y^{(i)}\big(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0\big)-1\Big\}\\
&=
\frac{1}{2}\|\mathbf{w}\|^2
-\sum_{i=1}^N\alpha_i y^{(i)}\mathbf{w}^\intercal\mathbf{x}^{(i)}
\color{gray}{\underbrace{-w_0 \sum_{i=1}^N\alpha_iy^{(i)}}_{=0}} +
\sum_{i=1}^N\alpha_i\\
&= \nonumber
\frac{1}{2}\sum_{i=1}^N\alpha_i y^{(i)}(\mathbf{x}^{(i)})^\intercal\sum_{j=1}^N\alpha_j y^{(j)}\mathbf{x}^{(j)}
-
\sum_{i=1}^N\alpha_i y^{(i)}(\mathbf{x}^{(i)})^\intercal\sum_{j=1}^N\alpha_j y^{(j)}\mathbf{x}^{(j)}
+ \sum_{i=1}^N\alpha_i\\
&=
\sum_{i=1}^N\alpha_i -
\frac{1}{2}\sum_{i=1}^N
\sum_{j=1}^N
\alpha_i
\alpha_j
y^{(i)}
y^{(j)}
(\mathbf{x}^{(i)})^\intercal
\mathbf{x}^{(j)}
\end{align*}
$$

* Dobili smo **dualni optimizacijski problem**:

> **Maksimizirati** izraz
> \begin{align}
\sum_{i=1}^N\alpha_i -
\frac{1}{2}\sum_{i=1}^N
\sum_{j=1}^N
\alpha_i
\alpha_j
y^{(i)}
y^{(j)}
(\mathbf{x}^{(i)})^\intercal
\mathbf{x}^{(j)}
\end{align}
> tako da:
> \begin{align*}
\alpha_i &\geq 0,\quad i=1,\dots,N \\
\sum_{i=1}^N\alpha_i y^{(i)} &= 0
\end{align*}

* Također vrijedi KKT-uvjet:
$$
\alpha_i\big(y^{(i)} h(\mathbf{x}^{(i)})-1\big) = 0
$$

* Ovo je i dalje problem kvadratnog programiranja, međutim broj varijabli se promijenio
  * Primarni problem: $n+1$ varijabli
  * Dualni problem: $N$ varijabli

* Dakle, prijelaz u dualni problem se računalno isplati ako $N\ll n$

* Općenito, složenost kvadratnog programiranja od $n$ varijabli je $\mathcal{O}(n^3)$
  * Posebni algoritmi su efikasniji: **slijedna minimalna optimizacija (SMO)** ima $\mathcal{O}(n^2)$

#### Model

* Već smo izračunali:
$$
\begin{align}
\mathbf{w} &= \sum_{i=1}^N \alpha_i y^{(i)}\mathbf{x}^{(i)}
\end{align}
$$

* Uvrštavanjem:
$$
\begin{equation}
h(\mathbf{x})=
\underbrace{\mathbf{w}^\intercal\mathbf{x}+w_0}_{\text{Primarno}} =
\underbrace{\sum_{i=1}^N \alpha_i y^{(i)}\mathbf{x}^\intercal\mathbf{x}^{(i)} + w_0}_{\text{Dualno}}
\end{equation}
$$

* Kako bismo klasificirali primjer $\mathbf{x}$, računamo skalarni produkt između $\mathbf{x}$ i svih primjera $\mathbf{x}^{(i)}$ iz skupa $\mathcal{D}$, pomnožen s težinom $\alpha_i$ i predznakom $\mathcal{y}^(i)$

* Računanje skalarnog produkta $\mathbf{x}^\intercal\mathbf{x}^{(i)}$ je zapravo računanje sličnosti između vektora $\mathbf{x}$ i $\mathbf{x}^{(i)}$, budući da:
$$
\mathbf{x}^\intercal\mathbf{y} = \sum_{i=1}^n x_i y_i
$$
(produkt će biti to veći što se vektori podudaraju u više komponenenata)

* Dakle, umjesto da pohranjujemo težine $\mathbf{w}$, trebamo pohraniti primjere i njihove oznake
  * Umjesto $h(\mathbf{x}|\mathbf{w})$ imamo $h(\mathbf{x}|\boldsymbol\alpha,\mathcal{D})$

* Složenost modela sada ovisi o broju primjera $\Rightarrow$ **neparametarski model**

* Zaključak: ako se trenira tako da se rješava primarni problem, SVM je parametarski model, a ako se rješava dualni problem, onda je neparametarski

#### Potporni vektori

* Iz KKT-uvjeta
$$
\alpha_i\big(y^{(i)} h(\mathbf{x}^{(i)})-1\big) = 0
$$
slijedi da za svaki primjer $\mathbf{x}^{(i)}$ iz $\mathcal{D}$ vrijedi
$$
\alpha_i=0
$$
ili
$$
y^{(i)}h(\mathbf{x}^{(i)})=1
$$

* To znači da je se u izrazu za model pojavljuju samo vektori koji leže točno na ravnini maksimalne margine $\Rightarrow$ **potporni vektori** (engl. *support vectors*)

* Svi ostali vektori za koje $\alpha_i=0$ uopće ne utječu na izlaz modela i možemo ih zanemariti kada radimo predikciju

* Alternativni pogled: hiperravnina (u primarnom problemu) definirana je linearnom kombinacijom potpornih vektora (u dualnom problemu)

```python
seven_X = sp.array([[2,1], [2,3], [1,2], [3,2], [5,2], [5,4], [6,3]])
seven_y = sp.array([1, 1, 1, 1, -1, -1, -1])
```

```python
plot_problem(seven_X, seven_y)
```

```python
from sklearn.svm import SVC

svc = SVC(kernel='linear')
svc.fit(seven_X, seven_y)
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, degree=3, gamma=0.0,
      kernel='linear', max_iter=-1, probability=False, random_state=None,
      shrinking=True, tol=0.001, verbose=False)

```python
plot_problem(seven_X, seven_y, svc.predict)
```

```python
svc.predict(sp.array([1, 1]))
```

    array([1])

```python
svc.predict(sp.array([6, 1]))
```

    array([-1])

```python
svc.predict(sp.array([3.5, 2]))
```

    array([1])

```python
svc.decision_function(seven_X)
```

    array([[ 1.99980469],
           [ 1.99921875],
           [ 2.99921875],
           [ 0.99980469],
           [-0.99960937],
           [-1.00019531],
           [-1.99960938]])

```python
svc.support_
```

    array([4, 5, 3], dtype=int32)

```python
svc.dual_coef_
```

    array([[  4.99707031e-01,   1.46484375e-04,  -4.99853516e-01]])

**NB:** Koeficijenti iz `dual_coef_` ne odgovaraju vrijednostima $\alpha_i$ već vrijednostima $-\alpha_i y^{(i)}$

```python
svc.support_vectors_
```

    array([[ 5.,  2.],
           [ 5.,  4.],
           [ 3.,  2.]])

#### Rekonstrukcija primarnog problema

* Nakon što je model naučen, možemo rekonstruirati primarne varijable, odnosno težine $\mathbf{w}$ i $w_0$

* Težine $\mathbf{w}$:
$$
\begin{align}
\mathbf{w} &= \sum_{i=1}^N \alpha_i y^{(i)}\mathbf{x}^{(i)}
\end{align}
$$

* Pomak $w_0$: za potporne vektore $\mathbf{x}^{(i)}\in S$ vrijedi

$$
\begin{align*}
y^{(i)} h(\mathbf{x}) &= 1\\
y^{(i)} \Big(\sum_{j\in S} \alpha_j y^{(j)}(\mathbf{x}^{(i)})^\intercal\mathbf{x}^{(j)} + w_0\Big) &= y^{(i)} y^{(i)}\\
w_0 &= y^{(i)} - \sum_{j\in S} \alpha_j y^{(j)}(\mathbf{x}^{(i)})^\intercal\mathbf{x}^{(j)}\\
\end{align*}
$$

* $w_0$ možemo izračunati na temelju jednog primjera $\mathbf{x}^{(i)}$. Međutim, radi numeričke stabilnosti, bolje je uprosječiti izračun nad svim potpornim vektorima:

$$
w_0 = \frac{1}{|S|}\sum_{i\in S} \Big( y^{(i)} - \sum_{j\in S} \alpha_j y^{(j)}(\mathbf{x}^{(i)})^\intercal\mathbf{x}^{(j)}\Big)
$$

```python
def w(X, y, dc, sv):
    s = 0
    for alpha, i in zip(dc, sv):
        s += -alpha * X[i]
    return s

def w0(X, y, dc, sv):
    s = 0
    for i in sv:
        r = 0
        for alpha, j in zip(dc, sv) :
            r += -alpha * sp.dot(X[i], X[j])
        s += y[i] - r
    return s / float(len(sv))
```

```python
w(seven_X, seven_y, svc.dual_coef_[0], svc.support_)
```

    array([ -9.99707031e-01,  -2.92968750e-04])

```python
svc.coef_
```

    array([[ -9.99707031e-01,  -2.92968750e-04]])

```python
w0(seven_X, seven_y, svc.dual_coef_[0], svc.support_)
```

    3.9995117187499982

```python
svc.intercept_
```

    array([ 3.99951172])

```python
def h_primal(x, w, w0):
    return sp.dot(w, x) + w0

def h_dual(x, X, y, dc, sv):
    xs = [ -alpha * sp.dot(x, X[i]) for alpha, i in zip(dc, sv) ]
    return sum(xs) + w0(X, y, dc, sv)
```

```python
svc.decision_function(seven_X)
```

    array([[ 1.99980469],
           [ 1.99921875],
           [ 2.99921875],
           [ 0.99980469],
           [-0.99960937],
           [-1.00019531],
           [-1.99960938]])

```python
map(lambda x: h_primal(x ,svc.coef_, svc.intercept_), seven_X)
```

    [array([ 1.99980469]),
     array([ 1.99921875]),
     array([ 2.99921875]),
     array([ 0.99980469]),
     array([-0.99960938]),
     array([-1.00019531]),
     array([-1.99960938])]

```python
map(lambda x: h_dual(x, seven_X, seven_y, svc.dual_coef_[0], svc.support_), seven_X)
```

    [1.9998046874999988,
     1.9992187499999989,
     2.999218749999998,
     0.99980468749999929,
     -0.99960937500000036,
     -1.0001953124999989,
     -1.9996093750000021]

#### Je li ovo jedini način za treniranje SVM-a?

* Nije. Mogli smo ostati na primarnom problemu, i riješiti ga npr. stohastičkim gradijentnim spustom uz penalizaciju (algoritam *Pegasos*)

* Postoji niz SVM solvera
  * https://cseweb.ucsd.edu/~akmenon/ResearchExam.pdf
  * https://mitpress.mit.edu/sites/default/files/titles/content/9780262026253_sch_0001.pdf

* Međutim, prednost dualne formulacije je da omogućava **jezgreni trik**

# Meka margina

$$
\begin{align}
y^{(i)}\big(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0\big) \geq 1 - \xi_i, \quad i=1,\dots,N
\end{align}
$$

$$
\xi_i \geq 0, i=1,\dots,N
$$

* Ciljna funkcija:

$$
\begin{equation}
\frac{1}{2}\|\mathbf{w}\|^2+ C\sum_{i=1}^N\xi_i
\end{equation}
$$

> $$
\mathrm{argmin}_{\mathbf{w},w_0}
\Big\{\frac{1}{2}\|\mathbf{w}\|^2+ C\sum_{i=1}^N\xi_i \Big\}
$$
> Uz ograničenja:
> $$
\begin{align*}
y^{(i)}\big(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0\big) &\geq 1 - \xi_i, \quad& i=1,\dots,N\\
\xi_i&\geq 0, & i=1,\dots,N\\
\end{align*}
$$

* Pripadna Lagrangeova funkcija:

$$
L(\mathbf{w}, w_0,\boldsymbol\xi,\color{red}{\boldsymbol\alpha},\color{red}{\boldsymbol\beta})=
\frac{1}{2}\|\mathbf{w}\|^2+ C\sum_{i=1}^N \xi_i
- \sum_{i=1}^N\color{red}{\alpha_i}\big(y^{(i)} h(\mathbf{x})-1 + \xi_i\big)
- \sum_{i=1}^N\color{red}{\beta_i}\xi_i
$$

* Minimizacija po $\mathbf{w}$, $w_0$ i $\xi_i$:

$$
\begin{align*}
\mathbf{w} &= \sum_{i=1}^N \alpha_i y^{(i)}\mathbf{x}^{(i)}\\
\sum_{i=1}^N\alpha_i y^{(i)} &= 0\\
\alpha_i &= C - \beta_i
\end{align*}
$$

* Dualna Lagrangeova funkcija:

$$
\tilde{L}(\boldsymbol\alpha) =
\begin{align}
\sum_{i=1}^N\alpha_i -
\frac{1}{2}\sum_{i=1}^N
\sum_{j=1}^N
\alpha_i
\alpha_j
y^{(i)}
y^{(j)}
(\mathbf{x}^{(i)})^\intercal
\mathbf{x}^{(j)}
\end{align}
$$

* Dualni optimizacijski problem:

> **Maksimizirati** izraz
> \begin{align}
\sum_{i=1}^N\alpha_i -
\frac{1}{2}\sum_{i=1}^N
\sum_{j=1}^N
\alpha_i
\alpha_j
y^{(i)}
y^{(j)}
(\mathbf{x}^{(i)})^\intercal
\mathbf{x}^{(j)}
\end{align}
> tako da:
> \begin{align*}
0 \leq \alpha_i &\leq C,\quad i=1,\dots,N \\
\sum_{i=1}^N\alpha_i y^{(i)} &= 0, \quad i=1,\dots,N
\end{align*}

```python
X1, y1 = sp.append(seven_X, [[3,3]], axis=0), sp.append(seven_y, -1)
X2, y2 = sp.append(seven_X, [[2,2]], axis=0), sp.append(seven_y, -1)
```

```python
def predict(model,x) :
    #h = sp.dot(model.coef_,x)+model.intercept_
    h = model.decision_function(x)
    if h >= -1 and h <= 1 : return 0.5
    else : return max(-1,min(1,h))
    return h
```

```python
from sklearn.metrics import hinge_loss

for C in [1e-2,1,1e2] :
    svc = SVC(C=C,kernel='linear')
    svc.fit(X1,y1)
    plot_problem(X1,y1,lambda x: predict(svc,x)); show()
    print svc.support_vectors_
    print "margin = ", 2/sp.linalg.norm(svc.coef_)
    print "loss = ", hinge_loss(y1,svc.decision_function(X1))
```

```python
for C in [1e-2, 1, 1e2] :
    svc = SVC(C=C, kernel='linear')
    svc.fit(X2, y2)
    plot_problem(X2, y2, lambda x: predict(svc,x)); show()
    print svc.support_vectors_
    print "margin = ", 2/sp.linalg.norm(svc.coef_)
    print "loss = ", hinge_loss(y1,svc.decision_function(X2))
```

# Gubitak zglobnice

TODO (v. skriptu)

# Jezgreni trik

TODO (v. skriptu)

# Mercerove jezgre

TODO (v. skriptu)

# Optimizacija hiperparametara

TODO (v. skriptu)

# Sažetak

TODO

```python
def plot_problem(X, y, h=None, surfaces=True) :
    '''
    Plots a two-dimensional labeled dataset (X,y) and, if function h(x) is given,
    the decision boundaries (surfaces=False) or decision surfaces (surfaces=True)
    '''
    assert X.shape[1] == 2, "Dataset is not two-dimensional"
    if h!=None :
        # Create a mesh to plot in
        r = 0.02  # mesh resolution
        x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
        y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
        xx, yy = np.meshgrid(np.arange(x_min, x_max, r),
                             np.arange(y_min, y_max, r))
        XX=np.c_[xx.ravel(), yy.ravel()]
        try:
            Z_test = h(XX)
            if shape(Z_test) == () :
                # h returns a scalar when applied to a matrix; map explicitly
                Z = sp.array(map(h,XX))
            else :
                Z = Z_test
        except ValueError:
            # can't apply to a matrix; map explicitly
            Z = sp.array(map(h,XX))
        # Put the result into a color plot
        Z = Z.reshape(xx.shape)
        if surfaces :
            plt.contourf(xx, yy, Z, cmap=plt.cm.Pastel1)
        else :
            plt.contour(xx, yy, Z)
    # Plot the dataset
    scatter(X[:,0],X[:,1],c=y, cmap=plt.cm.Paired,marker='o',s=50);
```


# Fizeau and Sansbury
> "Here we begin analysis of Fizeau's classical sawtooth experiment (1849) with Sansbury's proposed setup. While Fizeau studies a saw tooth wheel in uniform motion on the order of twenty rotations per second, Sansbury's setup can be interpreted as either the same wheel in nonuniform motion, or a wheel with variable saw tooth length in uniform motion. We argue that Sansbury's predictions could be tested if Fizeau's wheel could increase it's angular velocity from 20 revolutions per second, to 80 revolutions or 160 revolutions per second. "

- toc: false
- branch: master
- badges: false
- comments: true
- author: JHM
- categories: [fizeau, sansbury, c, SR]

As described in previous posts, our critical [essay](https://github.com/jhmartel/SR/) on foundations in SR is basically stable and readable. Here we begin to elaborate on the concluding section, which briefly mentions Ralph Sansbury's proposed experiment. Sansbury's experiment is this. We take $c$ the speed of light as equal to $1$ foot per nanosecond $[\mu s]=10^{-8}$ seconds.

# Sansbury's Experiment

We quote from Sansbury's [paper](http://www.naturalphilosophy.org/pdf/abstracts/abstracts_5955.pdf) and his book ["Faster than Light"](https://www.amazon.ca/Faster-Than-Light-Relativity-Reconsidered-ebook/dp/B00LQT8056).

__(Case 1)__ _A $15$ nanosecond light pulse from a laser was sent to a light detector, $30$ feet away. When the light pulse was blocked at the photodiode during the time of emission, but unblocked at the expected time of arrival, $31.2$ nanoseconds after the beginning of the time of emission, for $15$ nanosecond duration, little light was received. (A little more than the $4mV$ noise on the oscilloscope). This process was repeated thousands of times per second._

__(Case 2)__ _When the light was unblocked at the photodiode during the time of emission ($15$ nanoseconds) but blocked after the beginning of the time of emission, during the expected time of arrival for $15$ nanoseconds, twice as much light was received ($8mV$). This process was repeated thousands of times per second._

Sansbury's conclusion? That _this indicated that light is not a moving wave or photon, but rather the cumulative effect of instantaneous forces at a distance. That is, undetectable oscillations of charge can occur in the atomic nuclei of the photodiode that spill over as detectable oscillations of electrons after a delay._

Sansbury found the equipment necessary for the experiment too expensive to rent for an extended period of time, and he was possibly not a sufficient expert in calibrating the equipment. So Sansbury's experiment appears to have not been sufficiently investigated, and we would argue that the experiment has neither been reproduced nor properly reviewed. Thus we turn to the classical Fizeau experiment, and consider its similarities to Sansbury's setup.

# Fizeau's Saw Tooth Experiment (1849)

Now Sansbury's apparatus has some similarity with Fizeau's sawtooth apparatus, as used circa 1849 to prove some of the first sensible measurements of luminal velocity.

[Here](https://skullsinthestars.com/2008/03/31/fizeaus-experiment-the-original-paper/) is a blog which inlcudes a transcription of Fizeau's original 1849 paper in Comptes Rendus.

[Here](https://youtu.be/7CUE1Bpz4hM) is an interesting youtube video en francais sur la mesure de Fizeau. Around the 4-5 minute mark is the most interesting. While the speed of the wheel is increased, the received light signal becomes increasingly erratic and intermittent, until a sufficiently high speed of rotation is achieved and the received light signal becomes eventually null and _no light is received_.

[Here](https://physics.stackexchange.com/questions/315812/why-does-the-fizeau-measurement-of-the-speed-of-light-use-a-sawtoothed-gear-inst) is a useful physics stackexchange answer.

# Sansbury vs. Fizeau

Now Fizeau's original setup is a type of Sansbury test where the wheel is in uniform motion. Sansbury's setup involves either a wheel in _nonuniform_ motion, or equivalently a wheel in uniform motion with a nonuniform sawtooth distribution.

We remark that Fizeau was historically looking to estimate the luminal velocity $c$. Sansbury's experiment assumes $c$ as given, estimated at $1$ foot per nanosecond. Sansbury's goal is to distinguish light propagation from particle model, and his setup is meant to test whether light is even something that travels at all.

Let's review the basic math of Fizeau's apparatus, it's very simple. No matter how we setup the mirrors, we suppose light has some total travel time. In Fizeau's original setup, light travelled a total path of $\approx 16 km$. With an expected speed of light $c=3\times 10^8 km$, then the expected travel time is

\begin{equation}
\frac{16 \times 10^3 [m]}{3 \times 10^8 [m]/[sec]}=5.333\ldots \times 10^{-5} [sec].\end{equation}

Now consider the wheel with angular velocity $\omega$ having units of $[degrees]/[sec]$ and having a toothlength equal to $1/720$ degrees. The time required to turn one toothlength is therefore $\frac{1/720}{\omega}=\frac{1}{720 \omega} [sec]$. Thus we find that the expected travel time is equal to the time to rotate one toothlength if the following equality holds $$\frac{16 \times 10^3 }{3 \times 10^8} = \frac{1}{720 \omega}, $$ which implies $$\omega \approx 26 ~~~\frac{[rotations]}{[sec]}.$$

Sansbury's __(Case 1)__ could be realized if the wheel was allowed to rotate nonuniformly, i.e. if the wheel could be accelerated in "impulses" something closer to the actual discrete motions of a clock. For example, if the wheel is initially opened at the time of emission, then immediately rotated one saw tooth length (to the closed position) during the expected time of travel, and just prior to the expected time of arrival is rotated another tooth length (to the open position), then Fizeau would predict that the receiver would observe a strong light source. However Sansbury predicts that the receiver would observe rather a very weak signal. Notice here we require the wheel to move twice as fast as Fizeau's angular velocity. In otherwords the wheel must rotate two complete tooth lengths before the estimated arrival time.

Sansbury's __(Case 2)__ could be realized if the wheel was rotating _nonuniformly_. For example, if we keep the wheel fixed during the expected time of flight of the light particle, and turn the wheel one complete toothlength at the expected time of arrival, then Sansbury would predict a relatively strong signal would be received. Fizeau and the particle model would however predict no light would be received, since in the model it would be blocked by the sawtooth at the expected time of arrival.

N.B. Fizeau's conception of the sawtooth wheel is _classical_. But what happens if we retrospectively apply the SR methodology to the experiment, what results are obtained? It appears that SR has a null effect on the entire experiment, i.e. returns the same results as the classical case. While the sawteeth lengths are contracting in SR, this effects the circumference of the wheel but not the angular velocity. Thus Fizeau experiment appears insensitive to any Lorentz SR effects and an experiment which cannot prove SR in contrast to the classical mechanics.

We do not require mathematics at this stage, but rather to perform an experiment. However there is an intersting math aspect to the question, "How to keep a nonuniform wheel in uniform motion?".

Fizeau's original wheel was materially balanced: the distribution of teeth was equidistant and regular. The centre of mass corresponded with the axis of rotation.

But if we begin to study nonuniform wheels, then the behaviour becomes more difficult depending on, say, whether the centre of mass coincides with the axis of rotation.

Some comparisons between Fizeau and Sansbury:

1. Fizeau's involves several reflecting mirrors (beam splitters). Therefore there is more interaction involved in Fizeau's setup than with Sansbury's. For Fizeau's is a type of two-way trip of light, where the source and receiver are space-coincident. But Sansbury's is a one-way trip, requiring some electronics at the receiver namely a photodiode, to measure the amount of electrons released by the light emission.

2. If the phase of Fizeau's wheel could be controlled, then we could compare the behaviour of the experiment when the wheels differ by one saw tooth length. The trouble in Fizeau is that, because the source and receiver coincide, it's evident that no light is emitted when the phase is shifted one tooth length. Sansbury's experiment however does emphatically require the apparatus to be _open_ at the moment of emission.

# Faster Fizeau Wheels

We could test some of Sansbury's ideas if we could increase the angular velocity of the Fizeau wheel by factor of $4$, i.e. we need a wheel of roughly $100$ revolutions per second instead of $20$ revolutions per second.

Given such a revolution speed, then we could change the sawtooth pattern of the wheels, having some that are $1/4$ closed, $1/2$ closed, and $3/4$ closed wheels. For example, we could have the alternating sawtooth $$\ldots 0101010101 \ldots$$ or we could have $$\ldots 001100110011 \ldots$$ both of which are $1/2$ closed but having different patterns. And these patterns would have different predictions depending on the photon model or Sansbury's cumulative action-at-a-distance. Likewise it would be interesting to compare the predictions given a wheel having a $1/4$-closed sawtooth pattern $$\ldots 0001000100010001 \ldots$$ versus a $3/4$-closed pattern $$ \ldots 0111011101110111\ldots.$$

If we could get the Fizeau wheel to spin $200$ revolutions per second, then we could test the theories according to $8$-periodic patterns, i.e. with sawtooth patterns being $1/8, 2/8$, $\ldots$, $7/8$ths closed.

If we could build a larger wheel with more teeth, say, $1440$ teeth, then $2880$ teeth, then basic gear ratio would increase the speed of the initial _pinion_ wheel by factor of $\times 2$, $\times 4$, etc..

The heuristics by which we can determine reasonable revolutions per second depends probably on some energy estimates and would require smaller and smaller radii.

[To be continued...]


```python
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from IPython.display import Image
from astropy.cosmology import WMAP9 as cosmo
from astropy import constants as const

from causticpy import *
from functions_caustic import to_xyz_coordinates, linear_angle, data_query_id
```

 У нас имеются таблицы с данными каталогов:
- **SDSS (glist_s)**,
- **2MRS (glist_2)**.

**SDSS** - *Sloan Digital Sky Survey* с англ. — «Слоуновский цифровой небесный обзор») — проект широкомасштабного исследования многоспектральных изображений и спектров красного смещения звёзд и галактик при помощи 2,5-метрового широкоугольного телескопа в обсерватории Апачи-Пойнт в штате Нью-Мексико. Проект назван в честь фонда Альфреда Слоуна.

Исследования начались в 2000 году, в ходе работы проекта было проведено картографирование более 35 % небесной сферы с фотометрическими наблюдениями порядка 500 миллионов объектов и получением спектров более чем для 3 миллионов объектов. Среднее значение красного смещения по изображениям галактик составило 0.1; для ярких красных галактик вплоть до z=0,4, для квазаров до z=5. Наблюдения в рамках обзора способствовали обнаружению квазаров со сдвигом более 6.

Проект делится на несколько фаз: SDSS-I (2000—2005), SDSS-II (2005—2008), SDSS-III (2008—2014), SDSS-IV (2014—2020). Собранные в ходе обзоров данные публикуются в виде отдельных релизов (Data Release), последний из них, DR13 опубликован в августе 2016 года.

**2MASS Redshift Survey** -  aims to map the distribution of galaxies and dark matter in the local universe, out to a mean redshift of z = 0.03 (roughly equivalent to 115 Mpc or 370 million light-years). It is based on galaxy selection in the near infra-red from the *Two Micron All-Sky Survey* (**2MASS**). 2MASS has now mapped all of the sky in the near infra-red J, H and K-bands. This photometric survey is complete and fully available to the public (IRSA). The 2MASS extended source catalog (XSC) includes roughly half a million galaxies to a limiting K magnitude of K=13.5 mag. 2MRS ultimately aims to determine the redshifts of all galaxies in the XSC to a magnitude of K=12.2 mag (about 100,000 galaxies) and to within 5 deg of the Galactic plane. The second phase of 2MRS is now complete, providing an all-sky survey of 45,000 galaxies with redshifts to a limiting magnitude of K=11.75 mag. It is the densest sampled all-sky redshift survey to date and its selection in the near infra-red reduces the impact of the zone of avoidance (where the plane of our own Galaxy obscures extragalactic objects). 2MRS provides complementary redshift information to deeper surveys like SDSS and the 2dFRGS which cover much smaller fractions of the sky. It improves on the IRAS redshift survey IRAS PSCz which was not able to distinguish galaxies in regions of high density (ie. clusters).

```python
Image("projections.png")
```

В астрофизике принято рассматривать небесные объекты в проекции на небесную сферу.

Предполагается, что наблюдатель располагается в вершине угла $a$.
на рисунке обозначено:

$O$ - центр некоторого скопления,

$A_0$ - некоторая галактика из скопления,

$r_0$ - вектор, соединяющий глаз наблюдателя и центр скопления $O$,

$r_1$ - вектор, соединяющий глаз наблюдателя и объект $A_0$,

$P$ - плоскость построенная таким образом, что она перпендикулярна вектору   $r_0$ и проходит через центр скопления $O$,

$A$ - точка пересечения луча, идущего из глаза наблюдателя через объект $А_0$ с плоскостью $P$,

$H$ - перпендикуляр, опущенный из из  $A_0$ на плоскость $H$,

$a$ - линейный угол между векторами $r_0$ и $r_1$,

красные отрезки $OA(OH)$ -  проекции на плоскость Р.

###### Так как наблюдения проводятся из одной точки, то интересующая нас проекция - центральная.

Поскольку данные об углах  right ascention (RAJ2000) и declination (DEJ2000) указаны в градусах, а вычислениях мы будем пользоваться радианами, то с помощью формулы перейдем от одних единиц измерения к другим:

\begin{equation}
\alpha_{rad} = \pi \frac{\alpha_{grad}}{180}
\end{equation}

Через данные об углах наклонения (DEC=DEJ2000) и вознесения (RA=RAJ2000) вычисляются координаты соответствующего объекта на единичной сфере:

 \begin{equation}
\left\{ \begin{array}{ll}
    x = \cos(DEC)\cos(RA) \\
    y = \cos(DEC)\sin(RA)\\
    z = \sin(DEC)
\end{array} \right.
\end{equation}

Далее необходимо вычислить длинну проекции $OA$  для каждой галактики из соответствующего скопления по всем скоплениям с помощью формулы:

\begin{equation}
     r_{pr} = r_0 \tan(a),
\end{equation}

где угол $а$ - линейный угол между лучами, соединяющими наблюдателя с галактикой в скоплении и наблюдателя с центром скопления. Зная углы наклонения (DEC=DEJ2000) и вознесения (RA=RAJ2000), можно вычислить тангенс угла $а$:

 \begin{equation}
\left\{ \begin{array}{ll}
    x = \cos(DEC)\cos(RA) \\
    y = \cos(DEC)\sin(RA)\\
    z = \sin(DEC)
\end{array} \right.
\end{equation}

\begin{equation}
\cos(a) = x_{centr}x_{gal} + y_{centr}y_{gal}+z_{centr}z_{gal}
\end{equation}

\begin{equation}
\tan(a) = \frac{\sqrt{1-\cos^2(a)}}{\cos(a)}
\end{equation}

Введем используемые в дальнейшем исследовании космологические постоянные:

Скорости галактик в скоплении относительно центра этого скопления высчитывались с помощью формулы:

\begin{equation}
    v = c * (z_{gal} - z_{cent})
\end{equation}

Отметим, что все эти скорости могут вычислены только вдоль луча зрения (line-of-sight) на основании измерений красного смещения (red shift)  $z$ рассматриваемого объекта. Поперечные скорости определить сложно, так как время съемки телескопом данного участка неба по сравнению с перемещением объекта по небесной сфере ничтожно мало, а потому увидеть какие-то видимые измения в положении объекта на небе пока не представляется возможным.

Далее воспользуемся программой 'Caustic mass estimator for astrophysical systems', взятой с репозитория  Dan Gifford на GitHub http://github.com/giffordw . Данная программа позволяет на основе данных об углах   RA, DEC и красного смещения z строить каустические кривые для скоплений галактик. Программный код основывается на методах, описанных в статьях:
- A.Diafferio. Mass estimation in the outer regions of galaxy clusters. 1999.
- Gifford et al. A Systematic Analysis of Caustic Methods for Galaxy Cluster Masses. 2013.
- Gifford & Miller. Velocity Anisotropy and Shape Bias in the Caustic Technique. 2013.

Каустики - это некие кривые в фазовой плоскости (r,v), которые разделяют эту плоскость на несколько сегметов: центральный, нижний и верхний. Из взаимного положения галактик в коплении и соответствующих каустик на фазовой плоскости можно судить о будующем этого скопления. Например, если галактики лежат вне центральной области, ограниченной каустиками, то можно заключить, что данная галактика в будущем покинет это скопление.

```python
C = const.c.to('km/s') #300000  # km/s -  скорость света в вакууме
# Mpc = 3.086e+19 # km -  мегапарсек
```

```python
data = pd.read_csv('glist_2.csv')
```

```python
data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 1200 entries, 0 to 1199
    Data columns (total 21 columns):
    iGalID            1200 non-null int64
    iGrID             1200 non-null int64
    Name              1200 non-null object
    RAJ2000_gal       1200 non-null float64
    DEJ2000_gal       1200 non-null float64
    z_gal             1200 non-null float64
    logMstar_gal      425 non-null float64
    RAJ2000_group     1200 non-null float64
    DEJ2000_group     1200 non-null float64
    z_group           1200 non-null float64
    logLtot           1200 non-null float64
    logLobs           1200 non-null float64
    logMtot           1200 non-null float64
    logMstar_group    1165 non-null float64
    NMstar            1200 non-null int64
    logMdyn           1200 non-null float64
    sigma             1200 non-null float64
    Rad               1200 non-null float64
    angRad            1200 non-null float64
    DL                1200 non-null float64
    Ntot              1200 non-null int64
    dtypes: float64(16), int64(4), object(1)
    memory usage: 197.0+ KB

```python
# !pip install scikit-image
```

```python
iGrID = data['iGrID'].unique()[0:1]
```

```python
# data.columns
```

```python
data_query = data_query_id(data, iGrID[0], 'iGrID')
galaxydata = data_query.to_numpy()
```

```python
c = Caustic()
good_flag = c.run_caustic(galaxydata,
                          clus_ra=galaxydata[:,3].mean(),
                          clus_dec=galaxydata[:,4].mean(),
                          clus_z=galaxydata[:,5].mean())#, r200=0.743, clus_ra=195.095, clus_dec=19.131,clus_z=0.063)
```

    DATA SET SIZE 43
    Pre_r200= 0.35374227608257586
    Calculating Density w/Mirrored Data
    Vdisp from galaxies= 118.4137724813458
    Combined Vdisp= 118.4137724813458
    Calculating initial surface
    complete
    r200 estimate:  0.2338076934627482
    M200 estimate:  1550714469573.0642

```python
data_query.columns
```

    Index(['RAJ2000_gal', 'DEJ2000_gal', 'z_gal', 'RAJ2000_group', 'DEJ2000_group',
           'z_group', 'iGrID', 'DL', 'r_pr', 'v'],
          dtype='object')

```python
fig, ax = plt.subplots(1, 1, figsize=(7,5))
x_max = data_query['r_pr'].max()
y_max = abs(data_query['v']).max()
x = data_query['r_pr']
y = data_query['v']
ax.scatter(x, y, c=data_query['iGrID'], marker='*')
ax.plot(c.x_range, c.caustic_profile, c='red')
ax.plot(c.x_range, -c.caustic_profile, c='red')
ax.plot(c.r,c.v,'o', c='black')
ax.set_xlabel('Distance from center of the cluster, [Mpc]')
ax.set_ylabel('Relative velocity in the cluster, [km/s]')
ax.set_title('\n Caustic curve \n')
# ax.set_title('iGrID = %d' % (iGrID))
ax.set_ylim(top=y_max*1.1, bottom=-y_max*1.1)
ax.set_xlim((0,10))
ax.grid(True)
# plt.savefig("caustic.png")
```

```python
# Image("caustic.png")
```

```python
from pylab import *
plot(c.r,c.v,'o', c='black')
# plt.savefig("caustic.png")
# Image("caustic.png")
```

```python
# Проверить почему не работает???

# plot(data[data['iGrID']==iGrID]['r_pr'],data[data['iGrID']==iGrID]['v'],'o', c='black')
# show()
```

Понять как они определяют расстояния. Очевидно, что немного по-другому. Это может быть причиной различия в оцененных массах на основе каустик и Mtot из таблиц.

To make two pictures made above look simular we made same changes in the
- file "\__init\__"
- in class Caustic
- in function run_caustic
- in function findangle(self,ra,dec,clus_RA,clus_DEC)

Because with a previous code a projected radius was calculated another way.

```python
m = MassCalc(ri = c.x_range,
    A = c.caustic_profile,
    vdisp = c.vdisp_gal,
    clus_z = galaxydata[:,5].mean(),
    r200=2.0,
    conc1=5,
    beta=0.25,
    fbr=None,
    H0=100.0,)
```

```python
m.M200
```

    17403720159393.834

```python
12427715478374.973 # for native radius calculatiouns
12427715478374.607 # data[data['iGrID']==iGrID]['sigma'].mean()
12427715478374.607 # for c.vdisp_gal
```

    12427715478374.607

Let's put caustic profile onto the galaxy cluster in a phase space

```python

```