Posit AI Weblog: Implementing rotation equivariance: Group-equivariant CNN from scratch

Posit AI Weblog: Implementing rotation equivariance: Group-equivariant CNN from scratch

Convolutional neural networks (CNNs) are nice – they’re capable of detect options in a picture irrespective of the place. Effectively, not precisely. They’re not detached to simply any sort of motion. Shifting up or down, or left or proper, is ok; rotating round an axis shouldn’t be. That’s due to how convolution works: traverse by row, then traverse by column (or the opposite method spherical). If we wish “extra” (e.g., profitable detection of an upside-down object), we have to lengthen convolution to an operation that’s rotation-equivariant. An operation that’s equivariant to some kind of motion won’t solely register the moved function per se, but in addition, maintain observe of which concrete motion made it seem the place it’s.

That is the second put up in a sequence that introduces group-equivariant CNNs (GCNNs). The primary was a high-level introduction to why we’d need them, and the way they work. There, we launched the important thing participant, the symmetry group, which specifies what sorts of transformations are to be handled equivariantly. In the event you haven’t, please check out that put up first, since right here I’ll make use of terminology and ideas it launched.

As we speak, we code a easy GCNN from scratch. Code and presentation tightly comply with a notebook supplied as a part of College of Amsterdam’s 2022 Deep Learning Course. They’ll’t be thanked sufficient for making accessible such wonderful studying supplies.

In what follows, my intent is to elucidate the final considering, and the way the ensuing structure is constructed up from smaller modules, every of which is assigned a transparent goal. For that purpose, I gained’t reproduce all of the code right here; as a substitute, I’ll make use of the bundle gcnn. Its strategies are closely annotated; so to see some particulars, don’t hesitate to take a look at the code.

As of at present, gcnn implements one symmetry group: (C_4), the one which serves as a operating instance all through put up one. It’s straightforwardly extensible, although, making use of sophistication hierarchies all through.

Step 1: The symmetry group (C_4)

In coding a GCNN, the very first thing we have to present is an implementation of the symmetry group we’d like to make use of. Right here, it’s (C_4), the four-element group that rotates by 90 levels.

We are able to ask gcnn to create one for us, and examine its components.

# remotes::install_github("skeydan/gcnn")

C_4 <- CyclicGroup(order = 4)
elems <- C_4$components()
[ CPUFloatType{4} ]

Parts are represented by their respective rotation angles: (0), (frac{pi}{2}), (pi), and (frac{3 pi}{2}).

Teams are conscious of the identification, and know the way to assemble a component’s inverse:


g1 <- elems[2]
[ CPUFloatType{1} ]

[ CPUFloatType{} ]

Right here, what we care about most is the group components’ motion. Implementation-wise, we have to distinguish between them appearing on one another, and their motion on the vector area (mathbb{R}^2), the place our enter pictures reside. The previous half is the simple one: It might merely be applied by including angles. In truth, that is what gcnn does after we ask it to let g1 act on g2:

g2 <- elems[3]

# in C_4$left_action_on_H(), H stands for the symmetry group
C_4$left_action_on_H(torch_tensor(g1)$unsqueeze(1), torch_tensor(g2)$unsqueeze(1))
[ CPUFloatType{1,1} ]

What’s with the unsqueeze()s? Since (C_4)’s final raison d’être is to be a part of a neural community, left_action_on_H() works with batches of components, not scalar tensors.

Issues are a bit much less easy the place the group motion on (mathbb{R}^2) is worried. Right here, we want the idea of a group representation. That is an concerned subject, which we gained’t go into right here. In our present context, it really works about like this: We now have an enter sign, a tensor we’d wish to function on in a roundabout way. (That “a way” will likely be convolution, as we’ll see quickly.) To render that operation group-equivariant, we first have the illustration apply the inverse group motion to the enter. That completed, we go on with the operation as if nothing had occurred.

To offer a concrete instance, let’s say the operation is a measurement. Think about a runner, standing on the foot of some mountain path, able to run up the climb. We’d wish to report their peak. One choice we’ve is to take the measurement, then allow them to run up. Our measurement will likely be as legitimate up the mountain because it was down right here. Alternatively, we may be well mannered and never make them wait. As soon as they’re up there, we ask them to return down, and once they’re again, we measure their peak. The outcome is similar: Physique peak is equivariant (greater than that: invariant, even) to the motion of operating up or down. (In fact, peak is a fairly boring measure. However one thing extra attention-grabbing, akin to coronary heart fee, wouldn’t have labored so effectively on this instance.)

Returning to the implementation, it seems that group actions are encoded as matrices. There’s one matrix for every group component. For (C_4), the so-called customary illustration is a rotation matrix:

begin{bmatrix} cos(theta) & -sin(theta) sin(theta) & cos(theta) end{bmatrix}

In gcnn, the operate making use of that matrix is left_action_on_R2(). Like its sibling, it’s designed to work with batches (of group components in addition to (mathbb{R}^2) vectors). Technically, what it does is rotate the grid the picture is outlined on, after which, re-sample the picture. To make this extra concrete, that methodology’s code seems to be about as follows.

Here’s a goat.

img_path <- system.file("imgs", "z.jpg", bundle = "gcnn")
img <- torchvision::base_loader(img_path) |> torchvision::transform_to_tensor()
img$permute(c(2, 3, 1)) |> as.array() |> as.raster() |> plot()

A goat sitting comfortably on a meadow.

First, we name C_4$left_action_on_R2() to rotate the grid.

# Grid form is [2, 1024, 1024], for a 2nd, 1024 x 1024 picture.
img_grid_R2 <- torch::torch_stack(torch::torch_meshgrid(
      torch::torch_linspace(-1, 1, dim(img)[2]),
      torch::torch_linspace(-1, 1, dim(img)[3])

# Rework the picture grid with the matrix illustration of some group component.
transformed_grid <- C_4$left_action_on_R2(C_4$inverse(g1)$unsqueeze(1), img_grid_R2)

Second, we re-sample the picture on the remodeled grid. The goat now seems to be as much as the sky.

transformed_img <- torch::nnf_grid_sample(
  img$unsqueeze(1), transformed_grid,
  align_corners = TRUE, mode = "bilinear", padding_mode = "zeros"

transformed_img[1,..]$permute(c(2, 3, 1)) |> as.array() |> as.raster() |> plot()

Same goat, rotated up by 90 degrees.

Step 2: The lifting convolution

We need to make use of current, environment friendly torch performance as a lot as doable. Concretely, we need to use nn_conv2d(). What we want, although, is a convolution kernel that’s equivariant not simply to translation, but in addition to the motion of (C_4). This may be achieved by having one kernel for every doable rotation.

Implementing that concept is precisely what LiftingConvolution does. The precept is similar as earlier than: First, the grid is rotated, after which, the kernel (weight matrix) is re-sampled to the remodeled grid.

Why, although, name this a lifting convolution? The same old convolution kernel operates on (mathbb{R}^2); whereas our prolonged model operates on combos of (mathbb{R}^2) and (C_4). In math communicate, it has been lifted to the semi-direct product (mathbb{R}^2rtimes C_4).

lifting_conv <- LiftingConvolution(
    group = CyclicGroup(order = 4),
    kernel_size = 5,
    in_channels = 3,
    out_channels = 8

x <- torch::torch_randn(c(2, 3, 32, 32))
y <- lifting_conv(x)
[1]  2  8  4 28 28

Since, internally, LiftingConvolution makes use of an extra dimension to comprehend the product of translations and rotations, the output shouldn’t be four-, however five-dimensional.

Step 3: Group convolutions

Now that we’re in “group-extended area”, we will chain quite a few layers the place each enter and output are group convolution layers. For instance:

group_conv <- GroupConvolution(
  group = CyclicGroup(order = 4),
    kernel_size = 5,
    in_channels = 8,
    out_channels = 16

z <- group_conv(y)
[1]  2 16  4 24 24

All that continues to be to be finished is bundle this up. That’s what gcnn::GroupEquivariantCNN() does.

Step 4: Group-equivariant CNN

We are able to name GroupEquivariantCNN() like so.

cnn <- GroupEquivariantCNN(
    group = CyclicGroup(order = 4),
    kernel_size = 5,
    in_channels = 1,
    out_channels = 1,
    num_hidden = 2, # variety of group convolutions
    hidden_channels = 16 # variety of channels per group conv layer

img <- torch::torch_randn(c(4, 1, 32, 32))
[1] 4 1

At informal look, this GroupEquivariantCNN seems to be like several outdated CNN … weren’t it for the group argument.

Now, after we examine its output, we see that the extra dimension is gone. That’s as a result of after a sequence of group-to-group convolution layers, the module tasks all the way down to a illustration that, for every batch merchandise, retains channels solely. It thus averages not simply over places – as we usually do – however over the group dimension as effectively. A ultimate linear layer will then present the requested classifier output (of dimension out_channels).

And there we’ve the whole structure. It’s time for a real-world(ish) check.

Rotated digits!

The concept is to coach two convnets, a “regular” CNN and a group-equivariant one, on the same old MNIST coaching set. Then, each are evaluated on an augmented check set the place every picture is randomly rotated by a steady rotation between 0 and 360 levels. We don’t count on GroupEquivariantCNN to be “excellent” – not if we equip with (C_4) as a symmetry group. Strictly, with (C_4), equivariance extends over 4 positions solely. However we do hope it is going to carry out considerably higher than the shift-equivariant-only customary structure.

First, we put together the information; particularly, the augmented check set.

dir <- "/tmp/mnist"

train_ds <- torchvision::mnist_dataset(
  obtain = TRUE,
  remodel = torchvision::transform_to_tensor

test_ds <- torchvision::mnist_dataset(
  prepare = FALSE,
  remodel = operate(x) >
        levels = c(0, 360),
        resample = 2,
        fill = 0

train_dl <- dataloader(train_ds, batch_size = 128, shuffle = TRUE)
test_dl <- dataloader(test_ds, batch_size = 128)

How does it look?

test_images <- coro::gather(
  test_dl, 1
)[[1]]$x[1:32, 1, , ] |> as.array()

par(mfrow = c(4, 8), mar = rep(0, 4), mai = rep(0, 4))
test_images |>
  purrr::array_tree(1) |>
  purrr::map(as.raster) |>
  purrr::iwalk(~ {

32 digits, rotated randomly.

We first outline and prepare a standard CNN. It’s as much like GroupEquivariantCNN(), architecture-wise, as doable, and is given twice the variety of hidden channels, in order to have comparable capability general.

 default_cnn <- nn_module(
   initialize = operate(kernel_size, in_channels, out_channels, num_hidden, hidden_channels) {
     self$conv1 <- torch::nn_conv2d(in_channels, hidden_channels, kernel_size)
     self$convs <- torch::nn_module_list()
     for (i in 1:num_hidden) {
       self$convs$append(torch::nn_conv2d(hidden_channels, hidden_channels, kernel_size))
     self$avg_pool <- torch::nn_adaptive_avg_pool2d(1)
     self$final_linear <- torch::nn_linear(hidden_channels, out_channels)
   ahead = operate(x) >

fitted <- default_cnn |>
      loss = torch::nn_cross_entropy_loss(),
      optimizer = torch::optim_adam,
      metrics = list(
    ) |>
      kernel_size = 5,
      in_channels = 1,
      out_channels = 10,
      num_hidden = 4,
      hidden_channels = 32
    ) %>%
    luz::set_opt_hparams(lr = 1e-2, weight_decay = 1e-4) |>
    luz::match(train_dl, epochs = 10, valid_data = test_dl) 
Practice metrics: Loss: 0.0498 - Acc: 0.9843
Legitimate metrics: Loss: 3.2445 - Acc: 0.4479

Unsurprisingly, accuracy on the check set shouldn’t be that nice.

Subsequent, we prepare the group-equivariant model.

fitted <- GroupEquivariantCNN |>
    loss = torch::nn_cross_entropy_loss(),
    optimizer = torch::optim_adam,
    metrics = list(
  ) |>
    group = CyclicGroup(order = 4),
    kernel_size = 5,
    in_channels = 1,
    out_channels = 10,
    num_hidden = 4,
    hidden_channels = 16
  ) |>
  luz::set_opt_hparams(lr = 1e-2, weight_decay = 1e-4) |>
  luz::match(train_dl, epochs = 10, valid_data = test_dl)
Practice metrics: Loss: 0.1102 - Acc: 0.9667
Legitimate metrics: Loss: 0.4969 - Acc: 0.8549

For the group-equivariant CNN, accuracies on check and coaching units are loads nearer. That may be a good outcome! Let’s wrap up at present’s exploit resuming a thought from the primary, extra high-level put up.

A problem

Going again to the augmented check set, or somewhat, the samples of digits displayed, we discover an issue. In row two, column 4, there’s a digit that “beneath regular circumstances”, ought to be a 9, however, most likely, is an upside-down 6. (To a human, what suggests that is the squiggle-like factor that appears to be discovered extra usually with sixes than with nines.) Nonetheless, you may ask: does this have to be an issue? Possibly the community simply must study the subtleties, the sorts of issues a human would spot?

The best way I view it, all of it will depend on the context: What actually ought to be completed, and the way an utility goes for use. With digits on a letter, I’d see no purpose why a single digit ought to seem upside-down; accordingly, full rotation equivariance can be counter-productive. In a nutshell, we arrive on the similar canonical crucial advocates of honest, simply machine studying maintain reminding us of:

At all times consider the best way an utility goes for use!

In our case, although, there may be one other side to this, a technical one. gcnn::GroupEquivariantCNN() is an easy wrapper, in that its layers all make use of the identical symmetry group. In precept, there isn’t a want to do that. With extra coding effort, totally different teams can be utilized relying on a layer’s place within the feature-detection hierarchy.

Right here, let me lastly let you know why I selected the goat image. The goat is seen by way of a red-and-white fence, a sample – barely rotated, as a result of viewing angle – made up of squares (or edges, in the event you like). Now, for such a fence, forms of rotation equivariance akin to that encoded by (C_4) make loads of sense. The goat itself, although, we’d somewhat not have look as much as the sky, the best way I illustrated (C_4) motion earlier than. Thus, what we’d do in a real-world image-classification activity is use somewhat versatile layers on the backside, and more and more restrained layers on the high of the hierarchy.

Thanks for studying!

Picture by Marjan Blan | @marjanblan on Unsplash

Leave a Reply

Your email address will not be published. Required fields are marked *