Posit AI Weblog: Audio classification with torch

Variations on a theme

Easy audio classification with Keras, Audio classification with Keras: Wanting nearer on the non-deep studying elements, Easy audio classification with torch: No, this isn’t the primary put up on this weblog that introduces speech classification utilizing deep studying. With two of these posts (the “utilized” ones) it shares the final setup, the kind of deep-learning structure employed, and the dataset used. With the third, it has in widespread the curiosity within the concepts and ideas concerned. Every of those posts has a distinct focus – do you have to learn this one?

Effectively, in fact I can’t say “no” – all of the extra so as a result of, right here, you’ve got an abbreviated and condensed model of the chapter on this matter within the forthcoming guide from CRC Press, Deep Studying and Scientific Computing with R torch. By the use of comparability with the earlier put up that used torch, written by the creator and maintainer of torchaudio, Athos Damiani, important developments have taken place within the torch ecosystem, the top outcome being that the code obtained so much simpler (particularly within the mannequin coaching half). That stated, let’s finish the preamble already, and plunge into the subject!

Inspecting the info

We use the speech instructions dataset (Warden (2018)) that comes with torchaudio. The dataset holds recordings of thirty completely different one- or two-syllable phrases, uttered by completely different audio system. There are about 65,000 audio recordsdata general. Our job can be to foretell, from the audio solely, which of thirty potential phrases was pronounced.

library(torch)
library(torchaudio)
library(luz)

ds  speechcommand_dataset(
  root = "~/.torch-datasets", 
  url = "speech_commands_v0.01",
  obtain = TRUE
)

We begin by inspecting the info.

[1]  "mattress"    "hen"   "cat"    "canine"    "down"   "eight"
[7]  "5"   "4"   "go"     "pleased"  "home"  "left"
[32] " marvin" "9"   "no"     "off"    "on"     "one"
[19] "proper"  "seven" "sheila" "six"    "cease"   "three"
[25]  "tree"   "two"    "up"     "wow"    "sure"    "zero" 

Selecting a pattern at random, we see that the knowledge we’ll want is contained in 4 properties: waveform, sample_rate, label_index, and label.

The primary, waveform, can be our predictor.

pattern  ds[2000]
dim(pattern$waveform)

[1]     1 16000

Particular person tensor values are centered at zero, and vary between -1 and 1. There are 16,000 of them, reflecting the truth that the recording lasted for one second, and was registered at (or has been transformed to, by the dataset creators) a fee of 16,000 samples per second. The latter data is saved in pattern$sample_rate:

[1] 16000

All recordings have been sampled on the identical fee. Their size virtually all the time equals one second; the – very – few sounds which can be minimally longer we will safely truncate.

Lastly, the goal is saved, in integer type, in pattern$label_index, the corresponding phrase being out there from pattern$label:

pattern$label
pattern$label_index

[1] "hen"
torch_tensor
2
[ CPULongType{} ]

How does this audio sign “look?”

library(ggplot2)

df  knowledge.body(
  x = 1:size(pattern$waveform[1]),
  y = as.numeric(pattern$waveform[1])
  )

ggplot(df, aes(x = x, y = y)) +
  geom_line(dimension = 0.3) +
  ggtitle(
    paste0(
      "The spoken phrase "", pattern$label, "": Sound wave"
    )
  ) +
  xlab("time") +
  ylab("amplitude") +
  theme_minimal()

What we see is a sequence of amplitudes, reflecting the sound wave produced by somebody saying “hen.” Put in another way, now we have right here a time collection of “loudness values.” Even for consultants, guessing which phrase resulted in these amplitudes is an unimaginable job. That is the place area data is available in. The professional could not be capable to make a lot of the sign on this illustration; however they might know a option to extra meaningfully symbolize it.

Two equal representations

Think about that as a substitute of as a sequence of amplitudes over time, the above wave have been represented in a manner that had no details about time in any respect. Subsequent, think about we took that illustration and tried to recuperate the unique sign. For that to be potential, the brand new illustration would by some means should comprise “simply as a lot” data because the wave we began from. That “simply as a lot” is obtained from the Fourier Rework, and it consists of the magnitudes and part shifts of the completely different frequencies that make up the sign.

How, then, does the Fourier-transformed model of the “hen” sound wave look? We receive it by calling torch_fft_fft() (the place fft stands for Quick Fourier Rework):

dft  torch_fft_fft(pattern$waveform)
dim(dft)

[1]     1 16000

The size of this tensor is identical; nonetheless, its values will not be in chronological order. As an alternative, they symbolize the Fourier coefficients, akin to the frequencies contained within the sign. The upper their magnitude, the extra they contribute to the sign:

magazine  torch_abs(dft[1, ])

df  knowledge.body(
  x = 1:(size(pattern$waveform[1]) / 2),
  y = as.numeric(magazine[1:8000])
)

ggplot(df, aes(x = x, y = y)) +
  geom_line(dimension = 0.3) +
  ggtitle(
    paste0(
      "The spoken phrase "",
      pattern$label,
      "": Discrete Fourier Rework"
    )
  ) +
  xlab("frequency") +
  ylab("magnitude") +
  theme_minimal()

From this alternate illustration, we may return to the unique sound wave by taking the frequencies current within the sign, weighting them in keeping with their coefficients, and including them up. However in sound classification, timing data should certainly matter; we don’t actually need to throw it away.

Combining representations: The spectrogram

In truth, what actually would assist us is a synthesis of each representations; some form of “have your cake and eat it, too.” What if we may divide the sign into small chunks, and run the Fourier Rework on every of them? As you’ll have guessed from this lead-up, this certainly is one thing we will do; and the illustration it creates is named the spectrogram.

With a spectrogram, we nonetheless maintain some time-domain data – some, since there’s an unavoidable loss in granularity. Then again, for every of the time segments, we study their spectral composition. There’s an essential level to be made, although. The resolutions we get in time versus in frequency, respectively, are inversely associated. If we break up up the indicators into many chunks (known as “home windows”), the frequency illustration per window won’t be very fine-grained. Conversely, if we need to get higher decision within the frequency area, now we have to decide on longer home windows, thus dropping details about how spectral composition varies over time. What feels like an enormous downside – and in lots of instances, can be – gained’t be one for us, although, as you’ll see very quickly.

First, although, let’s create and examine such a spectrogram for our instance sign. Within the following code snippet, the scale of the – overlapping – home windows is chosen in order to permit for cheap granularity in each the time and the frequency area. We’re left with sixty-three home windows, and, for every window, receive 200 fifty-seven coefficients:

fft_size  512
window_size  512
energy  0.5

spectrogram  transform_spectrogram(
  n_fft = fft_size,
  win_length = window_size,
  normalized = TRUE,
  energy = energy
)

spec  spectrogram(pattern$waveform)$squeeze()
dim(spec)

[1]   257 63

We are able to show the spectrogram visually:

bins  1:dim(spec)[1]
freqs  bins / (fft_size / 2 + 1) * pattern$sample_rate 
log_freqs  log10(freqs)

frames  1:(dim(spec)[2])
seconds  (frames / dim(spec)[2]) *
  (dim(pattern$waveform$squeeze())[1] / pattern$sample_rate)

picture(x = as.numeric(seconds),
      y = log_freqs,
      z = t(as.matrix(spec)),
      ylab = 'log frequency [Hz]',
      xlab = 'time [s]',
      col = hcl.colours(12, palette = "viridis")
)
principal  paste0("Spectrogram, window dimension = ", window_size)
sub  "Magnitude (sq. root)"
mtext(facet = 3, line = 2, at = 0, adj = 0, cex = 1.3, principal)
mtext(facet = 3, line = 1, at = 0, adj = 0, cex = 1, sub)

We all know that we’ve misplaced some decision in each time and frequency. By displaying the sq. root of the coefficients’ magnitudes, although – and thus, enhancing sensitivity – we have been nonetheless in a position to receive an inexpensive outcome. (With the viridis colour scheme, long-wave shades point out higher-valued coefficients; short-wave ones, the alternative.)

Lastly, let’s get again to the essential query. If this illustration, by necessity, is a compromise – why, then, would we need to make use of it? That is the place we take the deep-learning perspective. The spectrogram is a two-dimensional illustration: a picture. With photos, now we have entry to a wealthy reservoir of strategies and architectures: Amongst all areas deep studying has been profitable in, picture recognition nonetheless stands out. Quickly, you’ll see that for this job, fancy architectures will not be even wanted; an easy convnet will do an excellent job.

Coaching a neural community on spectrograms

We begin by making a torch::dataset() that, ranging from the unique speechcommand_dataset(), computes a spectrogram for each pattern.

spectrogram_dataset  dataset(
  inherit = speechcommand_dataset,
  initialize = operate(...,
                        pad_to = 16000,
                        sampling_rate = 16000,
                        n_fft = 512,
                        window_size_seconds = 0.03,
                        window_stride_seconds = 0.01,
                        energy = 2) {
    self$pad_to  pad_to
    self$window_size_samples  sampling_rate *
      window_size_seconds
    self$window_stride_samples  sampling_rate *
      window_stride_seconds
    self$energy  energy
    self$spectrogram  transform_spectrogram(
        n_fft = n_fft,
        win_length = self$window_size_samples,
        hop_length = self$window_stride_samples,
        normalized = TRUE,
        energy = self$energy
      )
    tremendous$initialize(...)
  },
  .getitem = operate(i) {
    merchandise  tremendous$.getitem(i)

    x  merchandise$waveform
    # ensure that all samples have the identical size (57)
    # shorter ones can be padded,
    # longer ones can be truncated
    x  nnf_pad(x, pad = c(0, self$pad_to - dim(x)[2]))
    x  x %>% self$spectrogram()

    if (is.null(self$energy)) {
      # on this case, there's a further dimension, in place 4,
      # that we need to seem in entrance
      # (as a second channel)
      x  x$squeeze()$permute(c(3, 1, 2))
    }

    y  merchandise$label_index
    record(x = x, y = y)
  }
)

Within the parameter record to spectrogram_dataset(), notice energy, with a default worth of two. That is the worth that, except informed in any other case, torch’s transform_spectrogram() will assume that energy ought to have. Underneath these circumstances, the values that make up the spectrogram are the squared magnitudes of the Fourier coefficients. Utilizing energy, you’ll be able to change the default, and specify, for instance, that’d you’d like absolute values (energy = 1), every other optimistic worth (comparable to 0.5, the one we used above to show a concrete instance) – or each the actual and imaginary elements of the coefficients (energy = NULL).

Show-wise, in fact, the complete complicated illustration is inconvenient; the spectrogram plot would want a further dimension. However we could effectively ponder whether a neural community may revenue from the extra data contained within the “entire” complicated quantity. In spite of everything, when lowering to magnitudes we lose the part shifts for the person coefficients, which could comprise usable data. In truth, my checks confirmed that it did; use of the complicated values resulted in enhanced classification accuracy.

Let’s see what we get from spectrogram_dataset():

ds  spectrogram_dataset(
  root = "~/.torch-datasets",
  url = "speech_commands_v0.01",
  obtain = TRUE,
  energy = NULL
)

dim(ds[1]$x)

[1]   2 257 101

We have now 257 coefficients for 101 home windows; and every coefficient is represented by each its actual and imaginary elements.

Subsequent, we break up up the info, and instantiate the dataset() and dataloader() objects.

train_ids  pattern(
  1:size(ds),
  dimension = 0.6 * size(ds)
)
valid_ids  pattern(
  setdiff(
    1:size(ds),
    train_ids
  ),
  dimension = 0.2 * size(ds)
)
test_ids  setdiff(
  1:size(ds),
  union(train_ids, valid_ids)
)

batch_size  128

train_ds  dataset_subset(ds, indices = train_ids)
train_dl  dataloader(
  train_ds,
  batch_size = batch_size, shuffle = TRUE
)

valid_ds  dataset_subset(ds, indices = valid_ids)
valid_dl  dataloader(
  valid_ds,
  batch_size = batch_size
)

test_ds  dataset_subset(ds, indices = test_ids)
test_dl  dataloader(test_ds, batch_size = 64)

b  train_dl %>%
  dataloader_make_iter() %>%
  dataloader_next()

dim(b$x)

[1] 128   2 257 101

The mannequin is a simple convnet, with dropout and batch normalization. The actual and imaginary elements of the Fourier coefficients are handed to the mannequin’s preliminary nn_conv2d() as two separate channels.

mannequin  nn_module(
  initialize = operate() {
    self$options  nn_sequential(
      nn_conv2d(2, 32, kernel_size = 3),
      nn_batch_norm2d(32),
      nn_relu(),
      nn_max_pool2d(kernel_size = 2),
      nn_dropout2d(p = 0.2),
      nn_conv2d(32, 64, kernel_size = 3),
      nn_batch_norm2d(64),
      nn_relu(),
      nn_max_pool2d(kernel_size = 2),
      nn_dropout2d(p = 0.2),
      nn_conv2d(64, 128, kernel_size = 3),
      nn_batch_norm2d(128),
      nn_relu(),
      nn_max_pool2d(kernel_size = 2),
      nn_dropout2d(p = 0.2),
      nn_conv2d(128, 256, kernel_size = 3),
      nn_batch_norm2d(256),
      nn_relu(),
      nn_max_pool2d(kernel_size = 2),
      nn_dropout2d(p = 0.2),
      nn_conv2d(256, 512, kernel_size = 3),
      nn_batch_norm2d(512),
      nn_relu(),
      nn_adaptive_avg_pool2d(c(1, 1)),
      nn_dropout2d(p = 0.2)
    )

    self$classifier  nn_sequential(
      nn_linear(512, 512),
      nn_batch_norm1d(512),
      nn_relu(),
      nn_dropout(p = 0.5),
      nn_linear(512, 30)
    )
  },
  ahead = operate(x) {
    x  self$options(x)$squeeze()
    x  self$classifier(x)
    x
  }
)

We subsequent decide an appropriate studying fee:

mannequin  mannequin %>%
  setup(
    loss = nn_cross_entropy_loss(),
    optimizer = optim_adam,
    metrics = record(luz_metric_accuracy())
  )

rates_and_losses  mannequin %>%
  lr_finder(train_dl)
rates_and_losses %>% plot()

Primarily based on the plot, I made a decision to make use of 0.01 as a maximal studying fee. Coaching went on for forty epochs.

fitted  mannequin %>%
  match(train_dl,
    epochs = 50, valid_data = valid_dl,
    callbacks = record(
      luz_callback_early_stopping(persistence = 3),
      luz_callback_lr_scheduler(
        lr_one_cycle,
        max_lr = 1e-2,
        epochs = 50,
        steps_per_epoch = size(train_dl),
        call_on = "on_batch_end"
      ),
      luz_callback_model_checkpoint(path = "models_complex/"),
      luz_callback_csv_logger("logs_complex.csv")
    ),
    verbose = TRUE
  )

plot(fitted)

Let’s test precise accuracies.

"epoch","set","loss","acc"
1,"prepare",3.09768574611813,0.12396992171405
1,"legitimate",2.52993751740923,0.284378862793572
2,"prepare",2.26747255972008,0.333642356819118
2,"legitimate",1.66693911248562,0.540791100123609
3,"prepare",1.62294889937818,0.518464153275649
3,"legitimate",1.11740599192825,0.704882571075402
...
...
38,"prepare",0.18717994078312,0.943809229501442
38,"legitimate",0.23587799138006,0.936418417799753
39,"prepare",0.19338578602993,0.942882159044087
39,"legitimate",0.230597475945365,0.939431396786156
40,"prepare",0.190593419024368,0.942727647301195
40,"legitimate",0.243536252455384,0.936186650185414

With thirty courses to differentiate between, a remaining validation-set accuracy of ~0.94 seems like a really respectable outcome!

We are able to affirm this on the check set:

consider(fitted, test_dl)

loss: 0.2373
acc: 0.9324

An fascinating query is which phrases get confused most frequently. (After all, much more fascinating is how error chances are associated to options of the spectrograms – however this, now we have to depart to the true area consultants. A pleasant manner of displaying the confusion matrix is to create an alluvial plot. We see the predictions, on the left, “circulation into” the goal slots. (Goal-prediction pairs much less frequent than a thousandth of check set cardinality are hidden.)

Wrapup

That’s it for immediately! Within the upcoming weeks, anticipate extra posts drawing on content material from the soon-to-appear CRC guide, Deep Studying and Scientific Computing with R torch. Thanks for studying!

Photograph by alex lauzon on Unsplash