espnet2.schedulers package

espnet2.schedulers.noam_lr

Noam learning rate scheduler module.

class espnet2.schedulers.noam_lr.NoamLR(optimizer: torch.optim.optimizer.Optimizer, model_size: Union[int, float] = 320, warmup_steps: Union[int, float] = 25000, last_epoch: int = -1)

Bases: torch.optim.lr_scheduler._LRScheduler, espnet2.schedulers.abs_scheduler.AbsBatchStepScheduler

The LR scheduler proposed by Noam

Ref:

“Attention Is All You Need”, https://arxiv.org/pdf/1706.03762.pdf

FIXME(kamo): PyTorch doesn’t provide _LRScheduler as public class,

thus the behaviour isn’t guaranteed at forward PyTorch version.

NOTE(kamo): The “model_size” in original implementation is derived from

the model, but in this implementation, this parameter is a constant value. You need to change it if the model is changed.

get_lr()

lr_for_WarmupLR(lr: float) → float

espnet2.schedulers.cosine_anneal_warmup_restart

class espnet2.schedulers.cosine_anneal_warmup_restart.CosineAnnealingWarmupRestarts(optimizer: torch.optim.optimizer.Optimizer, first_cycle_steps: int, cycle_mult: float = 1.0, max_lr: float = 0.1, min_lr: float = 0.001, warmup_steps: int = 0, gamma: float = 1.0, last_epoch: int = -1)

Bases: torch.optim.lr_scheduler._LRScheduler, espnet2.schedulers.abs_scheduler.AbsBatchStepScheduler

optimizer (Optimizer): Wrapped optimizer. first_cycle_steps (int): First cycle step size. cycle_mult(float): Cycle steps magnification. Default: -1. max_lr(float): First cycle’s max learning rate. Default: 0.1. min_lr(float): Min learning rate. Default: 0.001. warmup_steps(int): Linear warmup step size. Default: 0. gamma(float): Decrease rate of max learning rate by cycle. Default: 1. last_epoch (int): The index of last epoch. Default: -1.

get_lr()

init_lr()

step(epoch=None)

espnet2.schedulers.warmup_reducelronplateau

ReduceLROnPlateau (with Warm up) learning rate scheduler module.

class espnet2.schedulers.warmup_reducelronplateau.WarmupReduceLROnPlateau(optimizer: torch.optim.optimizer.Optimizer, warmup_steps: Union[int, float] = 25000, mode='min', factor=0.1, patience=10, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08, verbose=False)

Bases: espnet2.schedulers.abs_scheduler.AbsBatchStepScheduler, espnet2.schedulers.abs_scheduler.AbsValEpochStepScheduler

The WarmupReduceLROnPlateau scheduler.

This scheduler is the combination of WarmupLR and ReduceLROnPlateau:

WarmupLR:

lr = optimizer.lr * warmup_step ** 0.5

• min(step ** -0.5, step * warmup_step ** -1.5)

WarmupReduceLROnPlateau:

if step <= warmup_step:

lr = optimizer.lr * warmup_step ** 0.5

• min(step ** -0.5, step * warmup_step ** -1.5)

else:

lr = (

optimizer.lr * factor if no improvement for a ‘patience’ number of epochs else optimizer.lr

)

Note that the maximum lr equals to optimizer.lr in this scheduler.

property in_cooldown

is_better(a, best)

load_state_dict(state_dict)

state_dict()

step(metrics=None, epoch=None)

espnet2.schedulers.__init__

espnet2.schedulers.warmup_step_lr

Step (with Warm up) learning rate scheduler module.

class espnet2.schedulers.warmup_step_lr.WarmupStepLR(optimizer: torch.optim.optimizer.Optimizer, warmup_steps: Union[int, float] = 25000, steps_per_epoch: int = 10000, step_size: int = 1, gamma: float = 0.1, last_epoch: int = -1)

Bases: torch.optim.lr_scheduler._LRScheduler, espnet2.schedulers.abs_scheduler.AbsBatchStepScheduler

The WarmupStepLR scheduler.

This scheduler is the combination of WarmupLR and StepLR:

WarmupLR:

lr = optimizer.lr * warmup_step ** 0.5

• min(step ** -0.5, step * warmup_step ** -1.5)

WarmupStepLR:

if step <= warmup_step:

lr = optimizer.lr * warmup_step ** 0.5

• min(step ** -0.5, step * warmup_step ** -1.5)

else:

lr = optimizer.lr * (gamma ** (epoch//step_size))

Note that the maximum lr equals to optimizer.lr in this scheduler.

get_lr()

espnet2.schedulers.abs_scheduler

class espnet2.schedulers.abs_scheduler.AbsBatchStepScheduler

Bases: espnet2.schedulers.abs_scheduler.AbsScheduler

abstract load_state_dict(state)

abstract state_dict()

abstract step(epoch: int = None)

class espnet2.schedulers.abs_scheduler.AbsEpochStepScheduler

Bases: espnet2.schedulers.abs_scheduler.AbsScheduler

abstract load_state_dict(state)

abstract state_dict()

abstract step(epoch: int = None)

class espnet2.schedulers.abs_scheduler.AbsScheduler

Bases: abc.ABC

abstract load_state_dict(state)

abstract state_dict()

abstract step(epoch: int = None)

class espnet2.schedulers.abs_scheduler.AbsValEpochStepScheduler

Bases: espnet2.schedulers.abs_scheduler.AbsEpochStepScheduler

abstract load_state_dict(state)

abstract state_dict()

abstract step(val, epoch: int = None)

espnet2.schedulers.warmup_lr

Warm up learning rate scheduler module.

class espnet2.schedulers.warmup_lr.WarmupLR(optimizer: torch.optim.optimizer.Optimizer, warmup_steps: Union[int, float] = 25000, last_epoch: int = -1)

Bases: torch.optim.lr_scheduler._LRScheduler, espnet2.schedulers.abs_scheduler.AbsBatchStepScheduler

The WarmupLR scheduler

This scheduler is almost same as NoamLR Scheduler except for following difference:

NoamLR:

lr = optimizer.lr * model_size ** -0.5

• min(step ** -0.5, step * warmup_step ** -1.5)

WarmupLR:

lr = optimizer.lr * warmup_step ** 0.5

• min(step ** -0.5, step * warmup_step ** -1.5)

Note that the maximum lr equals to optimizer.lr in this scheduler.

get_lr()