espnet.scheduler package

Initialize sub package.

espnet.scheduler.pytorch

PyTorch optimizer schdulers.

class espnet.scheduler.pytorch.PyTorchScheduler(schedulers: List[espnet.scheduler.scheduler.SchedulerInterface], optimizer: torch.optim.optimizer.Optimizer)[source]

Bases: object

PyTorch optimizer scheduler.

Initialize class.

step(n_iter: int)[source]

Update optimizer by scheduling.

espnet.scheduler.scheduler

Schedulers.

class espnet.scheduler.scheduler.CyclicCosineScheduler(key: str, args: argparse.Namespace)[source]

Bases: espnet.scheduler.scheduler.SchedulerInterface

Cyclic cosine annealing.

Parameters:
  • cosine_warmup (int) – number of warmup iterations.

  • cosine_total (int) – number of total annealing iterations.

Notes

Proposed in https://openreview.net/pdf?id=BJYwwY9ll (and https://arxiv.org/pdf/1608.03983.pdf). Used in the GPT2 config of Megatron-LM https://github.com/NVIDIA/Megatron-LM

Initialize class.

alias = 'cosine'
scale(n_iter)[source]

Scale of lr.

class espnet.scheduler.scheduler.NoScheduler(key: str, args: argparse.Namespace)[source]

Bases: espnet.scheduler.scheduler.SchedulerInterface

Scheduler which does nothing.

Initialize class.

alias = 'none'
scale(n_iter)[source]

Scale of lr.

class espnet.scheduler.scheduler.NoamScheduler(key, args)[source]

Bases: espnet.scheduler.scheduler.SchedulerInterface

Warmup + InverseSqrt decay scheduler.

Parameters:

noam_warmup (int) – number of warmup iterations.

Initialize class.

alias = 'noam'
scale(step)[source]

Scale of lr.

class espnet.scheduler.scheduler.SchedulerInterface(key: str, args: argparse.Namespace)[source]

Bases: object

Scheduler interface.

Initialize class.

classmethod add_arguments(key: str, parser: argparse.ArgumentParser)[source]

Add arguments for CLI.

alias = ''
classmethod build(key: str, **kwargs)[source]

Initialize this class with python-level args.

Parameters:

key (str) – key of hyper parameter

Returns:

A new instance of LMInterface.

Return type:

LMinterface

get_arg(name)[source]

Get argument without prefix.

scale(n_iter: int) → float[source]

Scale at n_iter.

Parameters:

n_iter (int) – number of current iterations.

Returns:

current scale of learning rate.

Return type:

float

espnet.scheduler.scheduler.dynamic_import_scheduler(module)[source]

Import Scheduler class dynamically.

Parameters:

module (str) – module_name:class_name or alias in SCHEDULER_DICT

Returns:

Scheduler class

Return type:

type

espnet.scheduler.scheduler.register_scheduler(cls)[source]

Register scheduler.

espnet.scheduler.chainer

Chainer optimizer schdulers.

class espnet.scheduler.chainer.ChainerScheduler(schedulers: List[espnet.scheduler.scheduler.SchedulerInterface], optimizer: chainer.optimizer.Optimizer)[source]

Bases: object

Chainer optimizer scheduler.

Initialize class.

step(n_iter: int)[source]

Update optimizer by scheduling.

espnet.scheduler.__init__

Initialize sub package.