espnet.scheduler package

Initialize sub package.

espnet.scheduler.pytorch

PyTorch optimizer schdulers.

class espnet.scheduler.pytorch.PyTorchScheduler(schedulers: List[espnet.scheduler.scheduler.SchedulerInterface], optimizer: torch.optim.optimizer.Optimizer)

Bases: object

PyTorch optimizer scheduler.

Initialize class.

step(n_iter: int)

Update optimizer by scheduling.

espnet.scheduler.scheduler

Schedulers.

class espnet.scheduler.scheduler.CyclicCosineScheduler(key: str, args: argparse.Namespace)

Bases: espnet.scheduler.scheduler.SchedulerInterface

Cyclic cosine annealing.

Parameters:

• cosine_warmup (int) – number of warmup iterations.

• cosine_total (int) – number of total annealing iterations.

Notes

Proposed in https://openreview.net/pdf?id=BJYwwY9ll (and https://arxiv.org/pdf/1608.03983.pdf). Used in the GPT2 config of Megatron-LM https://github.com/NVIDIA/Megatron-LM

Initialize class.

alias = 'cosine'

scale(n_iter)

Scale of lr.

class espnet.scheduler.scheduler.NoScheduler(key: str, args: argparse.Namespace)

Bases: espnet.scheduler.scheduler.SchedulerInterface

Scheduler which does nothing.

Initialize class.

alias = 'none'

scale(n_iter)

Scale of lr.

class espnet.scheduler.scheduler.NoamScheduler(key, args)

Bases: espnet.scheduler.scheduler.SchedulerInterface

Warmup + InverseSqrt decay scheduler.

Parameters:

noam_warmup (int) – number of warmup iterations.

Initialize class.

alias = 'noam'

scale(step)

Scale of lr.

class espnet.scheduler.scheduler.SchedulerInterface(key: str, args: argparse.Namespace)

Bases: object

Scheduler interface.

Initialize class.

classmethod add_arguments(key: str, parser: argparse.ArgumentParser)

Add arguments for CLI.

alias = ''

classmethod build(key: str, **kwargs)

Initialize this class with python-level args.

Parameters:

key (str) – key of hyper parameter

Returns:

A new instance of LMInterface.

Return type:

LMinterface

get_arg(name)

Get argument without prefix.

scale(n_iter: int) → float

Scale at n_iter.

Parameters:

n_iter (int) – number of current iterations.

Returns:

current scale of learning rate.

Return type:

float

espnet.scheduler.scheduler.dynamic_import_scheduler(module)

Import Scheduler class dynamically.

Parameters:

module (str) – module_name:class_name or alias in SCHEDULER_DICT

Returns:

Scheduler class

Return type:

type

espnet.scheduler.scheduler.register_scheduler(cls)

Register scheduler.

espnet.scheduler.chainer

Chainer optimizer schdulers.

class espnet.scheduler.chainer.ChainerScheduler(schedulers: List[espnet.scheduler.scheduler.SchedulerInterface], optimizer: chainer.optimizer.Optimizer)

Bases: object

Chainer optimizer scheduler.

Initialize class.

step(n_iter: int)

Update optimizer by scheduling.

espnet.scheduler.__init__

Initialize sub package.