relentless.optimize.Adam#

class relentless.optimize.Adam(stop, max_iter, step_size, beta_1=0.9, beta_2=0.999, epsilon=1e-08, scale=1.0)#

Adam optimization algorithm.

For an ObjectiveFunction \(f\left(\mathbf{x}\right)\), the Adam optimization algorithm seeks to approach a minimum of the function.

The optimization is performed using scaled variables \(\mathbf{y}\). Define \(\mathbf{X}\) as the scaling parameters for each variable such that \(y_i=x_i/X_i\). (A variable can be left unscaled by setting \(X_i=1\)). The gradient of the function with respect to the scaled variables is:

\[\mathbf{g} = \nabla f\left(\mathbf{y}\right) = \left[X_1 \frac{\partial f}{\partial x_1}, \cdots, X_n \frac{\partial f}{\partial x_n}\right]\]

Define \(\alpha\) as the descent step size hyperparameter. Adam iteratively minimizes the function by taking steps based on exponentially weighted first and second moment estimates of the gradient. Let \(\mathbf{g}_n\) be the gradient at iteration \(n\), and let \(\mathbf{m}_n\) and \(\mathbf{v}_n\) be the first and second moment estimates. If the scaled variables are \(\mathbf{y}_n\) at iteration \(n\), the next value of the variables is:

\[\begin{split}\mathbf{m}_n &= \beta_1 \mathbf{m}_{n-1} + \left(1-\beta_1\right)\mathbf{g}_n \\ \mathbf{v}_n &= \beta_2 \mathbf{v}_{n-1} + \left(1-\beta_2\right){\mathbf{g}_n}^2 \\ \hat{\mathbf{m}}_n &= \frac{\mathbf{m}_n}{1-{\beta_1}^n} \\ \hat{\mathbf{v}}_n &= \frac{\mathbf{v}_n}{1-{\beta_2}^n} \\ \mathbf{y}_{n+1} &= \mathbf{y}_n-\alpha \frac{\hat{\mathbf{m}}_n} {\sqrt{\hat{\mathbf{v}}_n}+\epsilon}\end{split}\]
Parameters:

  • stop (ConvergenceTest) – The convergence test used as the stopping criterion for the optimizer. Note that the result being tested will have unscaled variables and gradient.

  • max_iter (int) – The maximum number of optimization iterations allowed.

  • step_size (float) – The step size hyperparameter (\(\alpha\)).

  • beta_1 (float) – The exponential decay rate for the first moment estimates (defaults to 0.9).

  • beta_2 (float) – The exponential decay rate for the second moment estimates (defaults to 0.999).

  • epsilon (float) – A small constant added for numerical stability (defaults to 1e-8).

  • scale (float or dict) – A scalar scaling parameter or scaling parameters (\(\mathbf{X}\)) keyed on one or more ObjectiveFunction design variables (defaults to 1.0, so that the variables are unscaled).

Methods

optimize(objective, variables[, directory, ...])

Perform the Adam optimization for the given objective function.

Attributes

beta_1

Exponential decay rate for the first moment estimates.

beta_2

Exponential decay rate for the second moment estimates.

epsilon

A small constant for numerical stability.

max_iter

The maximum number of optimization iterations allowed.

scale

Scaling parameter.

step_size

The step size hyperparameter (\(\alpha\)).

stop

The convergence test used as the stopping criterion for the optimizer.
property beta_1#

Exponential decay rate for the first moment estimates.

Type:

float

property beta_2#

Exponential decay rate for the second moment estimates.

Type:

float

property epsilon#

A small constant for numerical stability.

Type:

float

property max_iter#

The maximum number of optimization iterations allowed.

Type:

int

optimize(objective, variables, directory=None, overwrite=False)#

Perform the Adam optimization for the given objective function.

If directory is specified and overwrite is True, directory will be cleared before the optimization begins. The output will be saved into a directory created for each iteration of the optimization, e.g., directory/0. To advance to the next iteration of the optimization (e.g., from iteration 0 to iteration 1), a directory directory/0/.next is created at iteration 0 to hold the proposed result at iteration 1.

Parameters:

  • objective (ObjectiveFunction) – The objective function to be optimized.

  • variables (IndependentVariable or tuple) – Design variable(s) to optimize.

  • directory (str or Directory) – Directory for writing output during optimization. Default of None requests no output is written.

  • overwrite (bool) – If True, overwrite the directory before beginning optimization.

Returns:

True if converged, False if not converged, None if no design variables are specified for the objective function.

Return type:

bool or None

Raises:

OSError – If directory is not empty and overwrite is False.

property scale#

Scaling parameter.

A scalar scaling parameter or scaling parameters (\(\mathbf{X}\)) keyed on one or more ObjectiveFunction design variables. Must be positive.

Type:

float or dict

property step_size#

The step size hyperparameter (\(\alpha\)). Must be positive.

Type:

float

property stop#

The convergence test used as the stopping criterion for the optimizer.

Type:

ConvergenceTest