tensorplay.autograd.gradcheck API

Check gradients computed via small finite differences against analytical gradients wrt tensors in inputs that are of floating point or complex type and with requires_grad=True.

The check between numerical and analytical gradients uses allclose().

For most of the complex functions we consider for optimization purposes, no notion of Jacobian exists. Instead, gradcheck verifies if the numerical and analytical values of the Wirtinger and Conjugate Wirtinger derivatives are consistent. Because the gradient computation is done under the assumption that the overall function has a real-valued output, we treat functions with complex output in a special way. For these functions, gradcheck is applied to two real-valued functions corresponding to taking the real components of the complex outputs for the first, and taking the imaginary components of the complex outputs for the second.

Parameters:

• func (function) – a Python function that takes Tensor inputs and returns a Tensor or a tuple of Tensors

• inputs (tuple of Tensor or Tensor) – inputs to the function

• eps (float, optional) – perturbation for finite differences

• atol (float, optional) – absolute tolerance

• rtol (float, optional) – relative tolerance

• raise_exception (bool, optional) – indicating whether to raise an exception if the check fails. The exception gives more information about the exact nature of the failure. This is helpful when debugging gradchecks.

• nondet_tol (float, optional) – tolerance for non-determinism. When running identical inputs through the differentiation, the results must either match exactly (default, 0.0) or be within this tolerance.

• check_undefined_grad (bool, optional) – if True, check if undefined output grads are supported and treated as zeros, for Tensor outputs.

• check_grad_dtypes (bool, optional) – if True, check that the gradient dtypes match the ones from the numerical computation. Defaults to False.

• check_batched_grad (bool, optional) – if True, check if we can compute batched gradients using tensorplay.vmap(). Defaults to False.

• check_batched_forward_grad (bool, optional) – if True, checks if we can compute batched forward gradients using forward ad and tensorplay.vmap(). Defaults to False.

• check_forward_ad (bool, optional) – if True, check that the gradients computed with forward mode AD match the numerical ones. Defaults to False.

• check_backward_ad (bool, optional) – if False, do not perform any checks that rely on backward mode AD to be implemented. Defaults to True.

• fast_mode (bool, optional) – if True, compare the jacobians projected onto random vectors instead of element by element, which costs one backward (or forward) pass per output instead of one per output element. Defaults to False.

• masked (bool, optional) – if True, the gradients of unspecified elements of sparse tensors are ignored. Defaults to False.

Returns:

True if all differences satisfy allclose condition

Check gradients of gradients computed via small finite differences against analytical gradients wrt tensors in inputs and grad_outputs that are of floating point or complex type and with requires_grad=True.

This function checks that backpropagating through the gradients computed to the given grad_outputs are correct.

The check between numerical and analytical gradients uses allclose().

Parameters:

• func (function) – a Python function that takes Tensor inputs and returns a Tensor or a tuple of Tensors

• inputs (tuple of Tensor or Tensor) – inputs to the function

• grad_outputs (tuple of [Tensor or None] or Tensor, optional) – The gradients with respect to the function’s outputs.

• eps (float, optional) – perturbation for finite differences

• atol (float, optional) – absolute tolerance

• rtol (float, optional) – relative tolerance

• gen_non_contig_grad_outputs (bool, optional) – if grad_outputs is None and gen_non_contig_grad_outputs is True, the randomly generated gradient outputs are made to be noncontiguous

• raise_exception (bool, optional) – indicating whether to raise an exception if the check fails. The exception gives more information about the exact nature of the failure. This is helpful when debugging gradchecks.

• nondet_tol (float, optional) – tolerance for non-determinism. When running identical inputs through the differentiation, the results must either match exactly (default, 0.0) or be within this tolerance. Note that a small amount of nondeterminism in the gradient will lead to larger inaccuracies in the second derivative.

• check_undefined_grad (bool, optional) – if True, check if undefined output grads are supported and treated as zeros

• check_batched_grad (bool, optional) – if True, check if we can compute batched gradients using tensorplay.vmap(). Defaults to False.

• check_fwd_over_rev (bool, optional) – if True, check the second derivative with forward-mode AD over the first-order backward. Defaults to False.

• check_rev_over_rev (bool, optional) – if True, check the second derivative with reverse-mode AD over the first-order backward. Defaults to True.

• fast_mode (bool, optional) – if True, run a faster implementation of gradgradcheck that no longer computes the entire jacobian.

• masked (bool, optional) – if True, the gradients of unspecified elements of sparse tensors are ignored (default, False).

Returns:

True if all differences satisfy allclose condition