Broadcasting semantics

Many TensorPlay operations support NumPy’s broadcasting semantics. See Numpy Documentation for details.

In short, if a TensorPlay operation supports broadcast, then its Tensor arguments can be automatically expanded to be of equal sizes (without making copies of the data).

General semantics

Two tensors are “broadcastable” if the following rules hold:

When iterating over the dimension sizes, starting at the trailing dimension, the dimension sizes must either be equal, one of them is 1, or one of them does not exist.

For Example:

>>> x=tensorplay.empty(5,7,3)
>>> y=tensorplay.empty(5,7,3)
# same shapes are always broadcastable (i.e. the above rules always hold)

>>> x=tensorplay.empty((0,))
>>> y=tensorplay.empty(2,2)
# x and y are not broadcastable, because the 0-sized dimension of x
# does not match the 2-sized dimension of y.

# can line up trailing dimensions
>>> x=tensorplay.empty(5,3,4,1)
>>> y=tensorplay.empty(  3,1,1)
# x and y are broadcastable.
# 1st trailing dimension: both have size 1
# 2nd trailing dimension: y has size 1
# 3rd trailing dimension: x size == y size
# 4th trailing dimension: y dimension doesn't exist

# but:
>>> x=tensorplay.empty(5,2,4,1)
>>> y=tensorplay.empty(  3,1,1)
# x and y are not broadcastable, because in the 3rd trailing dimension 2 != 3

If two tensors x, y are “broadcastable”, the resulting tensor size is calculated as follows:

If the number of dimensions of x and y are not equal, prepend 1 to the dimensions of the tensor with fewer dimensions to make them equal length.
Then, for each dimension size, the resulting dimension size is the max of the sizes of x and y along that dimension.

For Example:

# can line up trailing dimensions to make reading easier
>>> x=tensorplay.empty(5,1,4,1)
>>> y=tensorplay.empty(  3,1,1)
>>> (x+y).size()
tensorplay.Size([5, 3, 4, 1])

# but not necessary:
>>> x=tensorplay.empty(1)
>>> y=tensorplay.empty(3,1,7)
>>> (x+y).size()
tensorplay.Size([3, 1, 7])

>>> x=tensorplay.empty(5,2,4,1)
>>> y=tensorplay.empty(3,1,1)
>>> (x+y).size()
RuntimeError: The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 1

In-place semantics

One complication is that in-place operations do not allow the in-place tensor to change shape as a result of the broadcast.

For Example:

>>> x=tensorplay.empty(5,3,4,1)
>>> y=tensorplay.empty(3,1,1)
>>> (x.add_(y)).size()
tensorplay.Size([5, 3, 4, 1])

# but:
>>> x=tensorplay.empty(1,3,1)
>>> y=tensorplay.empty(3,1,7)
>>> (x.add_(y)).size()
RuntimeError: The expanded size of the tensor (1) must match the existing size (7) at non-singleton dimension 2.

Backwards compatibility

TensorPlay supports broadcasting, so the older behaviour — certain pointwise functions executing on tensors of different shapes as long as the number of elements matched, by viewing each tensor as 1-dimensional — is considered deprecated. It now generates a Python warning in cases where tensors are not broadcastable but have the same number of elements.

Note that the presence of broadcasting can cause backwards incompatible changes in the case where two tensors do not have the same shape, but are broadcastable and have the same number of elements. For Example:

>>> tensorplay.add(tensorplay.ones(4,1), tensorplay.randn(4))

would previously produce a Tensor with size: tensorplay.Size([4,1]), but now produces a Tensor with size: tensorplay.Size([4,4]). In order to help identify cases in your code where backwards incompatibilities introduced by broadcasting may exist, you may set tensorplay.utils.backcompat.broadcast_warning.enabled to True, which will generate a python warning in such cases.

For Example:

>>> tensorplay.utils.backcompat.broadcast_warning.enabled=True
>>> tensorplay.add(tensorplay.ones(4,1), tensorplay.ones(4))
__main__:1: UserWarning: self and other do not have the same shape, but are broadcastable, and have the same number of elements.
Changing behavior in a backwards incompatible manner to broadcasting rather than viewing as 1-dimensional.

Broadcasting semantics

General semantics

In-place semantics

Backwards compatibility

Help improve this page

On this page