First PyTorch computation - Deep Dive

Overview - First PyTorch computation

What is it?

PyTorch is a tool that helps computers learn from data by doing math with numbers called tensors. A tensor is like a multi-dimensional array, similar to a spreadsheet but with more dimensions. First PyTorch computation means creating these tensors and doing simple math with them, like adding or multiplying. This is the starting point to build smart programs that learn patterns.

Why it matters

Without being able to do these basic computations, computers cannot learn from data or make predictions. PyTorch makes it easy and fast to do these math operations, which are the foundation of all machine learning and AI. If we didn't have tools like PyTorch, building intelligent systems would be much harder and slower, limiting progress in areas like speech recognition, image understanding, and recommendation systems.

Where it fits

Before learning PyTorch computations, you should understand basic Python programming and simple math with arrays or lists. After mastering first computations, you can learn how to build neural networks, train models, and use GPUs to speed up calculations.

Mental Model

Core Idea

PyTorch lets you create and manipulate multi-dimensional arrays called tensors to perform fast math operations that power machine learning.

Think of it like...

Imagine tensors as flexible Lego blocks that can be stacked and connected in many ways, and PyTorch as the instruction manual that tells you how to snap these blocks together to build something useful.

Tensor (3D example):
┌─────────────┐
│ Layer 1     │
│ ┌───────┐   │
│ │ 1 2 3 │   │
│ │ 4 5 6 │   │
│ └───────┘   │
│ Layer 2     │
│ ┌───────┐   │
│ │ 7 8 9 │   │
│ │10 11 12│   │
│ └───────┘   │
└─────────────┘

Operations: add, multiply, etc. on these blocks.

Build-Up - 7 Steps

1

Foundation

Understanding Tensors as Arrays

Concept: Tensors are the main data structure in PyTorch, similar to arrays or lists but can have many dimensions.

In PyTorch, a tensor is like a container holding numbers arranged in rows, columns, or more dimensions. For example, a 1D tensor is like a list of numbers, a 2D tensor is like a table, and higher dimensions stack these tables. You can create tensors from Python lists using torch.tensor().

Result

You can create tensors of different shapes and see their contents and dimensions.

Understanding tensors as multi-dimensional arrays helps you grasp how data is stored and manipulated in PyTorch.

2

Foundation

Creating Your First Tensor

Concept: Learn how to create tensors with numbers and check their properties.

Use torch.tensor() to make a tensor from a Python list. You can check its shape (dimensions) and data type. For example: import torch x = torch.tensor([1, 2, 3]) print(x) print(x.shape) print(x.dtype)

Result

Output shows the tensor values, shape as (3,), and data type as int64.

Knowing how to create tensors and inspect them is the first step to using PyTorch effectively.

3

Intermediate

Performing Basic Tensor Operations

🤔Before reading on: do you think adding two tensors changes their original values or creates a new tensor? Commit to your answer.

Concept: You can do math operations like addition, subtraction, and multiplication on tensors, which create new tensors without changing the originals.

Example: import torch x = torch.tensor([1, 2, 3]) y = torch.tensor([4, 5, 6]) z = x + y print(z) print(x) # original x stays the same

Result

z is tensor([5, 7, 9]) and x remains tensor([1, 2, 3])

Understanding that tensor operations produce new tensors without modifying inputs prevents bugs and helps with clear code.

4

Intermediate

Using In-place Operations on Tensors

🤔Before reading on: do you think in-place operations save memory or risk unexpected bugs? Commit to your answer.

Concept: In-place operations modify the tensor directly, saving memory but can cause issues if not used carefully.

Example: import torch x = torch.tensor([1, 2, 3]) x.add_(5) # adds 5 to each element in-place print(x)

Result

x becomes tensor([6, 7, 8]) after in-place addition

Knowing when to use in-place operations helps optimize memory but requires caution to avoid unintended side effects.

5

Intermediate

Tensor Shapes and Broadcasting Rules

🤔Before reading on: do you think PyTorch can add tensors of different shapes automatically? Commit to your answer.

Concept: PyTorch can automatically expand smaller tensors to match larger ones in operations, called broadcasting, following specific rules.

Example: import torch x = torch.tensor([[1, 2, 3], [4, 5, 6]]) # shape (2,3) y = torch.tensor([10, 20, 30]) # shape (3,) z = x + y print(z)

Result

z is tensor([[11, 22, 33], [14, 25, 36]]) where y is broadcasted to match x's shape

Understanding broadcasting allows you to write concise code without manually reshaping tensors.

6

Advanced

Using GPU for Tensor Computations

🤔Before reading on: do you think tensors automatically use GPU if available or need explicit commands? Commit to your answer.

Concept: PyTorch requires explicit commands to move tensors to GPU for faster computation.

Example: import torch if torch.cuda.is_available(): device = torch.device('cuda') x = torch.tensor([1, 2, 3], device=device) y = torch.tensor([4, 5, 6], device=device) z = x + y print(z) else: print('GPU not available')

Result

If GPU is available, z is computed on GPU and printed; otherwise, a message shows.

Knowing how to use GPU explicitly unlocks PyTorch's full speed potential for large computations.

7

Expert

Autograd: Tracking Computations for Gradients

🤔Before reading on: do you think PyTorch tracks all operations automatically or requires manual setup? Commit to your answer.

Concept: PyTorch automatically records operations on tensors with requires_grad=True to compute gradients for learning.

Example: import torch x = torch.tensor([2.0, 3.0], requires_grad=True) y = x * x + 3 z = y.sum() z.backward() print(x.grad) # gradients of z with respect to x

Result

x.grad is tensor([4.0, 6.0]) showing derivatives dz/dx

Understanding autograd reveals how PyTorch powers learning by automatically computing derivatives behind the scenes.

Under the Hood

PyTorch uses a dynamic computation graph that records operations on tensors as they happen. Each tensor with requires_grad=True tracks its history of operations. When backward() is called, PyTorch traverses this graph in reverse to compute gradients using the chain rule. Tensors are stored in memory with metadata about shape, type, device (CPU/GPU), and gradient info. Operations are implemented in optimized C++ and CUDA code for speed.

Why designed this way?

PyTorch was designed for flexibility and ease of use, allowing dynamic graphs that change every run, unlike static graphs in older frameworks. This makes debugging and experimenting easier. The design balances speed with Python's simplicity, enabling researchers and developers to write intuitive code that runs efficiently on CPUs and GPUs.

┌───────────────┐
│ Input Tensors │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│  Operations   │
│ (add, mul, etc)│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Computation   │
│   Graph       │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Backward Pass │
│ (gradients)   │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does adding two tensors always change the original tensors? Commit to yes or no.

Common Belief: Adding two tensors changes the original tensors involved.

Reality: Tensor addition creates a new tensor and does not modify the original tensors unless an in-place operation is used.

Why it matters: Assuming originals change can cause bugs where data is unexpectedly altered, leading to wrong results or hard-to-find errors.

Quick: Do you think PyTorch automatically uses GPU for all tensor operations if available? Commit to yes or no.

Common Belief: PyTorch automatically runs all tensor operations on GPU if a GPU is present.

Reality: Tensors and models must be explicitly moved to GPU using .to('cuda') or device arguments; otherwise, operations run on CPU.

Why it matters: Expecting automatic GPU use can cause slow code and confusion when performance is poor.

Quick: Is it true that PyTorch tensors always require gradients by default? Commit to yes or no.

Common Belief: All PyTorch tensors track gradients automatically for learning.

Reality: By default, tensors do not track gradients; requires_grad=True must be set to enable autograd.

Why it matters: Not setting requires_grad leads to no gradient computation, so learning algorithms won't work.

Quick: Do you think broadcasting works with any tensor shapes? Commit to yes or no.

Common Belief: PyTorch can broadcast tensors of any shapes for operations.

Reality: Broadcasting follows strict rules; incompatible shapes cause errors.

Why it matters: Misunderstanding broadcasting leads to runtime errors or incorrect calculations.

Expert Zone

1

PyTorch's dynamic graph allows conditional code and loops in model definitions, unlike static graph frameworks.

2

In-place operations can save memory but may interfere with gradient computation if used carelessly.

3

Tensors on different devices (CPU vs GPU) cannot interact directly; explicit device management is crucial in complex systems.

When NOT to use

For very large-scale distributed training, frameworks like TensorFlow with static graphs or specialized libraries like DeepSpeed may be better. Also, if you need extremely low-level control over hardware, custom CUDA kernels might be preferred.

Production Patterns

In production, PyTorch models are often exported to TorchScript for optimized deployment. Mixed precision training is used to speed up training while saving memory. Data loading and preprocessing pipelines are carefully designed to feed tensors efficiently to the model.

Connections

NumPy Arrays

PyTorch tensors are similar to NumPy arrays but support GPU and autograd.

Knowing NumPy helps understand tensor operations, but PyTorch extends this with automatic differentiation and hardware acceleration.

Calculus - Chain Rule

Autograd uses the chain rule from calculus to compute gradients automatically.

Understanding the chain rule clarifies how PyTorch computes derivatives for learning.

Spreadsheet Formulas

Tensor operations are like spreadsheet formulas that update values based on others.

This connection helps grasp how changing one tensor affects others through operations, similar to linked cells in a spreadsheet.

Common Pitfalls

#1 Trying to add tensors on different devices without moving them.

Wrong approach:

import torch
x = torch.tensor([1, 2, 3])
y = torch.tensor([4, 5, 6], device='cuda')
z = x + y # error here

Correct approach:

import torch
device = 'cuda' if torch.cuda.is_available() else 'cpu'
x = torch.tensor([1, 2, 3], device=device)
y = torch.tensor([4, 5, 6], device=device)
z = x + y

Root cause:

Tensors must be on the same device to perform operations; mixing CPU and GPU tensors causes errors.

#2 Assuming tensor operations modify the original tensors.

Wrong approach:

import torch
x = torch.tensor([1, 2, 3])
x + 5
print(x) # expecting x to change

Correct approach:

import torch
x = torch.tensor([1, 2, 3])
y = x + 5
print(y) # new tensor with added values
print(x) # original unchanged

Root cause:

Tensor operations return new tensors unless explicitly done in-place.

#3 Not setting requires_grad=True when gradients are needed.

Wrong approach:

import torch
x = torch.tensor([2.0, 3.0])
y = x * x
z = y.sum()
z.backward() # error: no grad

Correct approach:

import torch
x = torch.tensor([2.0, 3.0], requires_grad=True)
y = x * x
z = y.sum()
z.backward()
print(x.grad)

Root cause:

Gradients are only tracked for tensors with requires_grad=True.

Key Takeaways

PyTorch tensors are multi-dimensional arrays that hold data for machine learning.

Basic tensor operations create new tensors and do not change originals unless done in-place.

PyTorch requires explicit commands to use GPU for faster computations.

Autograd automatically tracks operations on tensors with requires_grad=True to compute gradients.

Understanding tensor shapes and broadcasting rules is essential for writing efficient PyTorch code.