10Introduction to Push-Forward Generative Models – Generative Adversarial Networks (GANs)
Author
Mark Fuge
Published
October 12, 2025
In this chapter, we will build geometric intuition for push-forward generative models, which are essentially trying to transport probability distributions from one space to another. We will start with simple linear maps to gain a sense of how they modify a simple 2D data space, then scale up to a small MLP GAN on a ring-of-Gaussians toy dataset. We will also talk about common metrics to evaluate generative models. Even though GANs may be comparatively simple compared to more advanced generative models that we will explore later, they will nevertheless be useful in building intuition for how push-forward generative models work.
Learning Objectives
By the end of this chapter, you should be able to:
Describe a generative model as a push-forward map from a latent distribution, and say which of the three operations (forward map, inverse map, density) a GAN actually does.
Map the terms of the GAN objective to the discriminator and generator steps of the training loop.
Explain how each network’s update depends on the other: the generator moves its samples up the discriminator’s logit in an attempt to fool the discriminator, and the discriminator reshapes its logit around the current samples in an attempt to classify them correctly.
Explain why a single summary statistic of the samples, such as the chapter’s variance-based diversity, cannot certify coverage or quality.
Check conditional generated samples against the engineering requirement with a simulator and a constraint check.
Show Code
# Setup and Imports (shared utilities)import numpy as npimport torchimport torch.nn as nnimport torch.optim as optimimport osfrom matplotlib import pyplot as plt# Import shared utilities from the local modulefrom gen_models_utilities import ( RNG_SEED, device, _compute_axis_limits, create_ring_gaussians, GanHistory, make_loader, compute_diversity_metric, plot_model_diagnostics, plot_latent_interpolation, make_conditional_loader_onehot, plot_conditional_samples_discrete)# get the location of this filemain_fpath = os.path.abspath('')os.chdir(main_fpath)# Optional interactive widgetstry:from ipywidgets import interact, FloatSlider, IntSliderexceptException: interact =None FloatSlider = IntSlider =Noneprint(f"Using device: {device}")
Using device: cuda
10.1 What are Generative Models really doing?
Fundamentally, generative models are really just functions that transform a probability distribution from one space to another – you can think of them as “distribution transformers” or, more intuitively, as moving probability mass around in space. As an analogy, consider that you are sitting in a sandbox with a smooth mound of sand in front of you – the 2D Gaussian that we will use below is not too far from this, actually. A generative model is kind of like your hands, which you can use to push the sand around, creating hills and valleys, and moving the sand from one place to another. In this way, we are essentially moving the probability mass (sand, in this analogy) from a simple distribution (the smooth mound of sand) to a more complex distribution (the hills and valleys that you create with your hands). Different models that we will explore later (e.g., VAEs, normalizing flows, diffusion models) have different ways of doing this, but the core goal is the same, and they essentially all try to do one or more of three operations:
\(f(z) \rightarrow x\): Mapping points from one space (typically called the latent space) to another (typically, the data space).
\(f^{-1}(x) \rightarrow z\): Mapping points from data space back to latent space.
\(p(x) \Leftrightarrow p(z)\): Mapping probability densities from data space to latent space (and vice versa).
As we go forward, we will see how different types of models are better or worse at these different operations, and, in some cases, how they can be combined to create more powerful models.
10.2 Basic Deterministic Push-Forward Models: From Latent Space to Data Space
A push-forward generative model defines a mapping \(x = f(z)\), where \(z\) is sampled from a simple latent distribution (often a standard Gaussian, though it need not be) and \(f\) is a deterministic function (e.g., linear map or neural network).
We’ll start with a 2D latent \(z \sim N(0, I)\) (i.e., a standard Gaussian) and inspect how different choices of \(f\) reshape the distribution in data space. In this case, the latent space is 2D and the data space is also 2D, so we can visualize both spaces directly, however, in general, the latent space is often lower-dimensional than the data space.
# Linear push-forward demo with interactive slidersdef linear_pushforward_demo(W: np.ndarray, b: np.ndarray, n: int=1000, seed: int= RNG_SEED): rng = np.random.default_rng(seed) z = rng.standard_normal(size=(n, 2)) x = (z @ W.T) + breturn z, x# Default transformW0 = np.array([[1.2, 0.4],[0.0, 0.8]], dtype=float)b0 = np.array([0.0, 0.0], dtype=float)z_lin, x_lin = linear_pushforward_demo(W0, b0, n=2000)plot_pushforward(z_lin, x_lin, title='x = W z + b')
We can see from above that the simple linear map has slightly “moved” or shifted the location of the probability mass, just like if you were sculpting sand in your sandbox. You can play below with some of the sliders to manipulate the simple 2x2 weight matrix and 2x1 bias vector to see how this works interactively:
OK, now that we have some intuition for how simple linear maps can move probability mass around in space, let’s try a more complex example where it is not so clear that a simple linear map will be sufficient. Below, we will create a common toy dataset consisting of a “ring of Gaussians” – that is, several Gaussian blobs arranged in a circle. You can see from above that no combination of weights in a simple linear map will be able to move our original probability mass from a standard Gaussian to this ring of Gaussians, so we will need something more powerful. A simple starting point for this is something called a Generative Adversarial Network (GAN), which we will explore next.
10.4 What is a Generative Adversarial Network (GAN)?
A Generative Adversarial Network (GAN) is a framework for training generative models using two competing neural networks: a generator and a discriminator. The generator, \(G\), learns to map random noise from a latent space (e.g., \(z \sim p_z(z)\) or \(f(z)=x\), where \(G\) functions as \(f\) here) to data space, aiming to produce samples that resemble the real data. It is tempting to want to train just the Generator by minimizing Mean Squared Error between \(f(z)\) and a real sample \(x\), except we have one big problem for right now – we do not know a priori which samples of \(x\) correspond to which samples of \(z\), and so it is not straightforward to compute this MSE.1 Instead, what we will do is train a separate network, the Discriminator, that can help us push the samples produced by the Generator closer to the real data distribution.
1 Indeed, we will return to this idea in later notebooks once we have introduced the concept of Optimal Transport, which solves this mapping problem, but for now, let’s assume that we don’t know how to do this.
Specifically, the discriminator, \(D\), tries to distinguish between real data samples and those produced by the generator and our loss function will encourage the discriminator to become better and better at this task. In turn, the generator will be trained to produce samples that the discriminator classifies as real – that is to try to fool the discriminator.
The two networks are trained simultaneously in something called a minimax game:
The generator tries to “fool” the discriminator by generating realistic samples.
The discriminator tries to correctly classify real vs. generated samples.
10.4.1 Mathematical Formulation
The standard GAN objective, as introduced by Goodfellow et al. (2014), is:
\(p_{\text{data}}(x)\): Distribution of real data.
\(p_z(z)\): Prior distribution over latent variables (often standard normal).
\(G(z)\): Generator’s output given latent input \(z\).
\(D(x)\): Discriminator’s estimate of the probability that \(x\) is real.
The generator and discriminator are typically neural networks trained with stochastic gradient descent. The generator improves by producing samples that the discriminator cannot distinguish from real data (the second term on the right-hand side of the equation above), while the discriminator improves by getting better at distinguishing real from fake (it maximizes both terms: scoring real data high and generated data low).
Two terms from classification will help us read the training code. A classifier that gives a probability \(p\) that its input is real can just as well give the logit of that probability, its log-odds:
The sigmoid \(\sigma\) is the inverse of the logit: it maps a logit \(s\) back to a probability. A logit of 0 means \(p = 0.5\), a positive logit means “more likely real” and a negative logit means “more likely fake”. A logit can take any real value, so networks usually output it instead of \(p\). Our discriminator does this too: its last layer outputs a logit \(\ell(x)\), and \(D(x) = \sigma(\ell(x))\).
A classifier like this is trained with the binary cross-entropy (BCE) between a label \(y \in \{0, 1\}\) and the predicted probability \(p\):
\[
\mathrm{BCE}(p, y) = -\left[ y \log p + (1-y) \log (1-p) \right].
\]
Only one of the two terms is active for each sample, so the BCE is \(-\log\) of the probability that the classifier gave to the correct label. It is 0 when the classifier is confident and right, and it grows without bound when the classifier is confident and wrong: \(p = 0.01\) for a real sample costs \(-\log 0.01 \approx 4.6\). This is the log loss from the classification losses in Reviewing Supervised Linear Models, written for labels 0 and 1 instead of \(\pm 1\).
In the training loop below, loss_d = bce(d_real, torch.ones_like(d_real)) + bce(d_fake, torch.zeros_like(d_fake)) labels the real samples 1 and the generated samples 0. Per sample, it is \(-\log D(x) - \log(1 - D(G(z)))\), which is minus the per-sample \(V(D, G)\). So minimizing this loss maximizes \(V\). nn.BCEWithLogitsLoss takes the logits d_real and d_fake and applies the sigmoid inside, which is numerically more stable than taking the log of a sigmoid in two steps.
In practice, the generator does not minimize the second term above directly. Early in training, the discriminator easily rejects fakes, so \(D(G(z)) \approx 0\). There, \(\log(1 - D(G(z)))\) is almost flat, and the generator gets almost no gradient. So instead of minimizing \(\mathbb{E}_{z}[\log(1 - D(G(z)))]\), the generator minimizes the non-saturating loss:
which has the same goal (push \(D(G(z))\) toward 1) but provides stronger gradients when the generator is doing badly. This is what loss_g = bce(dg, torch.ones_like(dg)) computes in the training loop below.
Goodfellow et al. proposed this switch in the original GAN paper, in its Section 3 (Goodfellow et al. 2014). The two generator losses have the same end point, \(p_G = p_{\text{data}}\) with \(D = 1/2\) everywhere, and both push \(D(G(z))\) toward 1. But they are not the same objective. With the best discriminator for the current generator, the original minimax objective equals \(-\log 4 + 2\,\mathrm{JS}(p_{\text{data}} \parallel p_G)\), so the generator minimizes the Jensen-Shannon (JS) divergence. The non-saturating loss instead follows the gradient of \(\mathrm{KL}(p_G \parallel p_{\text{data}}) - 2\,\mathrm{JS}(p_{\text{data}} \parallel p_G)\)(Arjovsky and Bottou 2017). Its reverse Kullback-Leibler (KL) term punishes samples where there are no data much more than it punishes parts of the data that get no samples. This is one reason why GANs tend to drop modes rather than produce garbage. We met both divergences, and the mode-seeking behavior of the reverse KL, in Measuring Distribution Distances.
Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, et al. 2014. “Generative Adversarial Nets.”Advances in Neural Information Processing Systems 27.
Arjovsky, Martin, and Léon Bottou. 2017. “Towards Principled Methods for Training Generative Adversarial Networks.”International Conference on Learning Representations.
10.5 A Simple Generative Adversarial Network (GAN)
Now we will define a small GAN and see how it moves probability mass around after passing samples through the Generator. You can feel free to modify the architecture below if you like and see how that impacts the below results, but for now we will use an MLP with two hidden layers and LeakyReLU activations for both the Generator and Discriminator.
With the models defined, we will now set up the training loop to optimize both the generator and discriminator.
def train_vanilla_gan( data: np.ndarray, *, noise_dim: int=2, batch_size: int=256, epochs: int=120, lr_g: float=2e-4, lr_d: float=2e-4, hidden_dim: int=256, print_every: int=40) ->tuple[nn.Module, nn.Module, GanHistory]:# Load the data into a DataLoader for batching and make PyTorch happy loader = make_loader(data, batch_size)# Set up the basic networks G = build_generator(noise_dim=noise_dim, hidden_dim=hidden_dim) D = build_discriminator(hidden_dim=hidden_dim)# Instantiate the optimizers for each model opt_g = optim.Adam(G.parameters(), lr=lr_g, betas=(0.5, 0.999)) opt_d = optim.Adam(D.parameters(), lr=lr_d, betas=(0.5, 0.999)) bce = nn.BCEWithLogitsLoss()# Record the loss history for plotting later hist = GanHistory([], [], [], [], [])# Now we do the training loop for # epochs defined in `epochs`for ep inrange(epochs): d_losses=[] g_losses=[] real_scores=[] fake_scores=[]for (xb,) in loader:# Send the data to the GPU, if using. xb = xb.to(device)# Take a Discriminator step z = torch.randn(xb.size(0), noise_dim, device=device)with torch.no_grad(): x_fake = G(z) opt_d.zero_grad() d_real = D(xb) d_fake = D(x_fake) loss_d = bce(d_real, torch.ones_like(d_real)) + bce(d_fake, torch.zeros_like(d_fake)) loss_d.backward() opt_d.step() d_losses.append(float(loss_d.detach().cpu().item())) real_scores.append(d_real.mean().item()) fake_scores.append(d_fake.mean().item())# Take a Generator step z = torch.randn(xb.size(0), noise_dim, device=device) opt_g.zero_grad() xg = G(z) dg = D(xg) loss_g = bce(dg, torch.ones_like(dg)) loss_g.backward() opt_g.step() g_losses.append(float(loss_g.detach().cpu().item()))# We'll record some epoch metricswith torch.no_grad(): z_eval = torch.randn(2048, noise_dim, device=device) samples = G(z_eval) div = compute_diversity_metric(samples)# Now we'll record the metrics for plotting later and reporting hist.d_loss.append(float(np.mean(d_losses))) hist.g_loss.append(float(np.mean(g_losses))) hist.diversity.append(div) hist.real_scores.append(float(np.mean(real_scores))) hist.fake_scores.append(float(np.mean(fake_scores)))if (ep+1) %max(1, print_every) ==0or ep==0:print(f"Epoch {ep+1:03d}/{epochs} | D {hist.d_loss[-1]:.3f} | G {hist.g_loss[-1]:.3f} | Div {div:.3f}")return G, D, hist
You can feel free to modify some of the training elements here, such as the epoch length or learning rates of the Generator or Discriminator, respectively.
Epoch 001/120 | D 1.128 | G 0.734 | Div 0.809
Epoch 040/120 | D 1.300 | G 0.769 | Div 4.310
Epoch 080/120 | D 1.340 | G 0.754 | Div 4.409
Epoch 120/120 | D 1.368 | G 0.732 | Div 4.444
OK, now the model is trained, so let’s look at some basic visualizations of how it did. In the below plots, which we will re-use for other models later on, we will show four things:
Upper Left: Any training losses for the model, as a function of training epoch. In the case of a GAN, this will plot both the Generator and Discriminator losses. This plot type allows us to assess something about the convergence and stability of training.
Upper Right: The Sample Diversity, as a function of Epoch. This is computing and plotting the variance of \(\mathbf{x}\) as a function of epoch, where \(\mathbf{x}\) are samples drawn from the Generator at each epoch. This plot type allows us one way to assess whether the model is suffering from mode collapse (i.e., low diversity in the generated samples) or not.
Lower Left: A plot of the real data samples (in light grey) and samples drawn from the Generator (in reddish-orange). This plot type allows us to visually assess how well the model is capturing the data distribution. This will be easy to compare in this simple 2D case, but will be harder in higher dimensions, and in those cases you might have to resort to just comparing selected samples or overall distribution summary statistics.
Lower Right: This plot will vary depending on the specific model we are studying, but in this case it will show the average raw output (the logit) of the Discriminator (often called the “Critic”) on real and on fake samples. A positive logit means the Discriminator thinks a sample is real, and a negative logit means it thinks the sample is fake. In the context of a GAN model, this allows us to assess how well a Generator is fooling the discriminator, since a well-trained Generator should produce samples that the Discriminator cannot tell apart from real ones, and thus both the “Real score” and “Fake score” should hover around 0, which corresponds to a probability of \(\sigma(0) = 0.5\) (i.e., the Discriminator is effectively guessing randomly).
In addition to the above training diagnostics, it can also be useful to plot interpolations in latent space to see how smoothly the Generator can move between different modes of the data distribution. In the case of a well-trained GAN, we would expect that interpolating between two points in latent space should produce a smooth transition in data space, moving through intermediate samples that also look realistic. In this case, this means hopping smoothly between the different Gaussian blobs in our ring-of-Gaussians dataset, and not jumping back-and-forth randomly between them.
To give you some interactive control on visualizing the forward mapping \(f(z) \rightarrow x\), use the sliders below to move a single latent vector (left-side plot) and observe the corresponding generated point in data space (right-side plot). By moving around in \(z\) you can try to align the generated point with a particular mode on the ring.
Show Code
if interact isnotNone:def _move_latent(z1: float=0.0, z2: float=0.0):""" Visualize the effect of moving a latent vector z = [z1, z2] through the generator. Left: latent space (z1, z2) with current point highlighted. Right: generated data point in data space, overlaid on the real data. """ z = torch.tensor([[z1, z2]], dtype=torch.float32, device=device)with torch.no_grad(): x = G(z).cpu().numpy()[0] fig, axes = plt.subplots(1, 2, figsize=(11, 5))# Left: latent space axes[0].scatter(0, 0, s=40, color='gray', alpha=0.2, label='Origin') axes[0].scatter(z1, z2, s=120, color='crimson', edgecolors='k', linewidths=0.5, label='Current z') axes[0].set_title('Latent Space (z)') axes[0].set_xlabel('z1') axes[0].set_ylabel('z2') axes[0].set_xlim(ring_latent_limits[0]) axes[0].set_ylim(ring_latent_limits[1]) axes[0].grid(True, alpha=0.3) axes[0].legend(loc='upper left', frameon=False)# Right: data space axes[1].scatter(X_ring[:,0], X_ring[:,1], s=10, alpha=0.15, color='gray', label='Real') axes[1].scatter([x[0]], [x[1]], s=120, color='crimson', edgecolors='k', linewidths=0.5, label='Generated') axes[1].set_title('Generated Data Point') axes[1].set_xlabel('x1') axes[1].set_ylabel('x2') axes[1].set_xlim(ring_data_limits[0]) axes[1].set_ylim(ring_data_limits[1]) axes[1].grid(True, alpha=0.3) axes[1].legend(loc='upper left', frameon=False) plt.tight_layout() plt.show() interact( _move_latent, z1=FloatSlider(min=-3.0, max=3.0, step=0.05, value=0.0, description='z1'), z2=FloatSlider(min=-3.0, max=3.0, step=0.05, value=0.0, description='z2'), )else:print('ipywidgets not available; skipping latent sliders.')
10.7 What the Discriminator Looks Like
What you will notice from the above code is that the generator isn’t really concerned with where the actual data is located. We can see this by noticing that in either loss formulation it only ever depends on \(\mathbb{E}_{z \sim p_{(z)}}\), and never on \(\mathbb{E}_{x \sim p_{\text{data}}(x)}\). Because of this, no gradients for the generator actually depend directly on the input data, and the only gradients that affect the generator come from its interplay with the discriminator! Look at the training loop again: xb only enters the discriminator step. The only thing that reaches the generator is the gradient of its loss, \(\mathcal{L}_G = -\log \sigma(\ell(G(z)))\), where \(\ell(G(z))\) is the discriminator’s logit at \(G(z)\). That gradient flows back through the discriminator into each fake sample. So to gain a better understanding of what the generator learns, we can look at the discriminator’s logit as a map over the whole data space.
What would a perfect discriminator draw on that map? For a fixed generator with sample density \(p_G\), the discriminator that maximizes \(V(D, G)\) is
The logit is positive where the data have more mass than the generator and negative where the generator puts too much mass. Each fake sample moves in the direction of \(-\nabla_x \mathcal{L}_G = (1 - \sigma(\ell))\, \nabla_x \ell\): essentially trying to move samples such that they increase in probability w.r.t the discriminator’s logit. Below we draw the logit as a color map (red is higher likelihood of the discriminator thinking the point is real, blue is where the discriminator thinks the point might be fake, and the black line is \(\ell = 0\)), the real data is shown in gray, the generator’s samples are shown in orange, and optionally the green arrow shows the direction each generated sample would move to “make it more realistic” with respect to the discriminator.
Show Code
import copy# A fixed set of latent vectors, so every map below shows the same generator samplesZ_SHOW = torch.randn(500, 2, generator=torch.Generator().manual_seed(RNG_SEED)).to(device)def generator_push(D_net, x_fake):# Direction that lowers the generator loss -log(sigmoid(logit)) at each fake sample x = x_fake.clone().requires_grad_(True) loss_g = nn.functional.binary_cross_entropy_with_logits(D_net(x), torch.ones(len(x), device=device), reduction='sum') loss_g.backward()return-x.graddef plot_discriminator_map(D_net, G_net, ax=None, arrows=False, n_arrows=150, lim=None, title='', X_real=None):# D's logit over the data space, the real data (the ring unless X_real is given),# the generator's samples and (optionally) their pushif X_real isNone: X_real = X_ringif ax isNone: fig, ax = plt.subplots(figsize=(6.5, 5.5))with torch.no_grad(): x_fake = G_net(Z_SHOW) fake_np = x_fake.cpu().numpy()if lim isNone:# Show at least [-5, 5] x [-5, 5], and widen the window if the generator's samples lie outside it lim =max(5.0, 1.1* np.percentile(np.abs(fake_np).max(axis=1), 99)) grid_1d = np.linspace(-lim, lim, 201) xx, yy = np.meshgrid(grid_1d, grid_1d) grid = torch.tensor(np.stack([xx.ravel(), yy.ravel()], axis=1), dtype=torch.float32, device=device)with torch.no_grad(): logit = D_net(grid).cpu().numpy().reshape(xx.shape) v = np.abs(logit).max() im = ax.contourf(xx, yy, logit, levels=np.linspace(-v, v, 31), cmap='RdBu_r') ax.contour(xx, yy, logit, levels=[0], colors='k', linewidths=0.8) ax.scatter(X_real[:800, 0], X_real[:800, 1], s=4, color='0.35', alpha=0.4, label='Real') ax.scatter(fake_np[:, 0], fake_np[:, 1], s=6, color='tab:orange', label='Generated')if arrows: push = generator_push(D_net, x_fake[:n_arrows]).cpu().numpy()# The longest arrow is 16% of the window's half-width; only directions and relative lengths matter ax.quiver(fake_np[:n_arrows, 0], fake_np[:n_arrows, 1], push[:, 0], push[:, 1], color='darkgreen', angles='xy', scale_units='xy', scale=np.linalg.norm(push, axis=1).max() / (0.16* lim), width=0.004) ax.set_xlim(-lim, lim) ax.set_ylim(-lim, lim) ax.set_aspect('equal') ax.set_xlabel('x1') ax.set_ylabel('x2') ax.set_title(title, fontsize=11) plt.colorbar(im, ax=ax, fraction=0.046, label='Discriminator logit')return ax
Let’s look at the map early in training, when the generator is still bad. The cell below trains a fresh generator and discriminator for only 2 epochs with the chapter’s settings and keeps them as G_snap and D_snap (the chapter’s G and D stay as they are).
Epoch 001/2 | D 1.128 | G 0.734 | Div 0.809
Epoch 002/2 | D 1.040 | G 0.874 | Div 2.533
After two epochs the generator’s samples lie on an arc across the upper half of the ring, and the discriminator’s logit is negative (blue) around them. The logit is largest at the bottom left, where five of the eight modes have real data but no samples. Yet the green arrows do not point there, and many of them do not point at the nearest real cluster either. They point up the local slope of the logit. Along most of the arc that is upward and to the right, out of the blue basin over its nearest rim; at the right end of the arc it is down and to the right. The generator only uses the slope of the logit where it generated samples, and thus cannot see that there is a much higher slope toward the lower left corner.
In the training loop the two players take turns: one discriminator step, then one generator step. The function below continues training from the snapshot with only one player, or with both in turns, for a given number of steps. It works on copies, so G_snap and D_snap stay as they are.
Use the controls below to choose which player trains and for how many steps, starting from the snapshot. The default trains both players in turns for 200 steps, as the chapter’s loop does.
With both players taking turns, the samples spread over the ring within 200 steps, and the logit shrinks to within about \(\pm 1\) (look at the color bar). Try the other two settings and different numbers of steps. What happens when only one of the two players learns?
10.8 Conditional Generation Examples
So far, we’ve explored unconditional generative models that learn to produce samples from the overall data distribution. However, in many practical applications, we want to control what we generate by conditioning on additional information. For example, we might want to generate samples from a specific mode of the distribution, or generate samples with particular properties.
In this section, we’ll explore two types of conditioning:
Discrete Conditioning (One-Hot Mode Index): Condition the GAN on which Gaussian mode in our ring dataset the sample should come from.
Continuous Conditioning (Radius): Condition the GAN on a continuous scalar representing the radius from the origin.
These conditioning strategies are widely applicable and form the basis for more complex conditional generation tasks, such as class-conditional image generation or text-to-image synthesis.
10.8.1 Conditional Example A — One-Hot Mode Conditioning
In this example, we’ll condition the GAN on which of the 8 Gaussian modes in the ring dataset we want to generate from. We provide a one-hot encoded vector indicating the target mode to both the generator and discriminator.
The key idea is simple: we concatenate the one-hot label to the inputs of both networks. The generator receives both random noise \(z\) and the label \(y\), learning to map \((z, y) \rightarrow x\). The discriminator receives both the data point \(x\) and the label \(y\), learning to distinguish real pairs \((x, y)\) from fake pairs \((G(z, y), y)\).
Epoch 001/480 | D 1.272 | G 0.720 | Div 0.018
Epoch 040/480 | D 1.403 | G 0.692 | Div 4.806
Epoch 080/480 | D 1.387 | G 0.691 | Div 4.508
Epoch 120/480 | D 1.387 | G 0.694 | Div 4.579
Epoch 160/480 | D 1.382 | G 0.682 | Div 4.723
Epoch 200/480 | D 1.393 | G 0.672 | Div 4.008
Epoch 240/480 | D 1.385 | G 0.705 | Div 4.374
Epoch 280/480 | D 1.389 | G 0.711 | Div 4.745
Epoch 320/480 | D 1.388 | G 0.683 | Div 4.019
Epoch 360/480 | D 1.385 | G 0.691 | Div 4.852
Epoch 400/480 | D 1.384 | G 0.690 | Div 4.930
Epoch 440/480 | D 1.388 | G 0.697 | Div 4.347
Epoch 480/480 | D 1.388 | G 0.688 | Div 4.380
Now let’s visualize the conditional samples. We’ll generate samples for each of the 8 modes and plot them to see if the generator has learned to respect the conditioning signal.
10.8.2 Conditional Example B — Continuous Radius Conditioning
Instead of conditioning on discrete labels, we can also condition on continuous scalar values. In this example, we’ll condition the GAN on the radius (distance from the origin) of the generated samples. This allows us to control how far from the origin samples are generated.
To handle continuous conditioning, we typically embed the scalar through a small MLP before concatenating it with other inputs. This gives the network more flexibility in how it uses the conditioning signal.
Epoch 001/480 | D 1.271 | G 0.751 | Div 0.014
Epoch 040/480 | D 1.287 | G 0.766 | Div 4.688
Epoch 080/480 | D 1.288 | G 0.767 | Div 4.356
Epoch 120/480 | D 1.306 | G 0.772 | Div 4.473
Epoch 160/480 | D 1.335 | G 0.758 | Div 4.527
Epoch 200/480 | D 1.352 | G 0.745 | Div 4.411
Epoch 240/480 | D 1.361 | G 0.734 | Div 4.652
Epoch 280/480 | D 1.364 | G 0.728 | Div 4.435
Epoch 320/480 | D 1.365 | G 0.721 | Div 4.446
Epoch 360/480 | D 1.365 | G 0.728 | Div 4.633
Epoch 400/480 | D 1.364 | G 0.733 | Div 4.670
Epoch 440/480 | D 1.366 | G 0.722 | Div 4.713
Epoch 480/480 | D 1.366 | G 0.730 | Div 4.461
Now let’s generate samples at different radius values to see how the generator responds to continuous conditioning. We’ll sweep through several radius values and visualize the results.
OK, so we have seen how a basic GAN is set up and trained, and covered some diagnostic plots that give us insight into how the training went. This gave us the following sets of useful tools:
We could learn a forward mapping function \(f(z) \rightarrow x\) that could push forward a given sample in \(z\) to a corresponding sample in \(x\). This could learn distributions that were far more complex than a simple linear map could achieve above.
This \(f\) is a deterministic function – i.e., the same \(z\) will always produce the same \(x\).
We could produce latent interpolations in \(z\) that produced smooth transitions in \(x\) and seemed to capture the overall clustering behavior of certain distributions.
Unfortunately, GANs did not give us any of the following properties, which will turn out to be useful later and motivate other types of generative models:
We did not learn an inverse mapping \(f^{-1}(x) \rightarrow z\). This means that we cannot easily encode real data samples into the latent space, which would be useful for many applications. For example, if I wanted to slightly modify \(x\) – e.g., \(x^\prime = x+\delta x\) – and compute the corresponding latent coordinate \(z^\prime = z+\delta z\), GANs do not provide a built-in mechanism to do this.2 We will see how both Variational Autoencoders (VAEs) and Normalizing Flows (NFs) address this problem.
We also cannot yet easily talk about or compute probability mappings between \(p(x)\) and \(p(z)\), which would be useful for computing likelihoods of data samples or for evaluating how well the model fits the data distribution, both of which are quite useful. Again, we will see later how both VAEs and NFs address this problem.
2 Of course, I could try to get at this another way, for example, by applying automatic differentiation to the Generator to compute \(\frac{\partial f}{\partial z}\) and then use that to estimate \(\delta z \approx (\frac{\partial f}{\partial z})^{-1} \delta x\), but this requires extra work on our part and this is not always straightforward, especially if the Jacobian is ill-conditioned.
Before we move beyond GANs, however, we will first explore some of the complex training dynamics and other subtle issues that arise when applying them in practice. This will help you build up your intuition and experience in how to assess problems in generative models and will serve as a strong basis upon which to launch into more complex models later. It will also give us an opportunity to introduce the concept of Optimal Transport, which will be useful in its own right later on, and will also help us understand some of the limitations of GANs should you encounter their use in the future.
10.10 Exercises
The exercises reuse the chapter’s data, networks and training code, so run the chapter’s cells first. Rough time estimates are in the titles.
Exercise 1 uses a new data set, two moons, and follows single updates of the two networks. The cell below scales the two-moons data set from scikit-learn to about the size of the ring and trains the chapter’s GAN on it with train_vanilla_gan for only 3 epochs. Run it to produce three figures, all drawn with plot_discriminator_map:
The discriminator’s logit and the generator’s samples after these 3 epochs.
Panels A to D, which put an arrow on 80 of these samples to demonstrate the direction the generated sample would move.
The same discriminator again, and three candidate discriminators, A to C. Each candidate starts from that discriminator and takes a short burst of 25 discriminator steps.
Every map has its own color scale.
Epoch 001/3 | D 1.261 | G 0.728 | Div 0.621
Epoch 003/3 | D 1.467 | G 0.805 | Div 2.755
Epoch 001/5 | D 1.261 | G 0.728 | Div 0.621
Epoch 005/5 | D 1.465 | G 0.791 | Div 1.170
NoteExercise 1 (~5 min): One step at a time
In the chapter’s training loop, each network takes one step at a time against the other network as it is at that moment.
Which panel of the second figure shows the direction in which each sample moves when the generator takes a step against the discriminator shown? Why? Why are the arrows longer in some places than in others?
Which candidate in the third figure is the discriminator after 25 more discriminator steps against the samples shown? Why do the other two not fit?
Show Code
# You can use this code cell for this exercise, if you like.
Run the cell below to produce the figure for Exercise 2. It draws seven sets of 2048 samples each on top of the ring data (gray). The four ring sets use the ring’s own mode centers and spread; the Gaussian and the disk are centered at the origin; the last set comes from the chapter’s trained G. The chapter’s diversity score of each set, compute_diversity_metric, is printed in a collapsed box after the exercise.
NoteExercise 2 (~5 min): Seven generators, one score
The chapter tracks the diversity of the generated samples, the per-axis variance averaged over the two axes, as a check against mode collapse.
Before you open the scores below, rank the seven sets by how good a generator of the ring each would be, and predict their diversity scores. Why did you rank them that way?
Five very different sets score about 4.5. Why? What does the metric actually measure, and why does it come out nearly the same for 2, 4 and 8 ring modes?
The chapter’s G scores below the Gaussian blob. Does that make it the worse generator? Why, and what would you need to look at instead?
TipDiversity scores (open after you have made your predictions)
Show Code
print(f"{'Sample set':28s}{'diversity':>9s}")for name, s in sample_sets.items():print(f"{name:28s}{compute_diversity_metric(torch.from_numpy(np.asarray(s, dtype=np.float32))):9.2f}")
Sample set diversity
Full ring 4.56
Two opposite modes 4.54
Four modes 4.57
One mode 0.06
Gaussian, std 2.14 4.69
Uniform disk, radius 4.27 4.58
The chapter's G 4.42
Exercise 3 moves from the ring to an engineering data set. Beams2D, from the EngiBench benchmark, contains beam designs optimized by topology optimization. Each design is a \(50 \times 100\) grid of material densities between 0 (void) and 1 (solid). It is the right half of a symmetric, simply supported beam: the left edge is the symmetry line, a roller supports the bottom-right corner, and a downward load acts on the top edge at the red triangle. Each design minimizes the compliance (how much the beam deflects under its load, so lower is stiffer) for four conditions: the target volume fraction (the share of the grid filled with material), the minimum feature size rmin, the load position forcedist (0 at the left edge, 1 at the right) and an overhang flag.
The cell below loads a conditional GAN generator, CGANGenerator from engibench_gen_models.py, that was trained on the Beams2D training designs. It takes the four conditions and 32 noise values and returns a design. For three rows of the validation set, which the generator never saw, the cell draws the dataset’s optimal design next to three generated designs, each from a different noise vector.
Show Code
from engibench.problems.beams2d import Beams2Dfrom engibench_gen_models import CGANGeneratorBEAMS_GENERATOR_PATH ='../../data/beams2d_cgan_generator.pth'EX3_ROWS = [178, 708, 494] # rows of the Beams2D validation set used in Exercise 3beams = Beams2D()beam_keys = beams.conditions_keys # volfrac, rmin, forcedist, overhang_constraintbeams_val = beams.dataset['val']# latent_dim=32 as in the training run that produced the weights (beams2d_cgan_cnn_2d, seed 1)G_beams = CGANGenerator(latent_dim=32, n_conds=len(beam_keys), design_shape=(50, 100))G_beams.load_state_dict(torch.load(BEAMS_GENERATOR_PATH, map_location='cpu'))G_beams.eval()torch.manual_seed(RNG_SEED)z_beams = torch.randn(3, 32, 1, 1) # three noise vectors, used for every rowbeam_conditions, beam_optima, beam_designs = [], [], []for i in EX3_ROWS: cond = {k: beams_val[k][i] for k in beam_keys} c = torch.tensor([[float(cond[k]) for k in beam_keys]] *3, dtype=torch.float32).view(3, -1, 1, 1)with torch.no_grad():# The generator ends in tanh, so map its output from [-1, 1] to densities in [0, 1] designs = ((G_beams(z_beams, c) +1) /2).numpy().reshape(3, 50, 100) beam_conditions.append(cond) beam_optima.append(np.array(beams_val['optimal_design'][i])) beam_designs.append(designs)fig, axes = plt.subplots(len(EX3_ROWS), 4, figsize=(16, 2.6*len(EX3_ROWS)))for r, (i, cond) inenumerate(zip(EX3_ROWS, beam_conditions)): images = [beam_optima[r]] +list(beam_designs[r]) titles = ['Optimal design (dataset)', 'Generated, noise 1', 'Generated, noise 2', 'Generated, noise 3']for ax, im, title inzip(axes[r], images, titles): ax.imshow(im, cmap='gray_r', vmin=0, vmax=1, extent=(0, 100, 50, 0)) ax.plot([100* cond['forcedist']], [-2.5], marker='v', color='crimson', markersize=10, clip_on=False) # just above the top edge, so it hides no elements ax.set_title(title, fontsize=11) ax.set_xticks([]) ax.set_yticks([]) axes[r, 0].set_ylabel(f"Row {i}\nvolume fraction {cond['volfrac']:.3f}", fontsize=11)plt.tight_layout()plt.show()
NoteExercise 3 (~10 min): It looks like a beam
A generated design is only useful if it supports its load as well as the optimal design does.
From the pictures alone, predict for each row how the compliance of the generated designs compares with the optimal design’s: about the same, a few times larger, or far larger. Why?
Run the cell below. For every design it prints the volume fraction, whether the design passes beams.check_constraints (which checks that the volume fraction is within 0.01 of the target), its compliance from simulate, and the ratio to the optimal design’s compliance. Were your predictions right? Why or why not? Which designs would the constraint check alone have rejected, and why is that check not enough?
What broke the worst designs? The defect is small, so plot zoomed parts of the generated designs of rows 708 and 494 and of their optima (beam_designs and beam_optima) in the scratch cell below. Why does a defect of this size change the compliance by a factor of hundreds to thousands?
The discriminator that trained this generator also saw the four conditions. Why did it not penalize these designs?
The three designs in each row come from three different noise vectors. Why are they so alike, and what does that mean if you hope to fix a failed design by drawing more samples?
Show Code
print(f"{'Row':>4}{'design':>10}{'vol. frac.':>10}{'target':>7}{'constraints':>12}{'compliance':>11}{'ratio to optimum':>17}")for r, (i, cond) inenumerate(zip(EX3_ROWS, beam_conditions)):# Beams2D.simulate sets up its load case on the first call and keeps it,# so each condition gets a fresh Beams2D object. simulator = Beams2D() c_opt =float(simulator.simulate(beam_optima[r], config=cond)[0])for name, design in [('optimum', beam_optima[r])] + [(f'noise {k +1}', d) for k, d inenumerate(beam_designs[r])]: violations = beams.check_constraints(design, config=cond) compliance =float(simulator.simulate(design, config=cond)[0])print(f"{i:>4}{name:>10}{design.mean():>10.3f}{cond['volfrac']:>7.3f} "f"{'pass'ifnot violations else'fail':>12}{compliance:>11.1f}{compliance / c_opt:>17.3g}")
