Generative Perplexity

discrete_diffusion.evaluations.generative_ppl

Standalone Generative Perplexity evaluation.

Loads generated samples and evaluates NLL/PPL with a chosen eval LM.

Supported sample formats: - .pt: torch.Tensor of token ids (shape [N, T] or [N, 1, T]) - .npz: numpy array under key 'samples' (shape [N, T]) - .json: contains base64-encoded numpy array under key 'np_tokens_b64'

This script decodes using model_tokenizer and (optionally) retokenizes with the eval model's tokenizer before computing loss.