Particles with Optical Flows

Introduction

Baked billboard animated particles often suffer from the discreet transitions, especially if time slow motion is present or . There is a known method of solving it using motion vectors overviewed here.

This article assumes prior knowledge of rendering techniques and programmable shader pipelines.

Classical flipbook animations

An old approach similar to a gif animation - given an X by Y aligned grid of images and the normalised particle age (0-1), we can get the current frame.

// Convert (uv in [0,1], int index) to atlas UV
float2 SubUV(float2 uv, uint frame, float2 subImages)
{
    float2 size = 1.0 / subImages;
    uint cols = (uint) subImages.x;
    uint idx = frame % (uint) (subImages.x * subImages.y);
    uint cellY = idx / cols;
    uint cellX = idx - cellY * cols;
    return uv * size + float2(cellX, cellY) * size;
}

// pixel main
float4 Pixel(VS_TO_PS In) : SV_TARGET
{
	// current frame
	float fFrame = In.Age * Params.SubImages.x * Params.SubImages.y;
	uint frameA = (uint) floor(fFrame);
	
	// simpler path without motion vectors
	return Texture.Sample(Sampler, SubUV(In.UV, frameA, Params.SubImages));
}

And the result is as follows. Notice the discreet transitions between frames.

Naive lerp

You can think of just blending two consequent frames together, but the result is flawed.

Optical Flow Explanation

Optical flow estimates the motion of intensity patterns between two frames. It assumes brightness is constant and that the motion is small and smooth.

Formally, if
$I(x,y,t)$ is image intensity, then under motion $(u,v)$ the brightness constraint is:
$I(x,y,t)=I(x+u,y+v,t+1)$

Which with Taylor expansion gives the optical flow equation
$I_xu+I_yv+I_t=0$
Where $I_x$ and $I_y$ are spatial intensity gradients and $I_t$ is the temporal gradient.

Since it is an equation with two unknowns, smoothness assumption is made. The two approaches of solving it are Lucas-Kanade and Gunnar-Farneback methods (check references at the bottom for more info).

Lucas-Kanade method isn’t terribly hard to implement (it requires computing per pixel gradients and then running an N by N window over the images).

But thankfully, OpenCV already implements it.

Authoring particles with optical flows

In short - software that is used to author particle simulations can be used to capture flipbook animations with motion vectors.

Houdini, EmberGen (and possibly Blender) simulate a fully volumetric effect that can either be used directly (as a VDB) or rendered from a certain perspective into a discrete set of frames with calculated motion flow obtained directly from the simulation.

Calculating optical flow from particles

MotionFrame was used (which uses OpenCV Optical Flow) to generate this.

Rendering particles with OF

To support rendering particles with optical flows, the engine needs to provide just two things: an encoded RG motion vector texture and a MotionVectorStrength parameter (that is calculated offline).

With additional helper functions (notice MotionVectorStrength usage)

float2 SampleFlow(float2 uv, uint frame)
{
    float2 atlasUV = SubUV(uv, frame, Params.SubImages);
    float2 rg = MotionVectors.Sample(Sampler, atlasUV).rg;
    float2 flow = rg * 2.0 - 1.0;
    return flow * Params.MotionVectorStrength;
}

float4 SampleSprite(float2 uv, uint frame, float2 flowOffset)
{
    float2 atlasUV = SubUV(uv + flowOffset, frame, Params.SubImages);
    return Texture.Sample(Sampler, atlasUV);
}

We can sample optical flow first, and then use it to distort sampled sub-sprites.

float4 Pixel(VS_TO_PS In) : SV_TARGET
{
    float fFrame = In.Age * Params.SubImages.x * Params.SubImages.y;
    uint frameA = (uint) floor(fFrame);
	uint frameB = frameA + 1u;

	float alpha = frac(fFrame);
	
	// flow from current and next frames
	float2 flowA = SampleFlow(In.UV, frameA);
	float2 flowB = SampleFlow(In.UV, frameB);
	float2 flow = lerp(flowA, flowB, alpha);

	// distorted sprite samples at current and next frames, then frame blend
	float4 spriteA = SampleSprite(In.UV, frameA, flow);
	float4 spriteB = SampleSprite(In.UV, frameB, flow);
	float4 sprite = lerp(spriteA, spriteB, alpha);

	return In.Colour * sprite;
}

And a much smoother result

References

Klemen Lozar’s Frame Blending With Motion Vectors

OpenCV: Optical Flow Optical Flow in OpenCV (C++/Python) | LearnOpenCV #

Lucas–Kanade method - Wikipedia OpenCV - The Gunnar-Farneback optical flow - GeeksforGeeks

Simulation software: JangaFX - Real-time VFX Simulations Houdini - 3D modeling, animation, VFX, look development, lighting and rendering | SideFX

Unreal has a plugin called TFlow 🔥 TFlow - Motion vector generator - Distribution / UE Marketplace - Epic Developer Community Forums

Software that computes optical flow from existing flipbooks: FacedownFX ltd aki-null/MotionFrame: Generate motion vector for flipbook animation

Explosion flipbook was taken from there Free VFX image sequences and flipbooks MotionFrame was used to compute OF