Allow fragment shader to access vertex position

[ivanceras]

What problem does this solve or what need does it fill?

When writing a fragment shader for a material that needs the position of the point relative to the mesh origin.

For example, I'm writing a shader for Earth material. The mesh is destructible, so vertex is not only limited to surface vertices, but also to vertices of subterranean holes and crevices due to mesh destruction. So, fragments that is below the planet radius is using some procedural function to create a color similar to a lava.

What solution would you like?

• 1. Include the bevy_pbr::forward_io::Vertex::position into one of the fields for bevy_pbr::forward_io::VertexOutput.local_position.
• 2. Include the inverse of bevy_pbr::mesh_types::MeshTypes::world_from_local, name it: inv_world_from_local or local_from_world.

Either 1 or 2 will suffice for obtaining the mesh local position.

What alternative(s) have you considered?

Currently, I wrote a custom struct to hold an output of the vertex shader to include local_position like so:

struct MeshVertexOutput {
    // This is `clip position` when the struct is used as a vertex stage output
    // and `frag coord` when used as a fragment stage input
    @builtin(position) position: vec4<f32>,
    @location(0) world_position: vec4<f32>,
    @location(1) world_normal: vec3<f32>,
    // the vertex position relative to the scene, this is not yet converted into view space
    @location(2) local_position: vec4<f32>,
    @location(3) instance_index: u32,
}


#ifdef VERTEX_POSITIONS
    out.local_position = vec4<f32>(vertex.position, 1.0);
    out.world_position = mesh_functions::mesh_position_local_to_world(world_from_local, out.local_position);
    out.position = position_world_to_clip(out.world_position.xyz);
#endif

Then I can use the local_position in the fragment shader to give a color of the fragment based on its position relative to the object mesh.

let local_position = in.local_position;
let is_inside = is_inside_planet(local_position);

If an inverse_world_from_local is included in VertexOutput calculating the local_position would simply be:

let local_position = inverse_world_from_local * in.world_position;

In cases, where there is no access to vertex shader, ie: Meshlet material
I use a rather expensive inverse_mat4(world_from_local) function to get the inverse of the world_from_local. Then getting the local_position would then just be:

let local_position = inverse_mat4(world_from_local) * in.world_position`;

The code for inverse_mat4 is very expensive, and it is calculated for each fragment.

fn inverse_mat4(m: mat4x4<f32>) -> mat4x4<f32> {
      let a00 = m[0][0];
      let a10 = m[1][0];
      let a20 = m[2][0];
      let a30 = m[3][0];

      let a01 = m[0][1];
      let a11 = m[1][1];
      let a21 = m[2][1];
      let a31 = m[3][1];

      let a02 = m[0][2];
      let a12 = m[1][2];
      let a22 = m[2][2];
      let a32 = m[3][2];

      let a03 = m[0][3];
      let a13 = m[1][3];
      let a23 = m[2][3];
      let a33 = m[3][3];

      let b00 = a00 * a11 - a01 * a10;
      let b01 = a00 * a12 - a02 * a10;
      let b02 = a00 * a13 - a03 * a10;
      let b03 = a01 * a12 - a02 * a11;
      let b04 = a01 * a13 - a03 * a11;
      let b05 = a02 * a13 - a03 * a12;
      let b06 = a20 * a31 - a21 * a30;
      let b07 = a20 * a32 - a22 * a30;
      let b08 = a20 * a33 - a23 * a30;
      let b09 = a21 * a32 - a22 * a31;
      let b10 = a21 * a33 - a23 * a31;
      let b11 = a22 * a33 - a23 * a32;

      let det = b00 * b11 - b01 * b10 + b02 * b09 + b03 * b08 - b04 * b07 + b05 * b06;

  return mat4x4<f32>(
      (a11 * b11 - a12 * b10 + a13 * b09) / det,
      (a02 * b10 - a01 * b11 - a03 * b09) / det,
      (a31 * b05 - a32 * b04 + a33 * b03) / det,
      (a22 * b04 - a21 * b05 - a23 * b03) / det,
      (a12 * b08 - a10 * b11 - a13 * b07) / det,
      (a00 * b11 - a02 * b08 + a03 * b07) / det,
      (a32 * b02 - a30 * b05 - a33 * b01) / det,
      (a20 * b05 - a22 * b02 + a23 * b01) / det,
      (a10 * b10 - a11 * b08 + a13 * b06) / det,
      (a01 * b08 - a00 * b10 - a03 * b06) / det,
      (a30 * b04 - a31 * b02 + a33 * b00) / det,
      (a21 * b02 - a20 * b04 - a23 * b00) / det,
      (a11 * b07 - a10 * b09 - a12 * b06) / det,
      (a00 * b09 - a01 * b07 + a02 * b06) / det,
      (a31 * b01 - a30 * b03 - a32 * b00) / det,
      (a20 * b03 - a21 * b01 + a22 * b00) / det);
}

Additional context

destruct.mp4

[IceSentry]

Why can't you use the world_position for this?

Adding this to bevy would need to be optional because that would be a lot of wasted bandwidth for most games. The issue is I'm not sure how that would be implemented to let users control that.

[ivanceras]

Why can't you use the world_position for this?

Because world_position only works if the Mesh is not rotated or translated around.
If I use world_position, the mapping will be distorted for objects that are not in the origin and/or rotated at some axes.

[ivanceras]

Adding this to bevy would need to be optional because that would be a lot of wasted bandwidth for most games. The issue is I'm not sure how that would be implemented to let users control that.

There is only 1 field that needs to be added to VertexOutput which is a vec4<f32>, even a vec3<f32> would suffice. I don't think it would add a lot of bandwidth.

struct VertexOutput {
    ...
    @location(2) local_position: vec4<f32>,
    ...

[ivanceras]

Even better:

• Use the world_position to be set to the value of vertex.position in mesh.wgsl

#ifdef VERTEX_POSITIONS
    out.world_position = vertex.position;
    let position = mesh_functions::mesh_position_local_to_world(world_from_local, vec4<f32>(vertex.position, 1.0));
    out.position = position_world_to_clip(position.xyz);
#endif

If users want the old intended value for the world_position in the fragment shader, they can do so cheaply using:

let old_version_world_position = mesh_functions::mesh_position_local_to_world(world_from_local, in.world_position); 

[IceSentry]

There is only 1 field

It's still 128 bit more per vertices and it would not be used by the vast majority of people. I have no idea if that actually would make a noticeable performance impact. A PR implementing it would need benchmark numbers to be approved.

Use the world_position to be set to the value of vertex.position

That's not better, that's worse because now the world_position isn't the world_position and the much more common use case of the world_position that is already used by every mesh using the StandardMaterial would break.

[DGriffin91]

I think it probably makes sense to include the world to local matrix so the fragment world position (or any world position/direction) can be cheaply converted to local space. Part of it is already included:

bevy/crates/bevy_pbr/src/render/mesh.rs

Line 264 in 160bcc7

pub local_from_world_transpose_a: [Vec4; 2],

[tychedelia]

Linking this as another possible general solution that would allow access to whatever in the vertex shader to forward to fragment. #13373