antialiasing on polygons - not transitioning to mapbox-gl-native

[miles-dev]

Platform: Web (via above API) and mobile
Mapbox SDK version: Latest

Expected behavior

The same lines you see in web via Studio would persist to the mobile and raster-tiles-from-styles API maps.

Actual behavior

These antialias lines on the polygons are dropped when transitioning from mapbox-gl-js to mapbox-gl-native.

---

More details/screenshots:

When using https://www.mapbox.com/api-documentation/#retrieve-raster-tiles-from-styles, we noticed that anti-aliasing on polygons appears to not perform equivalent to mapbox-gl-js.

This can be seen in the below screenshots:

• Style as viewed in Studio (note the lines):
  
• Resulting tile from the above API (lines are missing):
  
• Same style shown on mobile (lines are still missing):
  
• GIF of enabling/disabling antialias in Studio to see how it matches the above screenshots:
  

[kkaefer]

It looks like this issue occurs when rendering fill layers that use DDS values for fill color. In this case, there is no antialiasing between polygons on the same layer. When we built the initial fill layer rendering, the assumption was that we don't want antialiasing between polygons in the same layer that line up to avoid artifacts.

We're rendering polygon outlines in the same color as the polygon in order to achieve an antialiasing effect. In this case, the rendering order works like this:

• (opaque pass) draw fill, write to depth buffer
• (translucent pass) draw outline, obeying depth buffer that cuts out half of the outline, so that we only render one "side" of the line

When the outline color didn't match the fill color, we did it this way:

• (opaque pass) draw fill, write to depth buffer
• (translucent pass) draw outline, not obeying depth buffer for this particular layer, but obeying it for other layers

This seems to be the part that broke:

• We no longer ignore the depth buffer when rendering because the sublayer for a colored outline now has value 2 (instead of 0, as it used to be)
• We're writing fragments we draw for outlines to the depth buffer, but should not do this, since the outline shader produces translucent fragments
• We're drawing the outline before drawing the fill when the fill color is translucent

[kkaefer]

Rendering in Native with the fix applied: