Poor performance with an image effects on a camera with "Depth Only" clear flag

I have a 2 camera setup:

1. Camera1: orthographic, depth: 0, clear flag: Solid Color, renders the background and objects X
2. Camera2: perspective, depth: 1, clear flag: Depth Only, renders objects Y

I end up with background and objects X rendered first, and then objects Y rendered on top with a perspective projection.

Now I want to add a color grading image effect to the entire screen. I add the Image Effect to Camera2. Everything seems to work fine in the editor, however...

Poor performance with an image effects on a camera with "Depth Only" clear flag