I have a patch incoming that turns off loading the image into memory by default (unless it is a jpeg).
And wow does it make a difference; for geotiff bluemarble in the default dataset the performance difference is like night and day. Not the image is still cached in a a JAI tileset - where it can do a decent job of lazy loading etc.
To help with this I have made a new render metrics context DRAW_IMAGE_COMPRESSED used to indicate problem formats like jpeg where drawing from disk is slowed by the requirement to read everything.
Actually using the gridcoveragereader even with jpeg is pretty darn amazing; I can now load my massive jpeg test image that fails with the GridCoverageLoader. And performance is way better then I remember it.
Could it be the the change to storing the JAI TileCache on the layer context is acting as a better version of loading an image into memory then is provided by GridCoverageLoader?