Photoshop: Rotating Smart Objects

Here’s a little photoshop quirk a lot of users may not be aware of. It has has to do with the rotation of smart objects or shape layers. I found when rotating a smart object with sharp straight edges at small angles (20 degrees or less, say) the edges would appear somewhat jagged.

It almost looked as though there was no anti-aliasing but it was not the case. Closer examination revealed that there was anti-aliasing, there just wasn’t enough. I was not happy with the situation and I couldn’t find any obvious solution. On one occasion while experiencing this problem I thought I would try to rasterize the layer and see if there was any change. There was no effect. On another occasion I thought I would try something different. As an experiment I rasterized the layer before I rotated it. Amazingly the object edge no longer had the jagged appearance it had before. Upon closer examination I could see that there was definitely more anti-aliasing than before.

In the image below, the upper rectangle is a rotated shape layer while the lower rectangle was rasterized before rotation.

Zooming in on the edges reveals the the differences in anti-aliasing.

There is no rhyme or reason why this happens but apparently there is a different anti-aliasing algorithm used for rotation depending on the type of object; something to be aware of. So as a general rule, if you are getting poor anti-aliasing results try rasterizing your layer before performing any transforms. Make a copy of the layer before rasterizing as a backup.