Content aware image cropping in .NET
In a recent project we had the requirement to automatically crop photos to various formats. The simplest solution that comes to mind first is to start from the center of the image and crop the edges depending on the target format.
While this simple approach may work in some cases, it quickly fails, especially when there are people involved. We don’t want to cut off persons completely (bad) or partially (even worse). Let’s look at an example:
Original image (990x750):
Cropped to 400x600:
Cognitive Services
We thought that we probably weren’t the first developers to encounter this problem, so we did some research for existing solutions. We were already using Microsoft’s Computer Vision API (part of the Cognitive Services) in the same project, so naturally that was the first place to check. Indeed, there is an endpoint for generating thumbnails in arbitraty sizes - and while per default it just seems to crop the center part of the image as described above, it is possible to enable “smart cropping” which generates better results:
Unfortunately it only returns the cropped image itself and no information about which part of the image was cropped. We need that information, so we couldn’t use this approach.
Entropy crop
The imaging library ImageSharp (among others) which we also happen to use in the project offers an algorithm called “entropy crop”. It is based on edge detection and tries to preserve as much information as possible. We tested with different images and got mixed results, but the major problem was that there is no way to specify the target image size, so we couldn’t use that either.
smartcrop.js
Another open source library we found was Jonas Wagner’s smartcrop.js. Although it is written in JavaScript and thus we couldn’t use it directly, it did look promising enough to give it a try and we were quite pleased with the results. It uses various heuristics to determine the best crop candidate, for example edge detection or color analysis. A detailed description of the algorithm is available on github.
There are a lot of options available to fine-tune the algorithm to your liking (e.g if it is allowed to “zoom in” when selecting the crop area). It also allows to specify so called “boost areas” in the image that increase the score of the region and thus prevent it from being cropped. This feature, of course, is immensely useful when combined with face detection. We happen to already know the positions of any faces from our prior analysis with the Cognitive Services, so we can just pass these rectangles as boost areas.
There are several ports of this awesome library for various languages…unfortunately, none for .NET…until now. :)
Smartcrop.net
As we were convinced by the results and the source code was available, we decided to port it to C# (using the aforementioned ImageSharp).
Smartcrop.net has all the features and options of the original. It is available as a .NET Standard 2.0 library on nuget, so it works with the full .NET framework and with .NET Core. And of course it is open source as well.
Head over to github to browse the code or try the sample app!
